Speaker:

Dr Martin Ravallion, Senior Research Manager, Development Research Group, World Bank

Discussants:

Kate Bird, ODI Research Fellow

Massimiliano Cali, ODI Research Officer

Chair:

Andrew Shepherd, Director of Programmes, Rural Policy and Governance Group (RPGG), ODI

1) Remarkably little is known about the long term impacts of project aid to lagging poor areas. This paper is a technical assessment of the impacts of a large World Bank-financed rural development project in Southwest China that was initiated in 1995 and on which disbursements ended in 2001. The assessment is made in the context of China's remarkable record in aggregate poverty reduction (from poverty rates of 53% in 1980 to 8% in 2001) but with a persistent problem with geographic disparities - specifically for resource-poor inland lagging regions;

2) The World Bank-supported Southwest Poverty Reduction Project (SWP) was a classic integrated rural development project. It aimed to reduce poverty by targeting poor areas (Guangxi, Guizhou and Yunan with a total population of some 120m people) and investing in poor farm households and social services and rural infrastructure. The four key components of the project were income generating activities; off-farm employment; social services and infrastructure and institution building and poverty monitoring. The US$460m project funds were applied to the following components: agriculture (43%), infrastructure (17%), rural enterprise (11%), labour mobility (10%), education (9%) and health, institution building and project and poverty mapping (all < 5%);

3) The project did not tackle a number of key structural constraints to rural development in China. These include the restrictions on internal migration (the houkou system); the administrative allocation of land undermining the development of a land market and the high level of uninsured risk for the poor - resulting in a high level of precautionary savings;

4) The Government of China traditionally focuses upon village and small-holder investment programmes as the main direct intervention for fighting poverty. The project was implemented through a project office with links to government structures. The SWP project demonstrated a greater extent of integration across sectors, higher levels of participation with beneficiary groups, more resources and a much clearer focus on monitoring the impact of the intervention - compared with conventional government-driven approaches in poor area programmes. Government tends to regard the impact of a project as inevitable if the implementation process is correct;

5) The assessment was very detailed and involved monitoring closely (daily income and consumption diaries for individuals, households and communities with monitoring checks every two or three weeks) over a ten year period 1996 to 2006 on 2000 households in 100 villages within the project area and 2000 households in 100 villages out-with the project area. Cash consumption and income was augmented with imputed values for in-kind benefits. The fact that the baseline assessment was in 1996 - a year after the start of the project - reflected a failure to adequately communicate the project start date between government and the evaluation team;

6) Fundamental to evaluation is the recognition that project impact is the difference between the relevant outcome indicator with the project and that without it (essentially evaluation is a missing data problem - how to compare the actual out-turn with the counterfactual situation in which the project intervention did not take place);

7) Methodologically, impact evaluation should balance any selection bias between the treatment group (100 villages in the project area) and the comparison group outside the project area that is not a result of the project itself. For instance, local political economy and knowledge may result in the spill-over of benefits from the project areas into the comparison group. As long as any bias is additive and time-invariant, 'diff-in-diff' will work. However diff-in-diff hides the true impact when targeted areas have lower growth prospects than the comparison group - which is obviously a serious issue with projects that specifically target lagging areas;

8) During the disbursement period, from 1995 to 2000, the SWP project resulted in a sizeable but transient impact on income in the treatment area. Consumption was not significantly affected - due to very high levels of household saving. Analysis indicates that this saving was precautionary and there is no evidence that the project had changed the returns to savings and that the project area was entering a new growth trajectory (the 'big push' or virtuous cycle growth scenario). The project did not change agricultural productivity significantly and there was some evidence of asset value increases in assets such as livestock and housing. There was a large initial impact on headcount poverty, as measured by income. However, consumption changes were mainly observed in the middle-income groups. There was no sign of any significant impact on overall perceived standard of living or satisfaction indices. The evidence from self-perception assessments indicated that much too much weight is given to current living standards - compared with the project baseline scenario;

9) There is evidence of a quite significant (about 40%) displacement of non-project funds from the project area to non-project areas during the project implementation period. It is also possible that non-project villages also gained knowledge as a result of the project. These 'spill-over' effects would have blurred the impact of the project itself.

Discussion

The discussion clustered around two sets of issues:

Evaluation methods

Questions were raised about:

In response the Speaker noted:

Generally a mixed methods approach is appropriate in evaluations, what this exercise has revealed is the dangers of using qualitative methods in isolation. There is no obvious role for civil society organisations in an evaluation of this nature. Generally the use of randomisation methods is not politically feasible - and may be ethically questionable. The stimulation to trade, through increasing demand in the project area, may equalise the benefits between the project and non-project areas. The key tool to identifying good comparison areas is a good quality poverty map. The lumpy nature of investments may, indeed, account for the very high levels of savings observed in this evaluation.

Implications for this type of project aid

Questions were raised about:

In response the Speaker noted:

The high standard errors and high levels of variance imply very differential project impacts in each local area - which may well support the case that aid effectiveness can be attributed to the quality of local institutions. A key blockage in rural China is the policy framework that prevents the development of a land market - to avoid devaluing the main non-labour asset of poor rural people. Also the population registration system, by reducing internal migration, has a significantly negative impact on poor rural areas. Reform in these two areas would have a much greater impact than external development assistance.