From August 2005 to March 2006 HEARD conducted a study, in partnership with ODI, to evaluate the Amajuba Family and Child Welfare Conference as an element of the Amajuba Child Health and Wellbeing Research Project (ACHWRP)'s broader advocacy and lobbying strategy. ACHWRP has both a scientific objective - to produce sound information to understand the welfare of children in the Amajuba district, in the context of HIV/AIDS - and a practical objective: to contribute to the development of coherent district child welfare management plan. In collaboration with district stakeholders, the conference, held in Newcastle on the 24th November 2005, brought together over 500 representatives of community organisations, government departments and interested residents of the Amajuba district. The event also allowed ACHWRP to present the baseline research results of the first of its three planned surveys.
This RAPID project was coordinated by HEARD's Research Director and involved the ACHWRP team, as well as a HEARD Researcher and Research Assistant (RA). The study had two primary objectives: 1) to measure whether the conference was a useful advocacy tool to influence policymakers; 2) to test some of ODI's RAPID framework tools developed to 'bridge' research and policy on HIV/AIDS. The assessment of the conference was done primarily by means of an ODI evaluation called Outcome Mapping, which focuses on outcomes as behavioural changes of people, groups and organisations with whom a programme works directly (referred to as 'boundary partners'). Also, Outcome Mapping does not assume causality: the project's activities may be considered to contribute to desired changes, but not to directly cause these outcomes.
In this case, Outcome Mapping was to describe precisely: the aims of the conference; how it was expected to change behaviours (thinking and actions of child welfare organisations) as part of the broader aim of ACHWRP to improve child welfare management in the Amajuba district; who ACHWRP's 'partners' were and what we wanted them to do; what the actual (intended and unexpected) outcomes were. During this exercise the team articulated the following mission for the ACHWRP Conference: "through the Amajuba Child and Family Welfare Conference, the ACHWRP team will work to contribute to the development of a vibrant, responsive, broad-based network of family and child welfare organisations. It will encourage links between family and CWOs and stimulate discussion on key issues and knowledge sharing among these, with the aim of identifying important gaps and challenges and establishing a task force which monitors, evaluates and makes recommendations on progress..."
Through this exercise, we chose to track the following monitoring priorities, given the potential use of the information to ACHWRP: 1) three priority partners, namely: a) the Department of Education (DOE) - chosen because of its strategic importance for ACHWRP and the existing relationship - and b) the Newcastle AIDS Council and c) Dannhauser Municipality respectively - chosen because of the challenge they represented in terms of establishing a relationship; 2) two conference-related strategies, i.e.: a) the dissemination of findings and conference-related information in a form which is useful to stakeholders and b) ACHWRP's strategy of activating or maintaining contacts, towards creating an alliance with key individuals in partner organisations; 3) ACHWRP's organisational practices related to the conference.
Specific instruments used to carry out the Outcome Mapping monitoring phase and post-conference evaluation included: workshops held with ACHWRP over a 4 ½ month period; the dissemination of a post-conference newsletter (circulated in both English and Zulu); questionnaires and interviews. Representatives from 24 Child Welfare Organisations (CWOs), who had attended the conference, were chosen to be interviewed for the newsletter (information dissemination strategy) and post-conference evaluation.
Below are key results of this study:
* The participants' impressions of the conference (logistics and content) were positive on the whole; it provided knowledge and brought together a number of CWOs. Some general recommendations were that conferences need to be longer, held more regularly and be more broadly participative.
* The conference gave rise to further networking amongst C and FWOs (child and family welfare organisations), but it is unclear as to whether these contacts will lead to any concrete follow-up in the future.
* Partner monitoring revealed that the conference has modified partners' perspectives on child and family welfare issues. However, there is no tangible evidence of follow-up and/or impact on planning and activities.
* Relationships with all three partners monitored have improved. This has been most evident with DOE (e.g. greater access of ACHWRP staff to DOE staff, discussions around closer collaboration with ACHWRP and the offer to provide ACHWRP with office space). However, there have also been smaller developments with the Newcastle AIDS Council and Dannhauser Municipality (such as invitations to Council/Municipal meetings and improved relationships with key representatives).
* The conference appears to have been successful in creating and/or enhancing interest in research in general, and in ACHWRP's work in particular. C and FWOs demonstrated enthusiasm around collaborating with ACHWRP, using research to inform policy and activities and/or receiving ACHWRP presentations.
* The newsletter was considered effective for information dissemination. However, further short-term conference follow-up activities would be welcomed, whether in the form of another newsletter, radio coverage, further conferences or a combination of these channels.
* Although it is still premature to evaluate HEARD's strategy of seeking alliances in key organisations, some positive developments are: the good reception that HEARD management received by representatives of Amajuba District Municipality and DOE; interest demonstrated by these stakeholders in working with ACHWRP in the future and taking greater 'ownership' of the project; interest expressed by the Premier's Office in working more closely with ACHWRP in the future.
* Reflections by the ACHWRP team, via the organisational practices monitoring, for future operations included: the need to ensure more timely dissemination activities, render meetings and dissemination more targeted and relevant to stakeholders, start the planning and budgeting process earlier and increase the involvement of all members of the ACHWRP team in the planning of major projects and events.
Conclusion and recommendations
The results of this study indicate that the conference was useful as a strategy, but only as one element of a broader and longer-term process of networking, advocacy and lobbying. Perhaps the most significant outcome is the apparent change in district stakeholders' perspective on child and family welfare issues. However, this was a short-term study and it is too early to evaluate impact on planning and/or partners' activities or to expect major behavioural changes. What did emerge clearly is wide-spread interest, amongst district stakeholders, to use research (ACHWRP and other) to inform interventions.
Notably, this Outcome Mapping exercise was valuable for revealing actions (particular contacts to be forged and/or strengthened, and specific activities) that the ACHWRP team needs to take, to capitalise on the conference. The exercise has enabled the ACHWRP to move towards achieving its expected outputs for the conference; vis: a refined advocacy strategy to influence child welfare agencies, additional information on the 'context' of child welfare for ACHWRP's Social Assessment; a 'Lessons learned' document to inform future studies of this nature.
Finally, in our opinion, this study has also shown that Outcome Mapping can be successfully applied to a small short-term project culminating in an event (e.g. the conference), provided the instrument is adapted as necessary. In this case some modifications were necessary, partly to render the process less time-consuming and more comprehensible to participants; these included simplifying some of the steps and the terminology used.
We have made a number of recommendations on the way forward, such as: continuing to encourage coordination and collaboration among stakeholders; continuing to build and nurture key partner relationships, to fully reap the fruits of the conference and related activities; encouraging a sense of 'ownership' of the ACHWRP project amongst key district stakeholders through a continued participatory approach; balancing clear long-term goals with short-term flexibility, which includes recognising and following up on unexpected developments; using 'lessons learnt' highlighted by ACHWRP staff to improve future organisational practices.