Terms of Reference

TERMS OF REFERENCE FOR THE EVALUATION OF

Baltic Agricultural Run-off Action programme 1998-2002, 

siauliai, Matsalu-haapsalu and väinameri projects

 Background

Degradation of the Baltic Sea Ecosystem has affected the water quality, modified the biodiversity of the ecosystem and affected the fishery. In response to these changes the Joint Comprehensive Environmental Action Programme for the Baltic Sea (JCP) was developed in the beginning of the 1990´s. It provides an environmental framework for restoration of the Baltic Sea ecosystem and one of the areas addressed is reduction of non-point pollution from agriculture. In 1992 the Swedish government allocated funds to co-operate with Estonia, Latvia, Lithuania, Poland and Russia in reducing pollution from agriculture to the Baltic Sea. The initiative was named the Baltic Agricultural Run-Off Action Programme (BAAP). The following projects aim to implement the JCP by reducing negative environmental impact from agriculture to the Baltic Sea.

The Baltic Agricultural Run-Off Action Programme (BAAP)

Sida allocated 25 MSEK in 1998 to the prolongation of the BAAP for the period 1998-2002. The Swedish Agricultural University (SLU) is the executing party of the programme. The BAAP is an environmental/agricultural co-operation programme designed to improve the water quality of surface and groundwater bodies and subsequently the Baltic Sea by reducing leakage of nutrient run-off from agriculture. The programme shall also enhance sustainable agricultural development and was in 2001 broadened to also include pesticide management. The programme covers co-operation with Estonia, Latvia, Lithuania, Poland and Russia.

The programme comprises agri-environmental measures based on recent developments in the Baltic Sea Region; the degradation of the Baltic Sea ecosystem; post-soviet restructuring of agricultural production; development of new agri-environmental policies; European Union enlargement; introduction of new technologies and practices; and increased regional co-operation around the Baltic Sea.

Activities include for example:

–        Strengthening the environmental surveillance system (monitoring of agricultural run-off)

–        Strengthening demonstration capacity for sustainable agriculture (demonstration watersheds and farms).

–        Introducing ecological management systems at farm level (EMS).

–        Assisting in developing agri-environmental policies and recommendations

–        Promoting regional co-operation (BAAP Regional Network)

–        Training in and demonstration of suitable techniques for better pesticide management

The programme is organised as a partnership between the countries and the different project teams including international and local partners

A number of institutes, universities, advisory services etc are partners of the programme.

The programme started already in 1994 and is since 1998 funded by Sida. It was evaluated in 1997 by Svenska Lantbrukssällskapens förbund, Finland. The current programme started in 1998 and will be finalised by the end of 2002. The pesticide component will be finalised by the end of 2003.

Haapsalu and Matsalu Bays Environment Project

*

Sida allocated 3. 4 MSEK for the Agricultural Run-Off Element as part of the Haapsalu and Matsalu Bays Environment Project in December 1997. The project is being finalised during spring 2002. SLU is the executing party of the project and partners in Estonia are the Ministry of Environment and the Ministry of Agriculture. Swedeagri and JTI were contracted to implement the project. 

 The overall objective of the project is to preserve the unique fauna and flora as well as agriculture in the sensitive Matsalu Bay Area. This should be attained by reducing the agricultural run-off and improving natural resources in the agro- ecosystem of the Bay Area and the Kasari River Catchment.

 The project comprises demonstration and information activities with the purpose to introduce new technologies and sustainable management practices. This has included demonstration activities, environmental monitoring, preservation of wildlife and biodiversity, information strategy, extension services and legislation activities.

 The project was reported on at a final seminar in Uppsala in March 2002. The final report is now being concluded.

 Siauliai Environment Project: Non point source pollution control in the Upper Lielupe River Basin, Lithuania

*

Sida allocated 2.2 MSEK to the Siaulai Environment Project in March 1999 The project is being finalised during the spring 2002. SLU are the executing agency and partners in Lithuania are the Siaulai Regional Environment Department and the Ministry of Environment. The Swedish Institute of Agricultural and Environmental Engineering and the Lithuanian Institute of Water Management were contracted to implement the project.

 The overall objective of the project is to attain a long-term measurable reduction of water pollution from agricultural sources in the Upper Lielupe river basin. There are large animal farms in this part if Lithuania causing pollution. Activities have included demonstration farm and water monitoring station, assessment of nutrients and pesticides from agriculture, implementation of Code of Good Agricultural Practice, establishment of a monitoring system, education, training and information activities.

 The project was reported on at a final seminar in Uppsala in March 2002. The final report is now being concluded.

Sustainable agriculture, eco-tourism, and regional Development in  Western Estonia 

 *

Sida has allocated 3 MSEK for the Väinameri project. This project includes agrienvironmental activities but is much broader in its scope. It started in 1999 and will be finalised during the fall 2002. It is included in the evaluation because there are several linkages to the BAAP programme and also to the up coming GEF Baltic Sea Regional Programme (BSRP).

Väinamäri is a coastal area located in western Estonia, which ranges from the Matsalu Bay to the islands Dagö and Ormsö. The inland sea Väinamäri has been defined a marine reserve, within the HELCOM network of marine protected areas; and the coastal area by the Matsalu Bay is on the Ramsar Convention list for international valuable wetlands. The Swedish WWF and Estonian environmental NGOs have co-operated for several years. Sida finances co-operation between the WWF and its Estonian sister organisation, with focus on sustainable agriculture and animal husbandry. Other complementary activities involve eco-tourism development and handicraft, to maintain the precious value of the nature and culture in the area and to achieve a sustainable regional development.

2 Purpose and Scope of the Evaluation

The purpose of the evaluation is to review the results achieved in the Sida financed components of the BAAP programme, the Matsalu-Haapsalu, Siaulia and the Väinameri projects (the four projects described in chapter 1) for the period 1998-2002. The overall results of the programme/projects are, however also, of central interest for an assessment of the effects of the Swedish contribution. The evaluation is undertaken mainly for the purpose of accountability of the programme/projects, but also learning and promotion are important objectives.

The evaluation is undertaken at this time since the BAAP programme is coming towards the end of implementation and the other projects are also about to be finalised. An evaluation of the BAAP programme was last undertaken in 1997 and considering the total financial support to the programme an evaluation is appropriate to undertake before finalisation. The other projects are included since they in several aspects are linked to the BAAP programme. The BAAP programme is not foreseen to continue in its present form, mainly since assistance to the Baltic States is being phased out. For Russia further support is planned and a feasibility study for such support was initiated in May 2002. Support to agrienvironmental activities for the coming years are foreseen to be closely linked to the up-coming GEF  BSRP programme.

Lessons learned from the programme/projects should be developed, and serve as a basis for recommendations for future programme/projects. The GEF BSRP needs to be taken into account when developing these recommendations.

The interested parties of the evaluation are expected to be Sida, SLU, WWF, project owners in the partner countries, project implementation teams in Sweden and in the Baltic States, the Ministry of Environment and the Ministry of Agriculture in Sweden and others concerned.

3 The Assignment

The following issues shall be covered in the evaluation of the four projects:

 Relevance

The relevance of the programme/project objectives as defined and documented during project preparation; i.e. in terms of reference for the project, the requests and needs of the project partners and co-operating countries. Have the interventions been relevant in relation to the goals and policies of Swedish development co-operation and to the needs and priorities of the partner countries and target groups?

Achievement of objectives (effectiveness)

Achievement and realism of the project goals as defined and documented in the terms of references, contracts and agreements for the projects, taking into account possible changes during implementation. Have outputs been produced as planned? Have programme and project objectives been fulfilled and is it possible to measure this?

4 Impact

What are the intended and unintended effects of the activities, including effects on the intended target groups and on others? What are the positive and negative effects in the short and long term?

Efficiency

The efficiency of the support provided should be analysed according to its adequacy in terms of the forms of inputs, their timing and duration.

Efficiency of project management, the quality of work plans, budgets, and reporting routines for the different components should be assessed. An assessment of the cost-effectiveness of the Swedish contribution shall be made. Are there more cost-effective methods of achieving the same results? What have the added value been of the Swedish financed projects?

Sustainability

Sustainability of the transfer of knowledge and institutional strengthening, improvements in advisory and farmers development and the possibilities and commitment of the co-operating partners to use the results of the programme/projects in the long-term should be addressed. To what extent will activities, outputs and effects are maintained or acceptable returns be provided when donor support has come to an end? What other funding or donor support exists that can continue funding these type of activities?

Other criteria

The evaluation shall take into account issues of public information, consultation and participation. Synergy effects between the programme and the different projects should be addressed and if it would have been beneficial to more closely tie them together.

4    Methodology, Evaluation Team and Time Schedule

Co-operating partners should be encouraged to be actively engaged in the evaluation (although not as part of the evaluation team). The learning aspect of the evaluation should be taken into consideration in the planning and design of the evaluation.

The evaluation process shall provide a forum for discussion among the stakeholders regarding methodology, implementation and sustainability of the programme/projects.

4.1 Methodology

Alternative methods and approaches for the evaluation than described below can be proposed.

Desk study and preparations in Sweden

The evaluation team shall review the documentation listed in Annex 1 and 2 and other possible documentation handed over at the beginning of the assignment. This includes the documentation forming the basis for the work; e.g. project preparations, terms of references, work plans, annual and progress reports.

Site visits and interviews

The evaluation team shall visit some of the project sites (tentatively 5 sites). The choice of which shall be discussed and agreed with Sida beforehand. The team shall conduct interviews with stakeholders in the countries i.e. the ministries, advisory services, municipal representatives, institutions and individual farmers.

Key persons in Sweden, involved in the project preparations, implementation and follow-up shall also be interviewed (Sida and other Swedish authorities, twinning partners, Swedish consultants and suppliers).

Reporting

An inception report, with the preliminary results of the desk study, shall be submitted to Sida within three weeks of the start of the assignment. The results from the desk study, visits and interviews will be presented to Sida in a draft report in English, and at a RGÖ (regional advisory group) meeting within 6 weeks of the field visits. Sida will then review the first draft report. The second draft report shall be submitted to Sida two weeks after Sida has commented on the first version. After revisions, Sida will distribute the second draft report to the involved parties for comments. The final version of the evaluation report shall be submitted to Sida, two weeks after Sida has commented upon it, in 10 copies and on diskette. Subject to decision by Sida, the report will be published and distributed as a publication within the Sida Evaluations series.

The evaluation report shall not exceed 40 pages, excluding annexes. Format and outline of the report shall follow the guidelines in Sida Evaluation Report - a Standardized Format (see Annex 3). The evaluation report shall be written in Word 6.0  for Windows (or in a compatible format) and should be presented in a way that enables publication without further editing. It shall have a summary of maximum 2 pages.

The evaluation assignment includes the completion of Sida Evaluations Data Work Sheet (Annex 4), including an Evaluation Abstract (final section, G) as defined and required by DAC. The completed Data Worksheet shall be submitted to Sida along with the final version of the report. Failing a completed Data Worksheet, the report cannot be processed.

Composition of Team

The team who will perform the evaluation shall have comprehensive international working experience, preferably in the transition countries in the Baltic region or eastern Europe, and have relevant knowledge of the 1) technical/ environmental and 2) management/ organisational and 3) financial issues. Experience of international development co-operation and of conducting evaluations is a requirement. At least one of the team members must be able to read and communicate in Swedish. An equal distribution of men and women in the evaluation team is desired.

Time Schedule

The assignment is expected to take the evaluation team 10-15 weeks effective time in total, including preparations at home office, interviews in Sweden, work in the partner countries, report writing and presentations.

The assignment will end with the submission of the final version of the evaluation report, in.

5 UNDERTAKINGS

Sida will inform the involved parties of the review and forthcoming visits by the evaluation team. The evaluation team will be responsible for practical arrangements in conjunction with the mission to Estonia, Latvia, Lithuania, Poland and Russia. If interviews cannot be carried out in Swedish/ Scandinavian or English, interpreters shall be hired and costs reimbursed by Sida. The evaluation team will be responsible for visits and arrangement in Sweden. Sida will ensure that all written material listed in Annex 2 will be made available. The evaluation team will during all phases of the review be assisted by Sida.

Enclosures

Annex 1: Financing decisions

Annex 2:         a. ToR BAAP + annual report

                        b. ToR Haapsalu-Matsalu + draft Final Report

                        c. ToR Siauliai + draft Final Report

                        d. ToR Väinameri + progress report

Annex 3: Sida Evaluation Report - a Standardized Format

Annex 4: Data Work Sheet