&EFA
Improving EPA's Performance
with Program Evaluation
An Evaluation of EPA's Safe Drinking Water
Program in Central America
Series No. 3
By continuously evaluating its programs, EPA is able to capitalize on lessons learned and incorporate that
experience into other programs. This enables the Agency to streamline and modernize its operations while promoting
continuous improvement and supporting innovation. This series of short sheets on program evaluation is intended
to share both the results and benefits of evaluations conducted across the Agency, and share lessons learned
about evaluation methodologies in this evolving discipline. For more information contact EPA's Evaluation Support
Division at www.epa.gov/evaluate.
At a Glance
Evaluation Purpose
To identify outcomes and determine the overall
effectiveness of EPA's Safe Drinking Water
Program activities, and to provide
recommendations for sustaining successes and
transferring the program to other regions.
Evaluation Type
Outcome Evaluation
Publication Date
May 2002
Partners
Office of International Activities, Office of
Policy, Economics and Innovation
Contact
Stephanie Adrian, OIA (202) 564-6444
Background: Why was an evaluation
performed?
In 1997, EPA launched the International Safe Drinking Water
Initiative to improve water quality. Central America was identified
as a priority region, and El Salvador was selected as a pilot country.
In October 1998, Hurricane Mitch devastated the region and EPA,
among other U.S. government agencies, was called upon by the
U.S. Agency for International Development (USAID) to assist in
the region's rehabilitation. With additional funds from USAID,
EPA expanded its program to include Honduras and Nicaragua.
This effort later became known as EPA's Safe Drinking Water
Program in Central America (Program). The Program included
four initiatives: (1) laboratory capacity-strengthening; (2) drinking
water treatment plant optimization; (3) source water protection;
and (4) safe drinking water program development. As the Safe
Drinking Water Program in Central America neared completion
at the end of 2001, EPA's Office of International Affairs (OIA)
conducted an evaluation to determine how effective the Program
had been in meeting its goals, and to identify lessons learned that
could increase the likelihood of sustaining the Program's outcomes
and help promote and guide safe drinking water efforts in other regions.
NCEI
NATIONAL CENTER FOR
ENVIRONMENTAL INNOVATION
-------
Basic Evaluation Approach: How
did they do it?
The evaluation was completed using roughly nine steps,
which are outlined below.
Step I: Develop key questions.
Step II: Review pertinent documents.
Step III: Develop a Safe Drinking Water Program
framework to better understand Program
goals and activities and determine how to
measure success.
Step IV: Develop an evaluation report outline to
better ensure that the final product would
match EPA's initial expectations about the
evaluation.
Step V: Prepare interview guides for each of the
four distinct Program components to help
answer each of the key questions.
Step VI: Conduct preparatory interviews with EPA
staff who helped guide each of the four
Program initiatives, to better focus the
subsequent interviews.
Step VII: Develop a potential list of interviewees for
individual and group interviews.
Step VIM: Conduct face-to-face individual and group
interviews in Central America involving 69
participants representing each of the four
Program components.
Step IX: Analyze data on Program initiative
outcomes and impediments to success to
develop findings and recommendations.
Evaluation Results: What was
learned?
The evaluation team found that, overall, the Program
has made good progress toward several of its short-
term goals. Much of the success achieved was at the
technical level, especially with the lab capacity and
treatment initiatives. For instance, for the Program's
laboratory capacity-strengthening initiative, which took
place primarily in El Salvador and Nicaragua, the team
found that results exceeded expectations for
improvements in the operations of the participant
laboratories. However, continued external support will
be needed if the Program is to succeed in its long-term
goal of improving drinking water quality in the region.
Factors influencing the success of Program initiatives
included EPA's effort early on to identify and cultivate
relationships with potential partners and stakeholders.
Also important was EPA's effort to be sensitive to
cultural differences and how they might impact
participants' views towards elements of the Program.
EPA's integration of the various Program components
to maximize learning was also critical. In El Salvador,
for instance, the treatment plant optimization and
laboratory capacity-strengthening initiatives were
conducted simultaneously, increasing the ability of
individuals to transfer information, share ideas and
collaborate on plans. In addition, EPA and its partners
greatly enhanced the quality of Program workshops
and training by incorporating the use of hands-on
activities.
Approach for this Evaluation
Step I
Develop Key Questions
Step II
Review Pertinent Documents
Step III
Develop Program Framework
Step IV
Develop Evaluation Report Outline
StepV
Prepare Interview Guides
Step VI
Conduct Background Interviews with EPA Staff
Step VII
Develop Potential List of Interviewees
Step VIM
Conduct Individual and Group Interviews
Step IX
Analyze Data and Prepare Findings and
Recommendations
-------
Initially, the lack of sufficient resources to fund the
purchase of laboratory equipment for all labs, to
finance pilot projects, or to make training more widely
available limited the ability of the Program to achieve
greater progress toward its goals. In addition, the
absence of a strong drinking water regulatory
framework in each country limited the degree to which
Program initiatives gained sufficient visibility to garner
enough resources to continue. The lack of trust
between organizations such as health ministries and
national water utilities at the senior management level
also limited the ability to share lessons learned among
Program initiatives, and subsequently limited the value
of the workshops. Finally, the relatively short period
of time (approximately two and a half years) during
which the Program has been up and running has
impacted the progress made toward longer-term goals
and objectives.
Evaluation Outcomes: What
happened as a result?
The evaluation identified some key impediments that
limited the Program's ability to achieve greater success.
Eleven recommendations were developed that
centered on each of the four Program initiatives and
that address the sustainability of the Program and its
transferability to new regions. The lessons learned
identified in the report are and will be applied to future
international water programs, and OIA indicated that
the evaluation has positively impacted its thinking
about how to implement future water programs.
United States
Environmental Protection
Agency
Office of Policy,
Economics and Innovation
(1807T)
June 2003
EPA-100-F-03-007
------- |