U.S. ENVIRONMENTAL PROTECTION AGENCY
OFFICE OF INSPECTOR GENERAL
Catalyst for Improving the Environment
Evaluation Report
Border 2012 Program Needs to
Improve Program Management to
Ensure Results
Report No. 08-P-0245
September 3, 2008
-------
Report Contributors:
Lauretta Ansah
Jerri Dorsey
Gabrielle Fekete
Jeffrey Harris
Olga Stein
Abbreviations
EPA
FTE
GIS
IBEP
NGO
NPM
OEI
OIA
OIG
ORD
OSW
RFP
SEMARNAT
U.S. Environmental Protection Agency
Full Time Equivalent
Geographic Information Systems
Integrated Border Environmental Plan
Non Governmental Organization
National Program Manager
Office of Environmental Information
Office of International Affairs
Office of Inspector General
Office of Research and Development
Office of Solid Waste
Request for Proposal
Secretariat for the Environment and Natural Resources (Mexico)
Cover photos: From U.S.-Mexico Environmental Program: Border 2012, Implementation
and Mid-Term Report: 2007.
-------
U.S. Environmental Protection Agency
Office of Inspector General
At a Glance
08-P-0245
September 3, 2008
Catalyst for Improving the Environment
Why We Did This Review
We initiated this review to
examine the impact of Border
2012's program management
and organization on its ability
to meet the program's
mission: to protect the
environment and public health
in the U.S.-Mexico border
region. Specifically, we
evaluated whether the
program management and
organizational structure of
Border 2012 impact program
outcomes.
Background
The Border 2012 Program
emphasizes a bottom-up,
regional approach,
anticipating that local
I decision-making, priority-
setting, and project
implementation will best
address environmental issues
in the border region. Program
goals address water, air, land,
environmental health,
chemical exposure, and
compliance concerns.
Border 2012 Program Needs to Improve Program
Management to Ensure Results
For further information,
contact our Office of
Congressional and Public
Liaison at (202) 566-2391.
To view the full report,
click on the following link:
What We Found
We found that the current organizational structure of the Border 2012 Program
allows it to achieve a collaborative relationship at the U.S.-Mexico border and
address environmental and public health issues unique to the border region. The
structure also creates opportunities for stakeholder involvement from local, State,
and national groups while providing the program with the ability to leverage
diverse partners and create an effective convening mechanism to discuss border
issues.
However, we found management controls do not ensure that project and program
results are documented or that the Border 2012 goals are achieved. Specifically,
we found that Border 2012 lacks a systematic roadmap that defines the
relationships between resources, activities, and intended outcomes. We also
found a lack of management oversight regarding program progress towards
meeting goals and objectives. For example, supporting documentation regarding
program accomplishments was not obtained or reviewed by Border 2012 staff.
Furthermore, the current performance measures focused on outputs rather than
outcomes; several of the performance measures were not assessable. As a
consequence of the conditions cited in this report, the Agency is unable to assess
the environmental and health benefits actually achieved.
What We Recommend
EPA should strengthen management controls to effectively demonstrate program
performance. We recommend the Agency develop a strategic plan, issue
guidance to better support program results, improve performance measures, and
develop criteria for determining what constitutes successful completion of
program goals. The Agency concurred with all recommendations.
www.epa.qov/oiq/reports/2008/
20080903-08-P-0245.pdf
-------
35
V
1U
o
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
OFFICE OF
INSPECTOR GENERAL
MEMORANDUM
SUBJECT:
FROM:
TO
September 3, 2008
Border 2012 Program Needs to Improve Program Management to
Ensure Results
Report No. OS-P-O""
>-!.*-
Wade T. Najjum
Assistant Inspector General for Program Evaluation
Scott Fulton, Principal Deputy Assistant Administrator
Office of International Affairs
This is our report on the Border 2012 Program evaluation conducted by the Office of
Inspector General (OIG) of the U.S. Environmental Protection Agency (EPA). This
report contains findings that describe the problems the OIG has identified and corrective
actions the OIG recommends. This report represents the opinion of the OIG and does not
necessarily represent the final EPA position. Final determinations on matters in this
report will be made by EPA managers in accordance with established audit resolution
procedures.
The estimated cost of this report - calculated by multiplying the project's staff days
by the applicable daily full cost billing rates in effect at the time - is $442,794.
Action Required
In accordance with EPA Manual 2750, you are required to provide a written response to
this report within 90 calendar days. You should include a corrective actions plan for
agreed upon actions, including milestone dates. We have no objections to the further
release of this report to the public. If you or your staff has any questions regarding this
report, please contact me at 202-566-0827; or Jeffrey Harris, Director of Special
Studies, at 202-566-0831 or harris.jeffrey@epa.gov.
-------
Border 2012 Program Needs to Improve 08-P-0245
Program Management to Ensure Results
Table of Contents
Chapters
1 Introduction 1
Purpose 1
Background 1
Noteworthy Achievements 2
Scope and Methodology 2
Prior Reviews 3
2 Unique Organizational Structure Created to Implement Bottom-up
Approach 4
Border 2012 Organizational Structure 4
Border 2012 Program and Project Management 6
3 Program Management Shortcomings Inhibit Verifying Results 7
Border 2012 Lacks Support for Its Claimed Accomplishments 7
Some Performance Objectives Are Not Measurable 8
Outcome Measures Are Being Developed but Challenges Remain 9
Systematic Planning for Program Goals Is Lacking 10
Conclusion 11
Recommendations 12
Agency Comments and OIG Evaluation 12
Status of Recommendations and Potential Monetary Benefits 14
Appendices
A Evolution of EPA's U.S.-Mexico Border Program 15
B Border 2012 Objectives for Goals 3 and 4 16
C Detailed Scope and Methodology 17
D Border 2012 Guiding Principles 18
E Logic Model of Border 2012 Goal 3 19
F Logic Model of Border 2012 Goal 4 20
G Agency Comments and OIG Evaluation 21
H Agency Comments 30
I Distribution 37
-------
08-P-0245
Chapter 1
Introduction
Purpose
As part of a U.S. Environmental Protection Agency (EPA) Office of
Inspector General (OIG) evaluation agenda to assess new approaches to
environmental protection, we initiated this review to examine the impact
of Border 2012's program management and organization on its ability to
meet the program's mission. That mission is to protect the environment
and public health in the U.S.-Mexico border region, consistent with the
principles of sustainable development. Specifically, we sought to
evaluate whether the organizational structure and program management
of Border 2012 impact program outcomes.
Background
Border 2012 is a joint U.S.-Mexico program organized around the
concept of a regional, bottom-up approach. The Border 2012 Program
was launched in 2002 as a 10-year joint effort to improve the
environment and protect the health of the nearly 12 million people living
along the border.1
The mission of Border 2012 is to protect the environment and public
health consistent with the principles of sustainable development.2
Border 2012 has six goals to support the program's mission:
Goal 1: Reduce water contamination.
Goal 2: Reduce air pollution.
Goal 3: Reduce land contamination.
Goal 4: Improve environmental health.3
Goal 5: Reduce exposure to chemicals as a result of accidental
chemical releases and/or acts of terrorism.
Goal 6: Improve environmental performance through
compliance, enforcement, pollution prevention, and promotion of
environmental stewardship.
1 See Appendix A for more on the evolution of the U.S.- Mexico Border Program.
2 Sustainable development is defined as conservation-oriented social and economic development that
emphasizes protecting and sustaining use of resources, while addressing both current and future needs as
well as current and future impacts of human actions.
3 See Appendix B for objectives related to Goals 3 and 4.
1
-------
08-P-0245
The map below depicts the U.S.-Mexico border region, which extends
more than 2,000 miles from the Gulf of Mexico to the Pacific Ocean,
and includes over 60 miles on each side of the border, encompassing 10
US and Mexican states.
Figure 1-1: A map of the U.S.-Mexico border region.
1-, CALIFORNIA J}
"/"' '. 'I.' "" fX
v Nogales
,* - -
NEW MEXICO
OKLAHOMA
100km border zone as defined
by the La Paz Agreement
i-ir-Ty^a'35 us Sda *
Ncgala NKO Muj- - ' palom»s Ju L \
^^\ ^Jresldlo /" ""V
^ ^Ojlnaga ^ if
\ '*'--/' cludadAcul^,
jf agle pi|
CaL°4aJ Texas-Coahuila-Nuevo Le6n-Tamaullpas "l ^,^«to v~-d__;
'1 VT , , 1-; 1 ' S^\ :
Border 2012 v^
Regional Workgroups
Source: EPA Website.
According to Border 2012 management, the program is largely a
voluntary (non-regulatory), consensus-driven program. The program
takes into account cultural, language, political, economic, legal,
jurisdictional, and environmental realities. It uses a collaborative
approach, involving a multitude of partners and stakeholders to develop
solutions to environmental and health issues affecting the border
communities.
Noteworthy Achievements
The Border 2012 Program emphasizes a bottom-up, regional approach.
Border 2012 is a unique EPA program because it is bi-national, and
addresses needs on both sides of the border. This program operates as a
convening mechanism by bringing together a variety of local, regional,
and national stakeholders to address the unique environmental and
human health issues along the border.
Scope and Methodology
This review assesses the performance of the Border 2012 Program on
the basis of its reported results, and the results and measures culled from
the grant program established under Border 2012's auspices. In
addressing our overall objective of "Does the organizational structure
-------
08-P-0245
and program management of Border 2012 impact program outcomes? ",
we conducted meetings with program staff, regional staff and grant
recipients. We also reviewed publicly-available program documents,
measures, budgets, and staffing, and requested information on training
requirements and internal program management.4 We examined
management controls as they related to our objective.
We conducted this performance evaluation in accordance with generally
accepted government auditing standards. Those standards require that
we plan and perform the evaluation to obtain sufficient, appropriate
evidence to provide a reasonable basis for our findings and conclusions
based on our evaluation objective. We believe that the evidence
obtained provides a reasonable basis for our findings and conclusions
based on our evaluation objective. We performed our field work from
October 2007 through March 2008.
Prior Reviews
The OIG conducted an audit of the U.S.-Mexico border's water
infrastructure projects. The audit objective was to answer the question,
"Does EPA 's U.S.-Mexico Border Program have adequate controls for
obligating and using water infrastructure grant funds? "5 The OIG
found that from 2005 to 2007, EPA took actions to implement
timeframes for the Border 2012 Program projects, reduce the scope of
projects, and reduce unliquidated obligations of projects. However,
EPA needed to make additional changes to the process it used to manage
the funds Congress appropriates for water infrastructure improvements
along the U.S.-Mexico border. The report also stated that border grant
work plans did not include sufficient detail such as specific projects,
measures, milestones, or costs associated with projects.
The report made several recommendations to improve accountability for
projects and funding. With one exception, EPA generally concurred
with the OIG's recommendations. EPA expressed reservations about
being able to make changes to the program without all stakeholders
agreeing on how projects should be funded
4 Appendix C provides further details on our scope and methodology.
5 Report number 08-P-0121, Improvements Needed to Ensure Grant Funds for U.S.-Mexico Border
Water Infrastructure Program Are Spent More Timely, March 31, 2008.
-------
08-P-0245
Chapter 2
Unique Organizational Structure Created to
Implement Bottom-up Approach
The Border 2012 Program created a series of interrelated workgroups,
forums, and task forces to execute its bottom-up regional approach for
decision making, priority setting, and project implementation. EPA
Regions 6 and 9 and National Program Offices coordinate soliciting,
selecting, ranking, and administering Border 2012 projects, primarily
through issuing grants.
Border 2012 Organizational Structure
The Border 2012 Program is structured to involve a variety of internal
and external stakeholders, including Federal, State, and local
governments in the United States and Mexico, and U.S. border tribes,
and to accommodate input from the U.S.-Mexico border communities.
Below, Figure 2-1 depicts the organizational structure of the Border
2012 Program, and the coordinating groups involved.
Figure 2-1: Border 2012 Program Organizational Structure
Nat
EPA
National Coordinators
SEMARNAT
Regional
Workgroups
California-Baja California
Arizona-Sonora
New Mexico-Texas-
Chihuahua
Texas-Coahuila-Nuevo Leon-
Tamaulipas
I
Border-wide
Workgroups
Environmental Health
Emergency Preparedness
and Response
Cooperative Enforcement
and Compliance
Policy
Forums
Water
Hazardous and Solid
Waste
Task Forces
Address specific regionally- and community-identified concerns by implementing site-specific projects
Source: Border 2012 Program.
-------
08-P-0245
The national program coordinators in Mexico's Secretariat for the
Environment and Natural Resources (SEMARNAT) and EPA are the top
level of management in the Border 2012 Program.6 EPA, as one of the
national coordinators, is responsible for implementing the U.S.
responsibilities under the La Paz Agreement and the Border 2012
Program.7 The national program coordinators manage overall program
implementation, and ensure cooperation, coordination, and
communication among all Border 2012 groups. The coordinators assist
the workgroups in maintaining focus on bi-national and transboundary
environmental and public health issues consistent with the program's 10
guiding principles.8
Under Border 2012, three types of coordinating bodies have been
created. These include regional workgroups, border-wide workgroups,
and policy forums. The role of the regional workgroups is to identify
and document priority regional issues consistent with the goals and
objectives of Border 2012. Regional workgroups also identify priority
regional issues that affect more than one region for consideration by
border-wide workgroups and policy forums.
Border-wide workgroups work closely with regional workgroups and
task forces to identify projects to fulfill their respective goal and
objectives. Border-wide workgroups leverage internal and external
resources to assist regional and task force efforts and projects.
The policy forums include water, air, and hazardous and solid waste.
Policy forums identify priority border-wide policy issues that can most
effectively be addressed through a federally-led effort. The policy
forums rely on the input from regional workgroups.
Each of the three coordinating groups mentioned above has the
opportunity to create task forces to implement projects at the local level
and regional level. Task forces address selected and community
identified concerns, implement projects, and make recommendations to
Border 2012 coordinating groups. Specifically, task forces develop
project proposals and identify potential funding sources.
Regional workgroups, border-wide workgroups, policy forums, and task
forces are expected to work on building consensus on program priorities.
Additional responsibilities include developing budgets for Border 2012
6SEMARNAT is a cabinet-level department in Mexico's executive branch. The national program
coordinator for EPA is the Assistant Administrator for the Office of International Affairs.
7 The La Paz Agreement was between the U.S. and the United Mexican States on cooperating to protect
and improve the environment in the border area, signed at La Paz on August 14, 1983.
Q
See Appendix D for the 10 guiding principles.
-------
08-P-0245
projects, collecting data to monitor progress of activities, and identifying
potential resources to achieve program goals.
Border 2012 Program and Project Management
The Border 2012 Program initiates projects primarily through issuing
grants. Agency officials explained that EPA's Office of International
Affairs (OIA) receives yearly appropriations to fund projects in the
border region to achieve the program goals established in the Border
2012: U.S.-Mexico Environmental Program Framework.9 EPA Regions
6 and 9 are also involved with the grant process. Initially, the regional
workgroups assess local and regional priorities and emerging issues by
consulting with the task forces, U.S. border tribes, and the Border 2012
regional workgroup State partners. Part of the priority-setting process
also includes considering applicable priorities identified by the national
coordinators at their annual meeting.
After the OIA priorities are established, requests for proposals (RFPs)
are written to balance national, regional, and local priorities. Potential
grantees apply by submitting a pre-proposal with their project
description. Pre-proposals are ranked by each region, and the highest
rated applicants are asked to complete a full proposal for funding
consideration. Criteria used to rank proposals include funding
leveraged, measurability of project results, and advancement of Border
2012 goals and objectives. Upon approval by the region, EPA assigns a
project officer and issues grant funds.
EPA's project officer is responsible for monitoring project progress.
Once a project is complete, a final report is prepared by the grantee and
submitted to the EPA project officer. The project officer reviews the
final report and project accomplishments, and determines whether all
EPA grant requirements have been met.
' U.S. EPA, Border 2012: US-Mexico Environment Program, May 2003, £P/4-160-R-03-001.
6
-------
08-P-0245
Chapter 3
Program Management Shortcomings
Inhibit Verifying Results
Border 2012 does not have a formal process to independently verify
results or to review supporting documentation for individual project
accomplishments. The program also lacks baseline data, quantifiable
measures, or outcome measures which could verify the program's
effectiveness. Furthermore, Border 2012 does not have a strategic plan
or similar document that guides the program's implementation efforts.
Nor does the program have a systematic implementation plan to link
resources, activities, and intended outcomes. Coupled with the
decentralized structure of the program described in Chapter 2, the result
is a lack of accountability and oversight. We were unable to verify the
accomplishments reported by the program due to the program's lack of
quantifiable goals, valid measures, or supporting data and documentation.
Border 2012 Lacks Support for its Claimed Accomplishments
Border 2012 reports accomplishments on a national program level. Yet,
we found that the program does not obtain or collect supporting
documentation for accomplished objectives. The program does not have
a process to independently verify project accomplishments. According
to the Border 2012 staff, accomplishments are discussed at program
meetings but it has no guidance or process to determine whether
objectives have been successfully completed. In addition to issuing
grants in support of program objectives, Border 2012 relies on partners
to assist in meeting Border 2012 objectives. However, we found Border
2012 has no documented process to assess projects completed by
partners. Also, we found that the Border 2012 Program does not have a
process to review results submitted by the grantees, or aggregate project
results.
For example, we analyzed support for the one of the objectives with a
quantifiable measure:
Goal 4, Objective 3B: By 2007, reduce pesticide exposure by
training 36,000 farm workers on pesticide risks and safe
handling, including ways to minimize exposure for families and
children.
-------
08-P-0245
We found that the program issued several grants in support of Objective
3B. We requested documentation supporting the successful completion
of this objective. In support of this objective, the EPA program staff
provided grant final reports that supported training approximately
15,000 farm workers. However, OIA did not maintain copies of these
final reports. According to OIA, it does not routinely obtain or review
reports from the Agency's project officers. OIA assumes that grantees
and the Agency's project officers ensure the accuracy of results.
We were advised by program staff that the remainder of the training was
conducted by Border 2012 partners and therefore the support is
maintained by outside organizations. According to program staff,
Border 2012 partners were responsible for training over 30,000
individuals along the border region. However, the program does not
have, nor does it request, any support for the accomplishments claimed
by its partners.
Some Performance Objectives Are Not Measurable
We reviewed the goals, objectives, and subobjectives in the Border 2012
framework document and found two of the objectives to be broad in
nature and non-measurable. For example, Goal 2, Objective 1 reads:
"By 2012 or sooner, reduce air emissions as much as possible toward
attainment of respective national ambient air quality standards, and
reduce exposure in the border region...." We also found objective that
include wording "extend efforts." The wording of these objectives
makes them neither clear nor measurable.
We also reviewed the goals and objectives provided in a draft planning
document and found that some subobjectives included wording such as
"expand current efforts." According to Border 2012 staff, subobjectives
that use the wording "expand current efforts" have already been
accomplished. If the Agency uses these objectives in final planning
documents, it will not be able to measure the successful completion of
projects when a desired endpoint is not clearly defined. OIA staff also
stated that they do not link all projects and subobjectives to a specific
measure. Also, grant results are not collected by Border 2012 staff and
applied to appropriate measures to show the accomplishment of
subobjectives and goals.
-------
08-P-0245
Outcome Measures Are Being Developed but Challenges
Remain
We found the program is output-oriented and project-driven; however,
outcome measures are under development.10 Many of the current
objectives and related measures are output-based. According to OIA
staff, the Border 2012 Program was initially developed as an output-
based program. While output measures are important and required, they
do not fully capture the impact of Border 2012. As an example, Table 3-
1 shows the program's current subobjectives in terms of potential current
measures for Goal 4:
Table 3-1: Border 2012 Goal 4 Measures by Type
Measure
Number of respiratory health indicators developed
that are comparable/compatible in both countries.
Number of gastrointestinal diseases tracked and
measured to determine changes in these diseases as
a result of water quality improvements in border
communities.
Harmonize a binational system for reporting acute
pesticide poisonings.
Number of farm workers trained on pesticide risks
and safe handling, including ways to minimize
exposure for families and children.
Number of distance learning post graduate degree
programs to support advanced training on
environmental health in conjunction with Pan
American Health Organization regional offices and
academic institutions established.
Number of health care providers receiving
environmental health training for pesticides and
water.
Type
Output
X
X
X
X
X
X
Outcome
Source: OIG analysis of Border 2012 Program documents.
Although Border 2012 is measuring its progress using output measures,
we found that the program is attempting to move toward using more
outcome measures. According to EPA staff, the ability to measure and
report environmental outcomes for Border 2012 is hindered by the
difficulties encountered in sharing data between the United States and
Mexico. Border 2012 created the Indicator Taskforce in 2003 to measure
results, address the program's need for indicators, and serve as a bridge
of data between the two nations.
10
Output measures are defined as quantitative or qualitative measures of activities, work products, or
actions. Outcome measures encompass the knowledge, behavior, or conditions that result from program
activities and are needed to achieve the desired objectives (example: improved human health or
environmental conditions).
-------
08-P-0245
The Taskforce is developing a prototype of an online system known as
the Border Environmental Gateway that will be designed to facilitate
updating Border 2012's indicators and progress. The gateway will be a
Web-oriented information site with environmental information and
indicators for Border 2012. According to EPA staff, the site is not on-
line due to budget constraints, but could still be used as an internal portal
for indicator data.
The Border 2012 Program also faces challenges in collecting baseline
data, and some data gaps exist. Within the Border 2012 Program are
several objectives where baseline information is needed. For instance,
the State of the Border Region Report11 listed data for pesticide use in the
Border region "as difficult to collect and often lacking due to reporting
practices." According to several EPA staff, obtaining water quality data
is a major challenge for the program. One challenge to obtaining this
information is the difficulty in gaining access to the Mexico side of the
border. Skepticism from Mexico exists on how the data would be used
and how they would be interpreted. Staff also cited problems with
tracking the number of people served by the program as a measure under
Border 2012. While OIA wants to have measures across the entire border
region, local indicators are desired as well. Using local indicators would
make it easier to show differences on a smaller scale. This would also
allow localities to track their baselines and measures their impact on a
local scale.
Systematic Planning for Program Goals Is Lacking
While Border 2012 has identified six goals and related objectives, the
program does not have a comprehensive strategy in place to address how
these objectives will be met based on the resources available.12 The
priorities, goals, and objectives for the program are currently contained
in separate plans and documents, including the regions' RFPs, Region
9's annual operating plans, and the national coordinators' communiques.
We found the program lacks an overall roadmap or implementation plan
that links resources, activities, and intended outcomes. While Border
2012 has reported accomplishments in several of the program's goals, it
is not clear how these grants' results are integrated in the overall Border
2012 framework document. Moreover, it is not consistently articulated
who is accountable for overseeing individual objectives and monitoring
results.
1 :U. S. EPA, State of the Border Region Report, Border 2012: U.S. -Mexico Environmental Program
Report, April 2006, EPA-600-R-06-015.
12Appendices E and F present logic models that illustrate the linkages between program resources, their
use, and expected outcomes.
10
-------
08-P-0245
We found that some projects are still being conducted even though the
sub objective under which they were located may have been declared
accomplished. The program has recently conducted a mid-term
assessment in which it reviewed the progress made towards meeting the
program's goals and objectives. However, at the time of the mid-course
review, program staff were considering funding priorities that were
already listed as accomplished.
As depicted in Figure 3.1, the Border 2012 budget has decreased over the
last few years. Considering the program's declining budget, planning is
vital to ensure successful implementation of the Border 2012 Program's
agenda.
Figure 3-1: Border 2012 Budget
FYO9
FYO8
Fiscal Year
FYO7
FYO6
Borde
I
r 2O12 i
3udge1
o 1,000 2,000 3,000 4,000 5,000
Dollars in the thousands
6,000 7,000
Source: OIG developed chart based on data provided by Border 2012 staff.
Conclusion
We found the Border 2012 Program's organizational structure allows for
a collaborative relationship at the U.S.-Mexico border. The structure
allows the program to address environmental issues unique to the border
region. However, management oversight is lacking, regarding program
progress towards meeting goals and objectives. It is unclear whether the
Border 2012 goals have been achieved. Furthermore, project and
program results are not documented.
Without a strategic plan that links resources, activities, and outcomes, or
a process to assess progress towards meeting the intended outcomes, the
program can not ensure the activities will address the critical needs of
the border region. The strategic plan should include a needs assessment
for the border area, strategies for addressing any impediments, and a
11
-------
08-P-0245
statement of measurable goals with intermittent milestones so that
progress can be periodically assessed.
Recommendations
To ensure that the Border 2012 Program improves planning,
measurement, and accountability to effectively demonstrate program
performance, the Assistant Administrator for the Office of International
Affairs should:
3-1 Develop a strategic plan for the Border 2012 Program that
describes how the program will achieve desired results. The plan
should include the following components:
A national set of goals, objectives, and measures.
A list of internal measures used to gauge project and
program success.
A logic model, or other similar document, that accurately
reflects outputs and short-, intermediate-, and long-term
outcomes of the program.
A description of how each component of the Border 2012
Program, including grants, collaborations, and
partnerships, directly contributes to program outcomes.
3-2 Develop guidance that outlines roles and responsibilities
regarding how the Border 2012 Program (a) accomplishes each
program goal, objective, and subobjective; (b) obtains and
maintains supporting documentation for accomplished measures;
(c) develops and monitors criteria for determining what
constitutes their successful completion; and (d) assures quality of
data provided by grantees.
3-3 Develop and utilize effective performance measures that are
quantifiable and measurable, particularly human health
indicators, to track and report project program outcomes.
Agency Comments and OIG Evaluation
The Agency concurred with our recommendations. The Agency agreed
to develop a strategic plan and detailed program guidance by December
2009. Additionally, the Agency agreed to develop and utilize
quantifiable and measurable performance measures by December 2009.
The Agency's written response, as well as our evaluation of Agency
comments, is presented in Appendix G. The Agency's complete written
response is presented in Appendix H.
12
-------
08-P-0245
The OIG has incorporated technical corrections and clarifications
requested by EPA as appropriate.
13
-------
08-P-0245
Status of Recommendations and
Potential Monetary Benefits
POTENTIAL MONETARY
RECOMMENDATIONS BENEFITS (in $OOOs)2
Planned
Rec. Page Completion Claimed Agreed To
No. No. Subject Status1 Action Official Date Amount Amount
3-1 12 Develop a strategic plan for the Border 2012 0 Assistant Administrator, 12/09
Program that describes how the program will Office of International Affairs
achieve desired results. The plan should include
the following components:
A national set of goals, objectives, and
measures.
A list of internal measures used to gauge
project and program success.
A logic model, or other similar document, that
accurately reflects outputs and short-,
intermediate-, and long-term outcomes of the
program.
A description of how each component of the
Border 2012 Program, including grants,
collaborations, and partnerships, directly
contributes to program outcomes.
3-2 12 Develop guidance that outlines roles and 0 Assistant Administrator, 12/09
responsibilities regarding how the Border 2012 Office of International Affairs
Program (a) accomplishes each program goal,
objective, and subobjective; (b) obtains and
maintains supporting documentation for
accomplished measures; (c) develops and
monitors criteria for determining what constitutes
their successful completion; and (d) assures quality
of data provided by grantees.
3-3 12 Develop and utilize effective performance 0 Assistant Administrator, 12/09
measures that are quantifiable and measurable, Office of International Affairs
particularly human health indicators to track and
report project program outcomes.
1 0 = recommendation is open with agreed-to corrective actions pending
C = recommendation is closed with all agreed-to actions completed
U = recommendation is undecided with resolution efforts in progress
2 Identification of potential monetary benefits was not an objective of this evaluation.
14
-------
08-P-0245
Appendix A
Evolution of EPA's U.S.-Mexico Border Program
In 1983, the Federal governments of the United States and Mexico signed the La Paz
Agreement to protect, improve, and conserve the environment along the U.S.-Mexico
border. In the agreement, the border region was defined as the area within 100 km
(about 62.5 miles) on either side of the border separating the two countries. Six
workgroups of importance to the two countries were initiated under the La Paz
Agreement: air, water, hazardous and solid wastes, pollution prevention, contingency
planning and emergency response, and enforcement. The La Paz Agreement called for
establishing at least one annual meeting between the two countries addressing
environmental issues addressed. The agreement required both the United States and
Mexico, to the fullest extent practical, to adopt the appropriate measures to prevent,
reduce, and eliminate sources of pollution in the respective territory which affect the
border area of the other and to cooperate in solving environmental problems of mutual
concern in the border area.
In 1992, the environmental authorities of the United States and Mexico released the
Integrated Border Environmental Plan (IBEP). This plan carried out its activities
through the six workgroups identified within the La Paz Agreement. According to
EPA, EPA was criticized for its lack of public involvement in developing the IBEP and
not being adequately attentive to natural resources and environmental health issues.
This concern led to the next phase of binational cooperation, the Border XXI Program.
Border XXI, announced in December 1996, was founded on the principles of ensuring
public involvement, decentralization, and interagency cooperation. It also added three
new workgroups to the original six identified under the La Paz agreement:
environmental health, natural resources, and environmental information resources.
At the conclusion of Border XXI, a new strategic list of goals was formulated which
officially became Border 2012 through a series of meetings and community forums
with the State and Federal partners from countries, the U.S. tribal governments, and
community stakeholders.
15
-------
08-P-0245
Appendix B
Border 2012 Objectives for Goals 3 and 4
Table B-1: Goals 3 and 4 Objectives.
Goal
Goal 3:
Reduce
land
contamination
Goal 4:
Improve
environmental
health
Objective
Objective 1
Objective 2
Objective 3
Objective 4
Objective 1
Objective 2
Objective 3A
Objective 3B
Objective 4A
Objective 4B
By 2004, identify needs and develop an action plan to improve
institutional & infrastructure capacity for waste management and
pollution prevention as they pertain to hazardous and solid
waste & toxic substances along the U.S. -Mexico border.
Starting in 2005, the plan will be implemented and conducted by
2012.
By 2004, evaluate the hazardous waste tracking system in the
United States and Mexico. During 2006, develop and
consolidate the link between both tracking systems.
By 2010, clean up three of the largest sites that contain
abandoned waste tires in the U.S. -Mexico border region, based
on policies and programs developed in partnership with local
governments.
By 2004, develop a binational policy of clean-up and restoration
resulting in the productive use of abandoned sites contaminated
with hazardous waste or materials, along the length of the
border, in accordance with the laws of each country. By 2007,
apply this policy at least once in each of the four geographic
regions.
By 2006, evaluate various measures of respiratory health in
children that might be tracked to assess changes that may result
from actions to improve air quality in border communities.
By 2006, evaluate various measures of gastrointestinal illness
that might be tracked to assess changes that may result from
actions to improve water quality in border communities.
By 2006, complete an assessment and pilot program that
explores the feasibility of harmonizing a binational system for
reporting acute pesticide poisonings.
By 2007, reduce pesticide exposure by training 36,000 farm
workers on pesticide risks and safe handling, including ways to
minimize exposure for families and children.
By 2006, establish a distance learning post graduate degree
program to support advanced training on environmental health
in conjunction with Pan American Health Organizational regional
offices and academic institutions.
By 2004, extend current efforts in binational environmental
health training for 100 health care providers each for pesticides
and water.
Source: Border 2012 framework document.
16
-------
08-P-0245
Appendix C
Detailed Scope and Methodology
To determine how Border 2012 met its programmatic goals, we examined the
program's organizational and management structure. We assessed how various levels
within these structures work together to accomplish program goals. We interviewed the
U.S. national coordinator, as well as staff from the policy forums, regional workgroups,
and task forces. We examined the planning and goal structure of the program. This
included a review of the EPA Strategic Plan, the program's framework document and
goals, and the Region 9's operating plans. We also reviewed Border 2012 background,
program management documents, indicator documents, as well as applicable EPA
budget and performance documents. In reviewing the organizational structure, we
developed program logic model for Goals 3 and 4 with input from Border 2012 staff.
(See Appendices D and E.) We also reviewed applicable policies, procedures, and
practices.
To examine how the program achieves and measures results at the ground level, we
selected a sample of grant projects identified by Border 2012 as accomplishing
objectives. While we assessed all of the Border 2012 goals and objectives in regard to
program planning, we chose to limit our case studies to two of the six goals (Goals 3
and 4) due to public data collection limitations. We analyzed a randomly selected
sample of six Goal 3 and Goal 4 projects to determine if they achieved their goals, and
how this achievement was measured and reported.
To have a representative sample, the projects included those that were completed, past
due, and/or ongoing. In addition, these projects were deemed to represent the larger
selection of projects within all six goals of the program. We developed and
administered a questionnaire to the project managers for the six grants we reviewed.
We met with grant recipients and EPA project officers to further our understanding of
the Border 2012 Program and its results over time.
To examine how program accomplishments are assessed and documented, we met with
Border 2012 staff and analyzed the supporting documentation for two objectives with
quantifiable measures (Goal 4, Objective 3B and Goal 6, Objective 1). We
supplemented our analysis by meeting with Headquarters and regional EPA personnel,
as well as recipients of grants under the Border 2012 Program.
17
-------
08-P-0245
Appendix D
Border 2012 Guiding Principles
Table D-1: Border 2012 Guiding Principles.
Reduce the highest public health risks, and preserve and restore the natural
environment.
Adopt a bottom-up approach for setting priorities and making decisions through
partnerships with State, local, and U.S. tribal governments.
Address disproportionate environmental impacts in border communities.
Improve stakeholder participation and ensure broad-based representation from
the environmental, public health, and other relevant sectors.
Foster transparency, public participation, and open dialogue through provision of
accessible, accurate, and timely information.
Strengthen capacity of local community residents and other stakeholders to
manage environmental and environmentally-related public health issues.
Achieve concrete, measurable results while maintaining a long-term vision.
Measure program progress through development of environmental and public
health-based indicators.
The U.S. recognizes that U.S. tribes are separate sovereign governments, and
that equity issues impacting tribal governments must be addressed in the U.S.
on a government-to-government basis.
Mexico recognizes the historical debt it has with its indigenous peoples.
Therefore, appropriate measures will be considered to address its specific
concerns, as well as to protect and preserve its cultural integrity within the
broader environmental purposes of this program.
Source: Border 2012 framework document.
18
-------
08-P-0245
Figure E-1: Logic Model of Border 2012 Goal 3: Reduce Land Contamination
Appendix E
Short Term
Outcomes
Increased awareness
about TP development
Regional workgroups
Improved coordination
with local communities
reporting and revie
#$c/aptire mgmt.
ments produced
Increased management
capacity in Mexico
# Tire management
regulations deyetf$ped
# Training sessions
held
TP action planN ^/\
development ^
State
Governments
#Training videos
distributed
Evaluation of US/Mexico
hazardous waste trackini
system
Local
governments
Increased effectiveness/
coordination between US
and Mexico
Compatible databases
Community
members
Hazardous waste
site identification
List of priority sites
identified in border area
Policy
development
Environmental health
database
Health data
collection
Intermediate
Outcomes
Increased air quality due
to decreased TP fires
Decreased mosquito-
borne illness due to TPs
Long Term
Outcomes
Improved environment
and public health
19
-------
08-P-0245
Figure F-1: Logic Model of Border 2012 Goal 4: Improve Environmental Health
Appendix F
Outputs
# Air measures
developed
># Water measures
developed
# Farm workers
trained on pesticide
risks and safe hand-
ling, including ways to
minimize exposure for
families and children
# Health care provid-
ers trained in environ-
mental health issues,
pesticides and water
# "Distance-learning,"
post-graduate degree
program to support
advanced training on
env ronmental health in
conjunction with PAHO
# Training sessions
hold
# Models and sensing
projects evaluated for
future use/aoDlicabilitv
# Documents shared
via contacts made in
health communitv
# Environmental health
curricula devplonpd
Environmental health
database
Short Term
Outcomes
Baseline development
Harmonized system
between the US and
Mexico for reporting acute
pesticide poisonings
Improved coordination
with local communities
Increased awareness of
environmental health
issues among health care
providers to make proper
diannoses
Increased management
capacity in Mexico
Decreased concentra-
tions of pesticides - chil-
dren's urinary metabolites
Reduced potential for
harmful exposure to
pesticides by farmworkers
Increased effectiveness/
coordination between US
and Mexico
Intermediate
Outcomes
Long Term
Outcomes
Reductions in air-related
respiratory diseases
Reductions in water-
borne illnesses
Improved environment
and public health
Reductions in pesticide-
related poisonings
20
-------
08-P-0245
Appendix G
Agency Comments and OIG Evaluation
General Comments
The OIG Report fails to acknowledge significant achievements along the U.S.-
Mexico Border. The Noteworthy Achievements section acknowledges only one aspect
of Border 2012 accomplishments - the binational convening mechanism that brings
stakeholders together to address environmental issues along the border. This section,
however, fails to recognize the real and concrete achievements of the Border 2012
Program and dismisses actual results as mere "claims." This is apparently due to an OIG
expectation that in order to be valid, the work of Border 2012 project sponsors and
program partners must meet data requirements and be "collected or obtained" following a
"documented process" and centrally managed by EPA.
Real and substantive accomplishments have been achieved over the past five years along
the U.S.-Mexico Border. These accomplishments have resulted in significant
environmental and public health improvements for many communities along the border
as reflected in our mid-term and indicators reports. We recommend that OIG consider
including some of the key accomplishments listed later in this comment paper.
OIG Response: We could not verify many of the claimed accomplishments. The
achievements reported are not all inclusive or definitive of the program. We
advised the Agency that we would include additional selected accomplishments in the
report if it could provide substantiating evidence. No support for Agency
representations was provided. Noteworthy accomplishments are background
information and have no impact on the programmatic issues described in the report.
The OIG Report fails to recognize the complex dynamic within which the Border
2012 Program operates. The draft OIG report and recommendations reflect an
unrealistic expectation or incomplete understanding of the complex binational framework
and relationships that must be worked through to accomplish measurable results along the
U.S.-Mexico Border. These unacknowledged relationships significantly impact strategic
planning, reporting, and accountability in a way that the OIG report fails to recognize.
OIG Response: We believe that Chapters 1 and 2 adequately describe the
program and the conditions under which it operates. Appendix A further describes the
evolution of and requirements for the Border 2012 Program. The complexity of the
program is further reason why it is important to develop a strategic plan that fully
describes the components of the program including the collaborations and partners that
directly contribute to program outcomes.
21
-------
08-P-0245
In the Border 2012: U.S. - Mexico Environmental Program document, the Mission
Statement reads:
"As a result of the partnership among federal, state, local
governments in the United States and Mexico, and with U.S. border
tribes, the mission of the Border 2012 program is: To protect the
environment and public health in the U.S. - Mexico border region,
consistent with the principles of sustainable development."
The OIG report portrays the Border 2012 Program as if it were "EPA's Program"
whereby projects are primarily implemented through EPA grants. In reality, the Border
2012 Program was greatly informed and shaped by the 10 Border States. The failure to
reflect this understanding is a fundamental pitfall in OIG's evaluation since it overlooks
the substantial achievements in both projects and capacity building attributable to the
efforts and resources of the Border 2012 partners.
OIG Response: We recognize that the Border 2012 Program is a bi-national
program that relies on its internal and external partners. However, the partnership
elements of the program do not make the Agency less accountable for accomplishing its
roles and responsibilities. We revised the report to more explicitly acknowledge the
program partners.
Based on the OIG's premise that the Border 2012 Program belongs primarily to EPA, the
OIG apparently determined that it could finalize the report based on the exclusive review
by EPA. However, the notion that this draft document with recommendations should be
finalized without giving the Border 2012 partners outside EPA an opportunity for review
and comment, reflects a critical misunderstanding of the program dynamics or,
alternatively, is dismissive of a prime founding and operating principle. This is
problematic if the OIG expects the program partners to embrace the recommendations
and be partners in implementing the changes. The recommended "fixes" presume a top-
down, EPA-centric, process to "improve" what is essentially a bottom-up partnership-
based program.
OIG Response: We evaluated EPA's responsibility for the program in the context
of the large number of stakeholders involved in it. In our scope and methodology, we
explain that our review focuses on management controls and organizational structure as
implemented and funded on the U.S. side of the border. OIG's recommendations are
directly targeted at EPA, not at program partners.
The OIG Report fails to recognize the rigorous management and accountability
mechanisms already in place for the Border 2012 Program. The OIG programmatic
evaluation (per the scope and methodology referenced in Appendix C) focused on a
limited review of six (of over eighty) grants, and focused on only two of the six Border
2012 goals. Yet, throughout the document, there are references, assumptions, and
conclusions that have been applied to the entire program based on this very limited
sampling of projects. As an example, one of the OIG findings states that there is a lack of
management oversight regarding progress towards meeting goals and objectives.
22
-------
08-P-0245
However, because of the limited scope previously mentioned, the OIG appears to have
not been aware of the rigorous internal management and accountability system for
tracking progress on various Border 2012 goals and objectives.
OIG Response: We reported on the aspects of Border 2012 related to our scope.
As explained in Appendix C, we assessed how various levels within the program's
organizational and management structure work together to accomplish program goals. In
conducting this evaluation, we interviewed the national coordinator; staff from the policy
forum, regional workgroups, and task forces; and examined the planning and goal
structure of the program. The review also included an examination of EPA's strategic
plan, and the program's framework document and goals. Our findings regarding the
necessity for overarching planning and results measurement in the program result from
the entirety of our evaluation.
For example, in coordination with OIA:
The Region 6 and Region 9 Regional Administrators and National Program Offices
hold quarterly conference calls with EPA's Deputy Administrator, Marcus Peacock,
to discuss progress on a number of Border 2012 measures. These calls are partially
designed for program accountability, but more importantly, they are utilized to
showcase best practices and better ways of doing our work along the border, based on
actual field examples and experience. These calls are also broadcast throughout EPA
via web-based media so that other parts of EPA may also benefit from the innovations
and best practice discussions.
The EPA National Program Offices, Region 6 and 9, the 10 Border States, and 26
U.S. Tribes each devote staff and resources to accomplish the goals and objectives of
Border 2012, which are overseen by their respective senior level leadership.
Some of the Border 2012 measures (in Goal 1 and Goal 3) are EPA Strategic Plan
Measures and OMB PART Measures that are closely monitored and tracked by the
Regions, NPMs, OMB, and OCFO; and,
Each year, Region 9 develops a comprehensive Border Operating Plan that highlights
annual commitments to achieve the Border 2012 goals and objectives. This strategic
plan is shared with the NPMs, OIA, Region 6, and State partners so that there is a
common understanding of the commitments and intended results.
As the National Coordinator, OIA works closely with Program partners to coordinate
the above efforts to better manage the program's results and efforts.
OIG Response: Meetings, phone calls, and assigning staff do not constitute a
system of accountability. Our evaluation found deficiencies in the validity and
reliability of the support for the results discussed. Additionally, despite OIG requests,
no documentation of the results from the quarterly conference calls (such as action items
or follow-up) was provided in the course of the evaluation. Finally, the report does
recognize the Region 9 operating plan, in contrast to Region 6 and the overall program.
The OIG Report fails to recognize the strategic framework already undertaken by
Border 2012. The OIG assumes that because every goal is not derived from the kind of
23
-------
08-P-0245
"logic model" proposed in Figure E-l of the report that the projects and efforts
themselves are not being implemented under a strategic framework and that no effort is
made to establish links between strategic objectives, activities, outputs, and outcomes.
This is incorrect.
The Border 2012 program aims to protect the environment and public health in two
countries with a common border and through a collaborative partnership of federal, state,
and local governments, and the U.S. border tribes. In this binational, multi-level, and
strategic framework (focused on the 6 goals and 23 objectives), which is embraced and
accepted by all the partners, we collectively strive to:
Require the best practicable measurements and outputs or outcomes in the
projects that receive grants, recognizing that there are different laws, ordinances,
programs, and reporting requirements implemented across the region;
Request the best practicable measurements and outputs or outcomes from
voluntary programs, recognizing their voluntary nature;
Continue to report Border 2012 program accomplishments through updates of the
Implementation Report and to develop best practicable measures through the
Indicators Task Force and updates of the Border 2012 Indicators Report.
OIG Response: We found that the program lacks an implementation plan to ensure
results. The report appropriately characterizes Border 2012 planning, management, and
measurement. No description exists of the implementation framework or connectivity
between activities and results. The logic models, developed with OIA's participation,
simply present what is being done.
The OIG Report references outdated draft working documents. We noticed that
throughout the document (but especially in Chapter 3), there are references to the Border
2012 mid-course refinement draft working document that EPA shared with OIG as a
courtesy copy. This document is a working draft and is currently very different from the
original copy that was shared with OIG. The conclusions reached by OIG are thus based
on outdated information - we request that all references regarding this mid-course
refinement document be removed from the report - it is still a working draft. Until the
midcourse refinement effort is completed, the Border 2012: US-Mexico Environmental
Program Document and its corresponding guidelines continue to be the primary working
framework
OIG Response: OIG has access to all Government records. All data and documents
analyzed and reviewed by the OIG team were current as of the end of fieldwork. Where
the refinement document is cited, we refer to it as a draft document. Moreover, the OIG
would expect that the final Agency document would identify and plan to correct the
management deficiencies identified in the interim document.
24
-------
08-P-0245
Specific Comments (Section 1)
Page 2, Noteworthy Achievements: We appreciate the OIG's acknowledgement of the
Border 2012 Program's success in convening various border stakeholders. However,
OIG fails to recognize the significant and real environmental improvements achieved
since the Program's inception. There are many examples that could have been referenced
in a section that purports to address achievements. For example, citizens in El Paso who
once breathed air that violated standards for ozone and carbon monoxide now breathe air
that meets those standards; 100,000 more homes are now served by wastewater service;
more than 3 million abandoned tires have been eliminated along the border to reduce the
threat of uncontrollable fires or disease vectors; household hazardous waste and waste oil
is better managed in Nuevo Laredo, Nogales, and other border cities so that their
wastewater treatment plants experiences fewer upsets; the 15 sister cities along the border
now have active emergency preparedness and response plans so that emergency
responders on both sides of the border are able to communicate with each other and
cooperate on a response (where once they could not); over 2,500 tons of hazardous waste
was removed from an abandoned waste site in Tijuana, eliminating risk to nearby
communities, and so many more. These examples are referenced in the Mid-Term report
and the indicators report.
OIG Response: We believe that we have appropriately the characterized Border
2012 Program in the report. During the evaluation we requested that OIA provide
additional accomplishments in the form of narrative and support and did not receive
these items.
In addition, the OIG report fails to accept the validity of direct observations by program
partners, including those documented by photographs that document waste tires which
once existed no longer do, having been removed for use as fuel in a cement kiln. The
value of removing tire piles in the United States is broadly accepted; an acceptance
derived from years of experience in Superfund emergency response and other programs
in dealing with tire fires and disease control. Yet, absent additional layers of strategic
planning, accountability, and explicit "logic models13," the OIG report does not seem
willing to acknowledge the implicit strategic value of tire removal within the goal of
reducing "land contamination" (Goal 3) or the accepted, but difficult to quantify,
outcome and outputs from such efforts, notwithstanding the fact that such activities and
the performance measurement associated therewith were undertaken in a manner
consistent with long standing non-border U.S. tire management programs. In short, the
report scarcely pays attention to "Noteworthy Achievements" on page 2, does not
acknowledge the program partners and their work, and is fundamentally flawed in this
respect.
13 Interestingly, with respect to waste tires, Figure E-l was apparently developed based on what we are
already doing, rather than presenting a new model that can be used to enhance strategic planning,
accountability, and outcome measures.
25
-------
08-P-0245
OIG Response: The OIG did not report or conclude that direct observation by
program partners (documented by photographs) is unacceptable evidence in support of
accomplishments. Although we advised program staff that if examples of
accomplishments were provided along with support, we could include those success
stories in the report. We were not provided such pictures or records of direct
observations in support of accomplishments and therefore could not report them as
noteworthy accomplishments or incorporate them into the final report.
Specific Comments (Chapter 3)
Page 7, Paragraph 1: The report states that "the program lacks baseline data,
quantifiable measures or outcome measures which could verify the program's
effectiveness". In addition, the report states that "coupled with the decentralized
structure of the program, the result is a lack of accountability and oversight". We
disagree with these two statements. First, the program does maintain baseline data for
over half of its goals and obj ectives and is in the process of obtaining data for the
remainder (for example, the Border 2012 Program has completed a comprehensive
inventory of tire piles along the border to serve as the baseline for future efforts; the
baseline for the number of homes connected to drinking water was established at the
beginning of the program so that ongoing efforts could be measured against this
baseline). Second, as explained in our general comments, the Border 2012 program
incorporates mechanisms to enhance accountability and oversight. For example, each
year Regional EPA staffs prepare annual Operating Plans that outline the annual priorities
and implementation strategy to accomplish the various goals and objectives of Border
2012 (these are reviewed and approved by the Deputy Regional Administrator). In
addition, the Regions issue the annual RFP that seeks to further advance completion of
the goals and objectives of Border 2012 via projects that include measurable results and
that leverage external resources (this process is closely monitored and overseen by the
Deputy Regional Administrator). Each year, the National Coordinators convene the
program partners to oversee and evaluate progress on the goals and objectives and to
issue guidance (as appropriate) for areas that may require additional support or effort.
They also issue a joint communique that include significant results for the year and new
annual priorities for accomplishing the various goals and objectives.
Finally, there is a very rigorous oversight process to document and monitor each and
every project that receives EPA grant funding. The oversight includes, among other
things, reviewing quarterly progress reports, monitoring of project budgets, and review
and approval of final project reports. This process is required by EPA grants policies and
procedures for any grant that the agency issues (not just for border grants) and is an on-
going practice for the U.S.-Mexico Border Program.
We request that OIG consider deleting or modifying the second, fifth, and sixth sentences
of this paragraph to reflect the comments above.
26
-------
08-P-0245
OIG Response: Our report demonstrates that the Agency needs to strengthen
weaknesses in performance measurement, oversight, and documentation of Border 2012
outcomes and results. We recognize the oversight process built into EPA's grants funding
for the program. However, OIA does not ensure that the results achieved by the projects
are linked to overall programmatic goals, nor do they have a system in place to feed any
project results back into the program as a whole. Border 2012 lacks mechanisms and
accountability for linking projects and program results.
Page 7, second paragraph, sentence 7: The report states that "we found that border
2012 program does not have a process in place to review results submitted by the
grantees". This statement is incorrect. As previously stated, per EPA grants policies and
regulations, each EPA Grant Project Officer is responsible for managing and monitoring
each and every project that is funded using EPA funds (including those issued for border
projects). EPA grants management guidelines and regulations require that project
sponsors certify that they have accomplished the intended results, and EPA makes
reasonable efforts to verify these results via site visits and other means.
OIG Response: We do not disagree that an EPA grants management process is in
place for reviewing these results. Our findings indicate that Border 2012 as a whole lacks
a mechanism to review, verify, document, and incorporate these results into the program.
Page 8, paragraph 4: This paragraph refers to the mid-course refinement document,
which is currently a working draft that is expected to be finalized by September, 2008.
EPA provided OIG with an early working draft to show that we are working on
improving the measurability of the various goals and objectives. The conclusions
reached in this paragraph are based on outdated information. We request that OIG
consider deleting this paragraph from the report.
OIG Response: We recognize that the refinement is a living document and
therefore refer to it as draft in the report. All data and documents analyzed and reviewed
bv the OIG team were current as of the end of fieldwork.
Page 10, paragraph 3: The report states that "the program does not have a
comprehensive strategy in place to address how these objectives will be met based on the
resources available". In addition, the report states that "the program lacks an overall
roadmap or implementation plan that links resources...." We disagree with these two
statements. The program framework, guiding principles, goals and objectives, and
operational guidelines are the collective strategy and roadmap for addressing the
completion of the objectives. In addition, as previously stated, each year Regional EPA
staff prepares annual Operating Plans that outline the annual priorities and
implementation strategy to accomplish the various goals and objectives of Border 2012.
In addition, the Regions issue the annual RFP that seeks to further advance completion of
the goals and objectives of Border 2012 via projects that includes measurable results and
that leverage external resources. And finally, each year, the National Coordinators issue
a joint communique that includes annual priorities for accomplishing the various goals
27
-------
08-P-0245
and objectives. All of the above actions are carried out in coordination with our state and
tribal partners and consider the existing resource realities and constraints. We request
that OIG consider deleting sentences 1 and 3 from the report.
OIG Response: Our report demonstrates that Border 2012 lacks a comprehensive
strategy or implementation plan to draw appropriate connections and conclusions from the
program. Our analysis included the documents cited here, including the operating plans
completed only by Region 9. We were informed verbally that both regions operating
under an annual operating plan. However, when we requested the annual plan for Region
6, it provided us with the Agency's strategic plan, the RFPs, and the Border 2012
framework document. Region 6 was unable to provide an operating plan.
Page 11, paragraph 1: This paragraph refers to the mid-course refinement document,
which is currently a working draft that is expected to be finalized by September, 2008.
We provided OIG with this courtesy working draft to show that we are working on
improving the measurability of the various goals and objectives. The OIG report refers to
internal, place-holder language that is no longer relevant since the document has been
revised based on comments received from our Border 2012 partners. The conclusions
reached in this paragraph are based on outdated information. We request that OIG
consider deleting this paragraph from the report.
OIG Response: We recognize that the refinement is a living document and
therefore refer to it as draft in the report. All data and documents analyzed and reviewed
by the OIG team were current as of the end of fieldwork. Moreover, we did not cite
conclusions from the report, only a statement of fact that measures need to be specific.
Response to Recommendations
3-1 Develop a strategic plan for the Border 2012 program that describes how the
program will achieve desired results. The plan should include the following components:
o A national set of goals, objectives, and measures.
o A list of internal measures used to gauge project and program success.
o A logic model, or other similar document, that accurately reflects outputs and short-,
intermediate-, and long-term outcomes of the program.
o A description of how each component of the Border 2012 program, including grants,
collaborations, and partnerships, directly contributes to program outcomes.
Action Official: Assistant Administrator for International Affairs
Response: We accept the recommendation.
Milestones/Dates: Completion by December 2009.
OIG Response: The Agency has accepted this recommendation.
28
-------
08-P-0245
3-2 Develop guidance that outlines roles and responsibilities regarding how the
Border 2012 program (a) accomplishes each program goal, objective, and subobjective;
(b) obtains and maintains supporting documentation for accomplished measures; (c)
develops and monitors criteria for determining what constitutes their successful
completion; and (d) assures quality of data provided by grantees.
Action Official: Assistant Administrator for International Affairs
Response: We accept the recommendation.
Milestones/Dates: Completion by December 2009.
OIG Response: The Agency has accepted this recommendation.
3-3 Develop and utilize effective performance measures that are quantifiable and
measurable, particularly human health indicators to track and report project program
outcomes.
Action Official: Assistant Administrator for International Affairs
Response: We accept the recommendation.
Milestones/Dates: Completion by December 2009.
OIG Response: The Agency has accepted this recommendation.
29
-------
08-P-0245
Appendix H
Agency Comments
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
OFFICE OF
INTERNATIONAL AFFAIRS
July 30,2008
MEMORANDUM
SUBJECT: Response to Draft Evaluation Report: Border 2012 Program Needs to
Improve Program Management to Ensure Results
Assignment No. 2007-903
FROM: Scott Fulton, Deputy Assistant Administrator
Office of International Affairs /S/
TO: Jeffrey Harris
Director for Program Evaluation, Special Studies
We are providing a written response to the findings and recommendations on the above
mentioned draft report. The response raises significant concerns regarding the findings in
the draft report but indicates acceptance of each proposed recommendation. The
response also indicates planned completion dates for all recommendations.
If you or your staff have any questions regarding this response, please contact me at (202)
564-6600 or Lisa Almodovar at (202) 564-6401 or almodovar.lisa@epa.gov.
Cc: Laura Yoshii, DRA Region 9
Lawrence Starfield, DRA Region 6
30
-------
08-P-0245
Comments Prepared by OIA in coordination w/ Regions 6 and 9
Evaluation Report: Border 2012 Program Needs to Improve Program
Management to Ensure Results
July 30, 2008
General Comments
The OIG Report fails to acknowledge significant achievements along the U.S.-
Mexico Border. The Noteworthy Achievements section acknowledges only one aspect
of Border 2012 accomplishments - the binational convening mechanism that brings
stakeholders together to address environmental issues along the border. This section,
however, fails to recognize the real and concrete achievements of the Border 2012
Program and dismisses actual results as mere "claims." This is apparently due to an OIG
expectation that in order to be valid, the work of Border 2012 project sponsors and
program partners must meet data requirements and be "collected or obtained" following a
"documented process" and centrally managed by EPA.
Real and substantive accomplishments have been achieved over the past five years along
the U.S.-Mexico Border. These accomplishments have resulted in significant
environmental and public health improvements for many communities along the border
as reflected in our mid-term and indicators reports. We recommend that OIG consider
including some of the key accomplishments listed later in this comment paper.
The OIG Report fails to recognize the complex dynamic within which the Border
2012 Program operates. The draft OIG report and recommendations reflect an
unrealistic expectation or incomplete understanding of the complex binational framework
and relationships that must be worked through to accomplish measurable results along the
U.S.-Mexico Border. These unacknowledged relationships significantly impact strategic
planning, reporting, and accountability in a way that the OIG report fails to recognize.
In the Border 2012: U.S. - Mexico Environmental Program document, the Mission
Statement reads:
"As a result of the partnership among federal, state, local
governments in the United States and Mexico, and with U.S. border
tribes, the mission of the Border 2012 program is: To protect the
environment and public health in the U.S. - Mexico border region,
consistent with the principles of sustainable development."
The OIG report portrays the Border 2012 Program as if it were "EPA's Program"
whereby projects are primarily implemented through EPA grants. In reality, the Border
2012 Program was greatly informed and shaped by the 10 Border States. The failure to
reflect this understanding is a fundamental pitfall in OIGs evaluation since it overlooks
the substantial achievements in both projects and capacity building attributable to the
efforts and resources of the Border 2012 partners.
31
-------
08-P-0245
Based on the OIG's premise that the Border 2012 Program belongs primarily to EPA, the
OIG apparently determined that it could finalize the report based on the exclusive review
by EPA. However, the notion that this draft document with recommendations should be
finalized without giving the Border 2012 partners outside EPA an opportunity for review
and comment, reflects a critical misunderstanding of the program dynamics or,
alternatively, is dismissive of a prime founding and operating principle. This is
problematic if the OIG expects the program partners to embrace the recommendations
and be partners in implementing the changes. The recommended "fixes" presume a top-
down, EPA-centric, process to "improve" what is essentially a bottom-up partnership-
based program.
The OIG Report fails to recognize the rigorous management and accountability
mechanisms already in place for the Border 2012 Program. The OIG programmatic
evaluation (per the scope and methodology referenced in Appendix C) focused on a
limited review of six (of over eighty) grants, and focused on only two of the six Border
2012 goals. Yet, throughout the document, there are references, assumptions, and
conclusions that have been applied to the entire program based on this very limited
sampling of projects. As an example, one of the OIG findings states that there is a lack of
management oversight regarding progress towards meeting goals and objectives.
However, because of the limited scope previously mentioned, the OIG appears to have
not been aware of the rigorous internal management and accountability system for
tracking progress on various Border 2012 goals and objectives. For example, in
coordination with OIA:
The Region 6 and Region 9 Regional Administrators and National Program Offices
hold quarterly conference calls with EPA's Deputy Administrator, Marcus Peacock,
to discuss progress on a number of Border 2012 measures. These calls are partially
designed for program accountability, but more importantly, they are utilized to
showcase best practices and better ways of doing our work along the border, based on
actual field examples and experience. These calls are also broadcast throughout EPA
via web-based media so that other parts of EPA may also benefit from the innovations
and best practice discussions.
The EPA National Program Offices, Region 6 and 9, the 10 Border States, and 26
U.S. Tribes each devote staff and resources to accomplish the goals and objectives of
Border 2012, which are overseen by their respective senior level leadership.
Some of the Border 2012 measures (in Goal 1 and Goal 3) are EPA Strategic Plan
Measures and OMB PART Measures that are closely monitored and tracked by the
Regions, NPMs, OMB, and OCFO; and,
Each year, Region 9 develops a comprehensive Border Operating Plan that highlights
annual commitments to achieve the Border 2012 goals and objectives. This strategic
plan is shared with the NPMs, OIA, Region 6, and State partners so that there is a
common understanding of the commitments and intended results.
As the National Coordinator, OIA works closely with Program partners to coordinate
the above efforts to better manage the program's results and efforts.
The OIG Report fails to recognize the strategic framework already undertaken by
Border 2012. The OIG assumes that because every goal is not derived from the kind of
32
-------
08-P-0245
"logic model" proposed in Figure E-l of the report that the projects and efforts
themselves are not being implemented under a strategic framework and that no effort is
made to establish links between strategic objectives, activities, outputs, and outcomes.
This is incorrect.
The Border 2012 program aims to protect the environment and public health in two
countries with a common border and through a collaborative partnership of federal, state,
and local governments, and the U.S. border tribes. In this binational, multi-level, and
strategic framework (focused on the 6 goals and 23 objectives), which is embraced and
accepted by all the partners, we collectively strive to:
Require the best practicable measurements and outputs or outcomes in the
projects that receive grants, recognizing that there are different laws, ordinances,
programs, and reporting requirements implemented across the region;
Request the best practicable measurements and outputs or outcomes from
voluntary programs, recognizing their voluntary nature;
Continue to report Border 2012 program accomplishments through updates of the
Implementation Report and to develop best practicable measures through the
Indicators Task Force and updates of the Border 2012 Indicators Report.
The OIG Report references outdated draft working documents. We noticed that
throughout the document (but especially in Chapter 3), there are references to the Border
2012 mid-course refinement draft working document that EPA shared with OIG as a
courtesy copy. This document is a working draft and is currently very different from the
original copy that was shared with OIG. The conclusions reached by OIG are thus based
on outdated information - we request that all references regarding this mid-course
refinement document be removed from the report - it is still a working draft. Until the
midcourse refinement effort is completed, the Border 2012: US-Mexico Environmental
Program Document and its corresponding guidelines continue to be the primary working
framework.
Specific Comments (Section 1)
Page 2, Noteworthy Achievements: We appreciate the OIG's acknowledgement of the
Border 2012 Program's success in convening various border stakeholders. However,
OIG fails to recognize the significant and real environmental improvements achieved
since the Program's inception. There are many examples that could have been referenced
in a section that purports to address achievements. For example, citizens in El Paso who
once breathed air that violated standards for ozone and carbon monoxide now breathe air
that meets those standards; 100,000 more homes are now served by wastewater service;
more than 3 million abandoned tires have been eliminated along the border to reduce the
threat of uncontrollable fires or disease vectors; household hazardous waste and waste oil
is better managed in Nuevo Laredo, Nogales, and other border cities so that their
wastewater treatment plants experiences fewer upsets; the 15 sister cities along the border
now have active emergency preparedness and response plans so that emergency
responders on both sides of the border are able to communicate with each other and
cooperate on a response (where once they could not); over 2,500 tons of hazardous waste
33
-------
08-P-0245
was removed from an abandoned waste site in Tijuana, eliminating risk to nearby
communities, and so many more. These examples are referenced in the Mid-Term report
and the indicators report.
In addition, the OIG report fails to accept the validity of direct observations by program
partners, including those documented by photographs that document waste tires which
once existed no longer do, having been removed for use as fuel in a cement kiln. The
value of removing tire piles in the United States is broadly accepted; an acceptance
derived from years of experience in Superfund emergency response and other programs
in dealing with tire fires and disease control. Yet, absent additional layers of strategic
planning, accountability, and explicit "logic models14," the OIG report does not seem
willing to acknowledge the implicit strategic value of tire removal within the goal of
reducing "land contamination" (Goal 3) or the accepted, but difficult to quantify,
outcome and outputs from such efforts, notwithstanding the fact that such activities and
the performance measurement associated therewith were undertaken in a manner
consistent with long standing non-border U.S. tire management programs. In short, the
report scarcely pays attention to "Noteworthy Achievements" on page 2, does not
acknowledge the program partners and their work, and is fundamentally flawed in this
respect.
Specific Comments (Chapter 3)
Page 7, Paragraph 1: The report states that "the program lacks baseline data,
quantifiable measures or outcome measures which could verify the program's
effectiveness". In addition, the report states that "coupled with the decentralized
structure of the program, the result is a lack of accountability and oversight". We
disagree with these two statements. First, the program does maintain baseline data for
over half of its goals and obj ectives and is in the process of obtaining data for the
remainder (for example, the Border 2012 Program has completed a comprehensive
inventory of tire piles along the border to serve as the baseline for future efforts; the
baseline for the number of homes connected to drinking water was established at the
beginning of the program so that ongoing efforts could be measured against this
baseline). Second, as explained in our general comments, the Border 2012 program
incorporates mechanisms to enhance accountability and oversight. For example, each
year Regional EPA staffs prepare annual Operating Plans that outline the annual priorities
and implementation strategy to accomplish the various goals and objectives of Border
2012 (these are reviewed and approved by the Deputy Regional Administrator). In
addition, the Regions issue the annual RFP that seeks to further advance completion of
the goals and objectives of Border 2012 via projects that include measurable results and
that leverage external resources (this process is closely monitored and overseen by the
Deputy Regional Administrator). Each year, the National Coordinators convene the
program partners to oversee and evaluate progress on the goals and objectives and to
issue guidance (as appropriate) for areas that may require additional support or effort.
14 Interestingly, with respect to waste tires, Figure E-l was apparently developed based on what we are
already doing, rather than presenting a new model that can be used to enhance strategic planning,
accountability, and outcome measures.
34
-------
08-P-0245
They also issue a joint communique that include significant results for the year and new
annual priorities for accomplishing the various goals and objectives.
Finally, there is a very rigorous oversight process to document and monitor each and
every project that receives EPA grant funding. The oversight includes, among other
things, reviewing quarterly progress reports, monitoring of project budgets, and review
and approval of final project reports. This process is required by EPA grants policies and
procedures for any grant that the agency issues (not just for border grants) and is an on-
going practice for the U.S.-Mexico Border Program.
We request that OIG consider deleting or modifying the second, fifth, and sixth sentences
of this paragraph to reflect the comments above.
Page 7, second paragraph, sentence 7: The report states that "we found that border
2012 program does not have a process in place to review results submitted by the
grantees". This statement is incorrect. As previously stated, per EPA grants policies and
regulations, each EPA Grant Project Officer is responsible for managing and monitoring
each and every project that is funded using EPA funds (including those issued for border
projects). EPA grants management guidelines and regulations require that project
sponsors certify that they have accomplished the intended results, and EPA makes
reasonable efforts to verify these results via site visits and other means.
Page 8, paragraph 4: This paragraph refers to the mid-course refinement document,
which is currently a working draft that is expected to be finalized by September, 2008.
EPA provided OIG with an early working draft to show that we are working on
improving the measurability of the various goals and objectives. The conclusions
reached in this paragraph are based on outdated information. We request that OIG
consider deleting this paragraph from the report.
Page 10, paragraph 3: The report states that "the program does not have a
comprehensive strategy in place to address how these objectives will be met based on the
resources available". In addition, the report states that "the program lacks an overall
roadmap or implementation plan that links resources...." We disagree with these two
statements. The program framework, guiding principles, goals and objectives, and
operational guidelines are the collective strategy and roadmap for addressing the
completion of the objectives. In addition, as previously stated, each year Regional EPA
staff prepares annual Operating Plans that outline the annual priorities and
implementation strategy to accomplish the various goals and objectives of Border 2012.
In addition, the Regions issue the annual RFP that seeks to further advance completion of
the goals and objectives of Border 2012 via projects that includes measurable results and
that leverage external resources. And finally, each year, the National Coordinators issue
a joint communique that includes annual priorities for accomplishing the various goals
and objectives. All of the above actions are carried out in coordination with our state and
tribal partners and consider the existing resource realities and constraints. We request
that OIG consider deleting sentences 1 and 3 from the report.
Page 11, paragraph 1: This paragraph refers to the mid-course refinement document,
which is currently a working draft that is expected to be finalized by September, 2008.
35
-------
08-P-0245
We provided OIG with this courtesy working draft to show that we are working on
improving the measurability of the various goals and objectives. The OIG report refers to
internal, place-holder language that is no longer relevant since the document has been
revised based on comments received from our Border 2012 partners. The conclusions
reached in this paragraph are based on outdated information. We request that OIG
consider deleting this paragraph from the report.
Response to Recommendations
3-1 Develop a strategic plan for the Border 2012 program that describes how the
program will achieve desired results. The plan should include the following components:
o A national set of goals, objectives, and measures.
o A list of internal measures used to gauge project and program success.
o A logic model, or other similar document, that accurately reflects outputs and short-,
intermediate-, and long-term outcomes of the program.
o A description of how each component of the Border 2012 program, including grants,
collaborations, and partnerships, directly contributes to program outcomes.
Action Official: Assistant Administrator for International Affairs
Response: We accept the recommendation.
Milestones/Dates: Completion by December 2009.
3-2 Develop guidance that outlines roles and responsibilities regarding how the
Border 2012 program (a) accomplishes each program goal, objective, and subobjective;
(b) obtains and maintains supporting documentation for accomplished measures; (c)
develops and monitors criteria for determining what constitutes their successful
completion; and (d) assures quality of data provided by grantees.
Action Official: Assistant Administrator for International Affairs
Response: We accept the recommendation.
Milestones/Dates: Completion by December 2009.
3-3 Develop and utilize effective performance measures that are quantifiable and
measurable, particularly human health indicators to track and report project program
outcomes.
Action Official: Assistant Administrator for International Affairs
Response: We accept the recommendation.
Milestones/Dates: Completion by December 2009.
36
-------
08-P-0245
Appendix I
Distribution
Office of the Administrator
Principal Deputy Assistant Administrator, Office of International Affairs
Deputy Regional Administrator, Region 9
Deputy Regional Administrator, Region 6
Agency Follow-up Official (the CFO)
Agency Follow-up Coordinator
Office of General Counsel
Associate Administrator for Congressional and Intergovernmental Relations
Associate Administrator for Public Affairs
Audit Follow-up Coordinator, Office of International Affairs
Deputy Inspector General
37
------- |