OFFICE OF INSPECTOR GENERAL
Catalyst for Improving the Environment
Evaluation Report
Strategic Agricultural Initiative
Needs Revisions to
Demonstrate Results
Report No. 2007-P-00040
September 26, 2007
-------
Report Contributors: Jerri Dorsey
Jeffrey Harris
Kalpana Ramakrishnan
Olga Stein
Abbreviations
EPA U.S. Environmental Protection Agency
FQPA Food Quality Protection Act
FY Fiscal Year
GAO Government Accountability Office
IPM Integrated Pest Management
OIG Office of the Inspector General
OPEI Office of Policy, Economics, and Innovation
OPP Office of Pesticide Programs
OPPTS Office of Prevention, Pesticides, and Toxic Substances
RFP Request for Proposal
SAI Strategic Agricultural Initiative
USDA U.S. Department of Agriculture
Cover photo: Brown Marmorated Stink Bug, Halyomorpha halys
(Maine Department of Agriculture)
-------
I
'o
U.S. Environmental Protection Agency
Office of Inspector General
At a Glance
2007-P-00040
September 26, 2007
Catalyst for Improving the Environment
Why We Did This Review
We evaluated how the U.S.
Environmental Protection
Agency's (EPA's) Strategic
Agricultural Initiative (SAI)
uses performance measures to
demonstrate results. We
specifically sought to
I determine if SAI uses
performance measurement
tools and efficiency measures
that provide for continuous
program improvement.
Background
The SAI is a program that
helps growers of minor crops
replace high-risk pesticides
phased out or restricted
because of the Agency's
pesticide reevaluations.
Since 2001, the program has
given out about $4 million in
grants. According to Agency
staff, SAI fills a role within
EPA's regulatory framework
by helping minor crop growers
transition to reduced risk and
alternative methods of pest
management.
For further information,
contact our Office of
Congressional and Public
Liaison at (202) 566-2391.
To view the full report,
click on the following link:
www.epa.qov/oiq/reports/2007/
20070926-2007-P-00040.pdf
Strategic Agricultural Initiative Needs Revisions
to Demonstrate Results
What We Found
The SAI program has not demonstrated how it fulfills its unique role of helping
growers transition away from Food Quality Protection Act high-risk pesticides.
The program does not have a strategic plan or similar documents that link project
mission and goals, logic model, performance measures, and the data collected by
the program. Headquarters and the regions have inconsistent priorities for
implementing the program. This lack of structure makes it difficult to measure
and validate results.
The SAI databases, which are used to gather data on project performance, lack
definitions and structure, and thus contain incomplete and extraneous information.
Therefore, the SAI program does not have performance measurement tools or
performance measures in place to ensure or facilitate continuous program
improvement.
What We Recommend
We recommend that EPA develop a needs assessment for the SAI program to
demonstrate how it fulfills its role in meeting Food Quality Protection Act
requirements. If the need is demonstrated, the Program Office should create a
strategic plan which sets clear priorities for the direction of the program.
For the SAI Projects database, the Agency should create guidance documents
and establish standards and procedures for data collection and entry into these
databases. SAI data and results should be accessible to grantees and other
interested stakeholders. EPA agreed to reassess the need for the SAI program.
If the SAI needs assessment demonstrates a unique need for SAI, EPA agreed to
develop a strategic plan, based on logic modeling, that will address the issues
identified in this evaluation and the SAI needs assessment.
These recommendations will result in approximately $1.5 million in annual grant
funds put to better use. This is because either the need for these grants will be
determined to no longer exist or, if needed, their effectiveness will be enhanced
following the creation of a strategic plan and associated goals, logic model,
performance measures, and data systems.
-------
\
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
OFFICE OF
INSPECTOR GENERAL
MEMORANDUM
SUBJECT:
FROM:
TO:
September 26, 2007
Strategic Agricultural Initiative Needs Revisions
to Demonstrate Results
Report No. 2007-P-00040
Wade T. Najjum
Assistant Inspector General for PnJgfam Evaluation
James B. Gulliford
Assistant Administrator, Office of Prevention, Pesticides,
and Toxic Substances
Debbie Edwards
Director, Office of Pesticide Programs
This is a report on the Strategic Agricultural Initiative conducted by the Office of Inspector
General (OIG) of the U.S. Environmental Protection Agency (EPA). This report is subject to
revision by the OIG and, therefore, does not represent the final position of the OIG on the
subjects reported. Final determinations on matters in this report will be made by EPA managers
in accordance with established resolution procedures.
The estimated cost of this report - calculated by multiplying the project's staff days by the
applicable daily full cost billing rates in effect at the time - is $ 307,570.
Action Required
In accordance with EPA Manual 2750, you are required to provide a written response to the
report within 90 calendar days. You should include a corrective action plan for agreed-upon
actions, including milestone dates. We have no objection to the further release of this report to
the public. This report will be available at http://www.epa.gov/oig.
If you or your staff have any questions regarding this report, please contact me at (202) 566-0827
or najjum.wade@epa.gov: or Jeffrey Harris, Director of Special Studies, at (202) 566-0831 or
harris.ieffrey@epa.gov.
-------
Strategic Agricultural Initiative Needs Revisions
to Demonstrate Results
Table of Contents
Chapters
1 Introduction 1
Purpose 1
Background 1
Noteworthy Achievements 2
Scope and Methodology 2
Prior Coverage 3
2 SAI Needs to Improve Program Design to Validate Results 4
SAI Program Priorities Are Inconsistent 4
SAI Program Design Lacks Several Components 5
SAI Databases Lack Guidance, Definitions, and Structure 7
Recommendations 9
Agency Comments and OIG Evaluation 10
Status of Recommendations and Potential Monetary Benefits 11
Appendices
A OPPTS Evaluation Request 12
B Detailed Scope and Methodology 15
C Agency Response 16
D Distribution 19
-------
Chapter 1
Introduction
Purpose
The purpose of this evaluation is to respond to a request from the U.S.
Environmental Protection Agency's (EPA's) Office of Prevention, Pesticides, and
Toxic Substances (OPPTS) to evaluate, in part, the performance measures of the
Strategic Agricultural Initiative (SAI) program.1 We specifically sought to
determine whether SAI utilized performance measurement tools and performance
measures that provide for continuous program improvement.
Background
In the Food Quality Protection Act (FQPA) of 1996, Congress amended pesticide
regulations to establish a more consistent, protective regulatory scheme, grounded
in sound science. Among its directives, FQPA mandated EPA to create incentives
for developing and maintaining effective crop protection tools for American
farmers. Specifically, FQPA required EPA to promote Integrated Pest
Management (IPM),2 and implement IPM education, research, and demonstration
programs. FQPA also required EPA to create incentives to maintain existing, and
develop new, minor use pesticides.3
As part of implementing FQPA, EPA is phasing out some toxic and persistent
pesticides that are no longer considered safe for human health or the environment,
also known as "high-risk pesticides." As certain pesticides are phased out and
new alternatives are developed, the Agency acknowledged that farmers needed
help in adopting new pest control practices and approaches to pest management.
Thus, EPA developed the SAI in 1998. The Agency stated that the SAI program
helps growers of minor crops replace high-risk pesticides phased out or restricted
because of the Agency's pesticide reevaluations. According to SAI, the mission
of the program is to "support and promote environmentally sound agricultural and
pest management practices across the United States that are economically viable
and socially responsible."
1 See Appendix A for the original evaluation request that OPPTS sent to the OIG.
2 A strategy generally viewed as a sustainable approach to managing pests by combining biological, cultural,
physical, and chemical methods so that risks to economic well being, human health, and the environment are
minimized.
3 Minor use pesticides are defined as (1) using pesticides on crops where the total U.S. acreage is less than 300,000
acres, or (2) where using pesticides does not provide sufficient economic incentives for a registrant to support initial
or continued registration.
-------
According to EPA, the SAI provides funds on a competitive basis for projects that
are intended to:
Increase the adoption of sustainable agricultural protection practices through
implementing IPM programs;
Contribute to pesticide re-registration decisions under FQPA; and
Employ performance measures to determine if adopting IPM practices
improves the ecosystem.
Since 2001, the program has given out approximately $4 million in grants.
According to Agency staff, SAI fills a role within EPA's regulatory framework by
providing outreach and support to minor crop growers.
According to SAI staff, the program tracks and reports several performance
measures in various Agency systems. The SAI program uses two databases,
Projects and Collaborations, to help track, measure and report program impacts
and results. The SAI Projects and Collaborations databases can be found on the
SAI Website.4 As of March 2007, the Projects database had information for
150 projects.
Noteworthy Achievements
Since the program began in 1998, SAI reports that it has helped implement
reduced risk pest management strategies on over 780,000 acres of farmland,
reducing the use of highly toxic pesticides on these acres by over 30 percent.
Two documents that highlight SAI program accomplishments are the Results
Across the Nation summary, FY (Fiscal Year) 2001 - FY 2006; and SAI 2005
Snapshot, FY 2005. EPA reported, for example, that one project achieved
reducing 16,000 pounds of active ingredients of pesticide and produced a savings
of nearly $17 per acre. Another project resulted in removing two lakes from the
Clean Water Act 303(d) impaired waters list for atrazine.
Scope and Methodology
We interviewed stakeholders of the program to gather feedback about
performance measures and the data they collect to report results.5 Internal
stakeholders included EPA staff, including officials in OPPTS and its Office of
Pesticide Programs (OPP), and Division Directors. Also, we interviewed Agency
contractors and the U.S. Department of Agriculture (USD A).
We reviewed SAI program documents and guidance, and the Federal IPM Road
Map. We assessed the SAI databases to determine if projects had information on
4 http://www.aftresearch.org/sai/protected/. The SAI program has an assistance agreement with the American
Farmland Trust to develop and maintain the SAI Website and databases on behalf of the EPA. One needs a login
code and password to enter the Projects database.
5 During our evaluation, we did not interview growers associated with the SAI program or others in the agricultural
community.
-------
program and project specific performance measures, and whether this information
could be retrieved for reporting. Specifically, we selected the following data
fields because they are used to track overall project and program performance
measures: grant amount, acres impacted, pesticides transit!oned from,
collaborating organizations, and matching funds. We determined the percentage
of incomplete information (of the selected data fields) by region.
We performed our evaluation in accordance with generally accepted government
auditing standards, issued by the Comptroller General of the United States.
We performed our field work from November 2006 through February 2007.
See Appendix B for additional details on our scope and methodology.
Prior Coverage
The Government Accountability Office (GAO) and OIG have issued several
reports that address Federal Integrated Pest Management programs, FQPA, and
EPA performance measurement. The Office of Management and Budget
performed a 2004 Program Assessment Rating Tool assessment of OPP field
programs. Although these are not specifically targeted to the SAI program, they
contain applicable information and findings related to our evaluation of SAI
performance measures.
A 2006 OIG report, 2006-P-00028, Measuring the Impact of the Food Quality
Protection Act: Challenges and Opportunities, discusses the importance of
performance measurement for FQPA programs at EPA. The report emphasizes
that effective performance measurement enables an agency to establish baselines;
identify and prioritize problems; and evaluate, manage, and improve programs.
The 2004 Office of Management and Budget's Program Assessment Rating Tool
assigned the Field Programs a "Results Not Demonstrated" rating. The
assessment reported that "The budgets of the Field Programs taken independently
can not tie to program-specific goals because no adequate program-specific goals
have been established."
A GAO report, GAO-05-52, issued in 2004, discusses performance measurement
at EPA. This report, Environmental Indicators: Better Coordination Is Needed to
Develop Environmental Indicator Sets That Inform Decisions, found that EPA has
challenges with developing environmental indicator sets to inform decisions.
GAO also found the Agency had problems in linking specific environmental
management actions and program activities to changes in environmental
conditions and trends.
-------
Chapter 2
SAI Needs to Improve Program Design
to Validate Results
The SAI program has not demonstrated how it fulfills its unique role of helping
growers transition away from FQPA high-risk pesticides. Headquarters and the
regions have inconsistent priorities for implementing the program. No strategic
plan or similar document exists that describes the linkage between program
mission and goals, logic model, performance measures, and data the program
collected. This lack of structure makes it difficult to measure and validate results.
Also, the Projects database lacks definitions and structure, which prevents the
program from capturing data for performance measurement. Therefore, the SAI
program does not have performance measurement tools in place to ensure or
facilitate continuous program improvement and validate results.
SAI Program Priorities Are Inconsistent
The national SAI coordinator and regional staff have developed individual
priorities for implementing the program. While Headquarters and the regions
both stress FQPA in their Requests for Proposal (RFPs), the focus for how to
address FQPA requirements varies among regions. For example, some regions
focus on a particular group of pesticides; others prioritize by a specific pesticide
or have added endangered species issues to their RFPs.
According to Agency documents, projects that the SAI program funds must
support FQPA implementation by addressing high-risk chemicals. However, we
found that not all SAI projects focus solely on FQPA priorities. The SAI national
coordinator stated that she has not received guidance regarding which pesticides
are identified as FQPA priority pesticides for the program to address. We also
found EPA does not have guidance or planning documents showing the
relationship between the FQPA re-registration decisions and its projects and
program measures. Such program measures include, for example, potential acres
impacted and percentage reduction of high-risk pesticides as identified by the
Agency.
OPPTS has not performed a program needs or baseline assessment that identifies
the unique role and status of SAI in meeting FQPA requirements. For example, to
date, OPPTS has not identified the number, uses, and milestones of high-risk
pesticides to be phased out due to FQPA implementation. Furthermore, we
observed that the mission of the SAI program and how it will be implemented is
described differently in various documents. This divergence in documenting how
the mission is achieved makes it difficult for the program to establish priorities
and capture results.
-------
SAI Program Design Lacks Several Components
SAI Program Does Not Have a Strategic Plan
SAI does not have a strategic plan to chart and coordinate the direction of the
program and identify which outputs and outcomes it needs to measure against an
established baseline or a desired goal. The lack of the strategic plan has created
variation in the goals and objectives for the national program and regional RFPs.
Also, the SAI program has not clearly demonstrated how the projects funded by
the program contribute to transitioning from high-risk pesticides.
SAI cannot identify how each of its projects has contributed to the impact and
effectiveness of the overall program. The regions fund various types of grant
projects, which include demonstration, research, education, and implementation.
We found that for the projects that listed project type in the SAI Projects database,
45 percent are classified as demonstration, 32 percent as research, 16 percent as
education, and 7 percent as implementation.6 We found that the SAI program
does not have any guidance or documents that address: (1) when each project type
should be used, (2) what constitutes a successful project (including timeline for
completion), (3) how different project types will be measured against one another,
or (4) how each type of project ultimately relates to FQPA goals. One regional
coordinator remarked that the types of projects have great degrees of variation in
the time needed to be effective. For example, a research project may take much
longer than an implementation project because much of the IPM research has
already been completed.
Many of the regions' RFPs vary, and require grantees to address various priorities
with different reporting and performance requirements. Certain regions are
funding mostly research-based projects, while other regions give grants for a
mixture of demonstration and education projects. For example, Region 9 has
entered 81 percent of its projects into the database as demonstration projects,
while Region 4 has close to half, or 46 percent, as research projects. Neither
Headquarters nor the regions have documents that explain how a particular choice
of projects meets the SAI goal of transitioning growers away from high-risk
pesticides.
Changes Needed to Improve SAI Logic Model
While the Agency has developed a logic model for the SAI program, we found it
is not designed according to Office of Policy, Economics, and Innovation (OPEI)
Guidelines and other documents which define output and short-, intermediate-,
and long-term outcomes. Table 2.2 further defines these performance
measurement terms.
Of the 150 projects in the database, 112 projects provided information under project type.
-------
Table 2.2: Performance Measurement Terms
Term
Inputs
Outputs
Intermediate
Outcomes
End
Outcomes
Definition
Personnel, funds, and other resources that contribute to an activity
Quantitative or qualitative measures of activities, work products, or
actions
Changes in knowledge, behavior, or conditions that result from
program activities and are needed to achieve the end outcome
The ultimate outcomes of program activities -the results compared
to their intended purpose
Source: OIG analysis
Specifically, some of the items currently labeled as "outcomes" in the SAI logic
model should be classified as outputs, or lower level (i.e., short-term outcomes).
For example: "Safer environment for farmers, the public and wildlife" is listed as
a long-term outcome, but this qualitative statement does not measure quantitative
environmental or health indicators. While the "Nationwide appreciation of EPA
from growers and the IPM community " is listed as an intermediate outcome, it
does not represent changes in knowledge of high-risk pesticides or grower
behavior. The national coordinator provided OIG with a 2006 document, titled
Data Fields In SAI Database That Correspond To SAI Logic Model., which lists
data fields used to capture short-, intermediate-, and long-term outcomes. The
data fields listed as long-term outcomes include worker safety issues, outreach,
and use of the SAI Transition Gradient. These should be classified as
intermediate outcomes because they affect behavior or conditions rather than
providing specific environmental or health measurements.7
According to the F Y 2006 document, Guidelines for Measuring the Performance
of EPA Partnership Programs, a logic model is a visual diagram and text that
shows the relationship between a program's work and its desired results.
Furthermore, it states a logic model describes the logical (causal) relationships
among program elements such as resources, activities, outputs, target decision-
makers, and desired outcomes. The OPEI Guidelines add that a well-thought-out
logic model can easily lead managers to establishing their program's performance
measures and serves as a basic road map for the program.
Examples of long-term outcomes are found on the SAI Website. Human-related outcomes include pesticide
exposure levels in study population and levels of residues of high-risk pesticides in food. Water quality measures
include mortality to non-target aquatic and terrestrial organisms caused by pesticides and populations of beneficial
organisms in field and adjacent habitat.
-------
SAI Needs to Establish Criteria for Program and Project Effectiveness
Existing quantitative criteria do not capture the SAI program's impact and
effectiveness towards reducing the amount of high-risk pesticides used by
growers. In addition, OPPTS allocates grant funds to regions based on minor
crop sales, and not on the regions' project performance or contributions to
program goals. The national coordinator gauges the success of the SAI program
based on "(1) a cohesive team that achieves national (common) goals and (2) the
program serves the Agency, i.e., it has performance measurement and
accountability," rather than how well the SAI program contributes to EPA's
strategic goals.
SAI cannot identify how each of its projects has contributed to the impact and
effectiveness of the overall program. The SAI program also has not established
criteria to gauge project effectiveness. Because no consistent criteria exist for an
effective project, it is difficult to identify which SAI projects have been the most
successful, and why. For example, we found many of the regional coordinators
and the national coordinator have a different working definition of a successful
project. The national coordinator defined a successful project as one that
"(1) Achieves stated project goals, (2) has potential for adoption for other growers
in that commodity, and (3) addresses OPP priorities." Regional coordinators'
definitions included: "Success means positive feedback from growers-grower
chatter," and "Gauges of a successful project are: (1) What new information is
made available? (2) Is there movement on the transition gradient? (3) Is there
good grower involvement? (4) Is it economically viable? and (5) Did the project
educate more growers?" According to one regional coordinator, one measure
currently being used by SAI (number of acres per grant dollar spent) did not
provide a good representation of results for projects that were done on the same
amount of acreage each year.
SAI Databases Lack Guidance, Definitions, and Structure
The SAI databases lack guidance, definitions, and a clear structure for entering
information.8 We found the lack of guidance has led to inconsistencies in how
each region characterizes and counts collaborations activities and project data.
When descriptive information is entered for numeric data fields in the Projects
database, aggregating information to conduct quantitative trend analyses for the
SAI program as a whole is difficult. The SAI program is unable to utilize the
databases for performance measure reporting.
We found that data fields are often missing data, or numerical data is entered in
descriptive form. Sometimes different units of measurement are used within the
same field for different projects. We analyzed the Projects database to determine
8 In November 2006, the national SAI coordinator stated proposed changes would be made to definitions for data
fields in the Projects database. As of March 2007, these changes to the Projects database have not been
implemented.
-------
what percentage of information was available for data fields or if they were
incomplete.9 Figure 2.1 shows each region's percentage of incomplete
information for selected data fields. Some regions had a large amount of
information incomplete in their project reports. For example, two regions had
approximately 33 to 40 percent of their data field entries incomplete, with data
classified as unavailable, unknown, or to be determined. In contrast, some other
regions had only 10 to 13 percent of their entries incomplete.
Figure 2.1. Regional Percentage of Incomplete Information for Selected
Data Fields
100%n
90%-
Si
ง. 80%-
E
8 o 70%-
5 & 60%-
1 1 50%-
| 1 40%-
ฐ ฃ 30%-
c c
8 20%-
-------
Recommendations
We recommend that the Assistant Administrator, Office of Prevention, Pesticides,
and Toxic Substances:
2-1 Perform a needs assessment for the SAI program to establish to what
extent SAI fulfills a unique role for meeting FQPA requirements. This
assessment should also illustrate how program resources are used to fulfill
specific needs of the FQPA. If the assessment results show that the SAI
program does not fulfill a unique role in meeting FQPA requirements,
OIG recommends that the Agency phase out the SAI program and shift the
funding to other OPP priorities.
We recommend that the Director, Office of Pesticide Programs:
2-2 If a valid need is identified by the above assessment, create a strategic plan
with:
a national set of goals, objectives, and measures;
clear, strategic priorities with a focus on transition to reduced-risk
practices;
links to projects to EPA and OPP Strategic Plan goals and
performance measures;
internal measures used to gauge project and program success with
the database information;
a roadmap that articulates how each type of proj ect (research,
demonstration, implementation, education) directly contributes to
program outcomes; and
a logic model that adheres to OPEI guidelines and accurately
reflects outputs and short-, intermediate-, and long-term outcomes
of the program.
2-3 Develop an overarching RFP that aligns national goals of the SAI program
and also allows for regional priorities.
2-4 Create guidance documents for the use of the Projects and Collaborations
databases. These documents should:
include a clear and consistent data dictionary for all fields,
provide a quality control/quality assurance check for data entry,
specify how the data will be used for reporting and analysis by the
SAI national and regional coordinators, and
make SAI data and results accessible to grantees and other
interested stakeholders.
-------
These recommendations will result in approximately $1.5 million in annual grant
funds put to better use. This is because either the need for these grants will be
determined to no longer exist or, if needed, their effectiveness will be enhanced
following the creation of a strategic plan and associated goals, logic model,
performance measures, and data systems.
Agency Comments and OIG Evaluation
We received Agency comments on August 15, 2007, regarding our draft report
findings and recommendations. While the Agency's response discussed the OIG
recommendations, it did not address how OPPTS would assess the need for the
SAI program, or resolve the structural issues identified in Recommendations 2-1
and 2-2. Subsequently, OPPTS issued an additional response on September 17,
2007 (see Appendix C). This second response concurred with our
recommendations, and clarified the Agency's commitment to perform a needs
assessment for the program.
The Agency has agreed to develop a work plan and schedule to complete a needs
assessment by the first quarter of 2008. According to Agency staff, the needs
assessment will be conducted by an independent contractor to ensure objectivity.
The assessment will include the views of key stakeholders, including USDA
counterparts who share responsibilities under the FQPA.
The Agency stated that if the needs assessment demonstrates that SAI fulfills a
unique role within OPP, it will implement Recommendations 2-2 through 2-4 by
developing a strategic plan based on a logic model, and will restructure the
program as needed. OPP will utilize a single national RFP in future grant cycles.
10
-------
Status of Recommendations and
Potential Monetary Benefits
RECOMMENDATIONS
POTENTIAL MONETARY
BENEFITS (in $OOOs)
Rec.
No.
2-1
Page
No. Subject Status1
9 Perform a needs assessment for the SAI program to 0
establish to what extent SAI fulfills a unique role for meeting
FQPA requirements. This assessment should also illustrate
how program resources are used to fulfill specific needs of
the FQPA. If the assessment results show that the SAI
program does not fulfill a unique role in meeting FQPA
requirements, OIG recommends that the Agency consider
phasing out the SAI program and shift the funding to other
OPP priorities.
Action Official
Assistant
Administrator,
Office of
Prevention,
Pesticides, and
Toxic Substances
Planned
Completion
Date
12/31/07
Claimed
Amount
Agreed To
Amount
$1,500
2-2 9 If a valid need is identified by the above assessment, create
a strategic plan with:
a national set of goals, objectives, and measures;
clear, strategic priorities with a focus on transition to
reduced-risk practices;
links to projects to EPA and OPP Strategic Plan goals
and performance measures;
internal measures used to gauge project and program
success with the database information;
a roadmap that articulates how each type of project
(research, demonstration, implementation, education)
directly contributes to program outcomes; and
a logic model that adheres to OPEI guidelines and
accurately reflects outputs and short-, intermediate-,
and long-term outcomes of the program.
2-3 9 Develop an overarching RFP that aligns national goals of the
SAI program and also allows for regional priorities.
2-4 9 Create guidance documents for the use of the Projects and
Collaborations databases. These documents should:
include a clear and consistent data dictionary for all
fields,
include a quality control/quality assurance check for
data entry,
specify how the data will be used for reporting and
analysis by the SAI national and regional coordinators,
and
make SAI data and results accessible to grantees and
other interested stakeholders.
Director,
Office of Pesticide
Programs
Director,
Office of Pesticide
Programs
Director,
Office of Pesticide
Programs
0 = recommendation is open with agreed-to corrective actions pending;
C = recommendation is closed with all agreed-to actions completed;
U = recommendation is undecided with resolution efforts in progress
11
-------
Appendix A
OPPTS Evaluation Request
7/12/2006
EPA's Strategic Agricultural Initiative (SAI)
Program Evaluation Proposal
1. Title: Evaluation of the SAI Program.
EPA's Strategic Agricultural Initiative helps to ensure the safety and sustainability of
American agriculture by working with farmers, researchers, and other agencies to implement
reduced risk pest management strategies. EPA's Office of Pesticide Programs (OPP) established
SAI in 1998 to address Food Quality Protection Act (FQPA) priorities and help reduce the use of
high-risk pesticides on fruits and vegetables. These pesticides can harm human health and can
also have adverse effects on air, water, and biodiversity. To help producers transition away from
using these pesticides, SAI provides funds on a competitive basis for projects that increase the
adoption of sustainable agricultural production practices through the implementation of
Integrated Pest Management (IPM) programs, contribute to pesticide re-registration decisions,
and employ performance measures to determine if the adoption of IPM improves the ecosystem.
Unlike more traditional programs, SAI uses innovative online tools to manage and track the
results of projects and outreach efforts. These include 1) a searchable project database that can
provide regional and national summaries across 35 data fields, 2) a collaborations database to
record outreach activities, and 3) a toolbox of information on grant management, grant
opportunities, performance measurement, baseline data sources, reporting outcomes, and
evaluation. SAI specialists in each EPA region use these online resources to manage projects
and make final decisions on funding. The specialists also maintain regional outreach programs,
and work with other programs within the Agency, state and federal agencies, commodity groups,
farm organizations, and stakeholder groups to further SAI objectives. Within OPP, the national
SAI coordinator oversees the program and ensures that project results are aligned with OPP
goals.
2. Questions:
(i) How can SAI help more farmers adopt successful technologies, field-tested through
the SAI grants program, to achieve greater technology transfer and help farmers
comply with FQPA?
(ii) Does SAI utilize the appropriate performance measurement tools and efficiency
measures that provide for continuous program improvement?
(iii) How can SAI improve its databases (projects and collaborations) to collect more
pertinent information for the Agency?
(iv) How can the structure of SAI be improved to produce better program outcomes?
(v) What role can SAI play in future EPA/OPP activities?
12
-------
3. Justification:
Over the last three years, SAI has emphasized the use of outcome, performance, and
efficiency measures. These measures were created in a self-directed team effort to focus the
program on outcomes and efficiency. Now, at the request of several other federal agencies,
including USD A, the Department of Defense and the Department of the Interior, SAI is
beginning to share these measures to help other agencies measure the impacts of IPM adoption.
Thus, it is important to determine if SAI has indeed adequately developed and used these
measures to fulfill the tenets of FQPA and the Government Performance and Results Act, as well
as answered questions posed by the Office of Management and Budget. We welcome the
evaluation as a means of determining if modifications in SAI would make the program more
effective. If the evaluation finds that our focus on measures and outcomes is adequate, then other
programs and divisions within EPA and other Agencies may be able to benefit by following a
similar process to develop measures and record progress. In this way, SAI can serve as a
template for other federal programs that need better measurement tools. Also, if OPP field
programs are scheduled to be PARTED in the future, results of this evaluation could be used to
adjust the program to be better positioned to respond to the PART questions.
This evaluation would analyze the outcomes of SAI and its efficiency. For the outcome
components, evaluators will use a multiple group, non-experimental approach that uses
observations from key customers after the program has been implemented. Evaluators will
interview farmers, (end users of the program), as well as federal personnel (internal customers.)
This will allow evaluators to determine if SAI achieves three key outcomes: (1) increased
adoption of sustainable agricultural practices on farms, (2) increased use of performance
measures to quantitatively document reductions in risk due to the adoption of sustainable
agricultural practices, and (3) increased communication with other EPA programs and other
agencies to help implement EPA Strategic Plan Goal 4.
The efficiency evaluation will focus on how SAI achieves its outcomes. Evaluators will
interview the SAI coordinator and the 10 regional SAI specialists to determine if SAI conducts
its operations in an efficient manner. In addition, the efficiency evaluation will use quantifiable
data from the SAI projects database to calculate efficiency measures. By emphasizing outcomes
and efficiency, this evaluation can determine whether SAI has been effective in helping the
Agency implement FQPA.
4. Information Needed:
Interviews with EPA staff, U.S. Department of Agriculture staff, SAI grantees, and
farmers will provide most of the outcome-related information. SAI will give the evaluator(s) a
list of twenty or so Agency contacts with detailed knowledge of SAI and its past activities. SAI
will also provide the evaluator(s) with a list of nine grantees and/or farmers who can comment on
the effectiveness of the program. Since we limit this part of the evaluation to only nine
interviews involving non-federal employees, this evaluation will not need an Information
Collection Request (ICR). SAI will also provide the evaluator(s) with complete access to SAFs
online tools (e.g. SAI toolbox, project database, and collaborations database) to further explore
and analyze program outcomes.
13
-------
For the efficiency part of the evaluation, interviews with the SAI coordinator and the 10
regional SAI specialists will provide most the information needed. Additional information can
be gleaned from SAI's online tools that contain efficiency data (e.g. SAI project database). The
SAI coordinator and specialists will provide any other information requested by the evaluator(s).
5. Contacts:
Frank Ellis, Acting Branch Chief
Environmental Stewardship Branch,
Biopesticides and Pollution Prevention Division
Office of Pesticide Programs
Regina Langton, National SAI Coordinator
Environmental Stewardship Branch
Biopesticides and Pollution Prevention Division
Office of Pesticide Programs
14
-------
Appendix B
Detailed Scope and Methodology
We generally performed our evaluation in accordance with generally accepted government
auditing standards, issued by the Comptroller General of the United States. We performed our
field work from November 2006 through February 2007.
To determine if the SAI has been utilizing the appropriate performance measurement tools and
efficiency measures that provide for continuous program improvement, we interviewed internal
stakeholders of the program to gather feedback about performance measures and the data they
collect to report results. We interviewed Division Directors from the Biopesticides and Pollution
Prevention Division and the Biological and Economic Analysis Division. We attended the 2006
SAI Interagency Conference on IPM in Dallas. During this conference, we interviewed the
national and regional SAI coordinators. We also met with staff of the USDA and American
Farmland Trust who collaborate with SAI on collecting performance measures. In addition, we
met with former SAI staff with detailed knowledge of SAI and its past activities in order to
gather information about performance measures used by the program.
To understand the historic context of SAI the program, we read SAI Regions' Proposals written
in 1997, which detailed how the program would be implemented in the SAI Pilot Regions (4, 5,
9, and 10). We reviewed documents containing program guidance such as the FY 2001, 2005,
and 2006 SAI Budget Guidance Memorandums, the 2006 SAI Draft Marketing/Communications
Strategy, and FY 2006 Regional Distributions for SAI grant monies. We analyzed the 2004
USDA National Road Map for IPM to identify any guidance that may be relevant to SAI and
adoption of performance measures. We reviewed regional annual summaries and regional RFPs
to determine if regions had similar priorities when funding projects. We also examined the 2006
SAI Logic Model to evaluate the outputs and outcomes listed for the program.
We analyzed the SAI Projects and Collaborations databases. We examined databases used by
SAI to see if they capture consistent, reportable, and measurable information. To analyze the
Projects database, we first used the database generator located on the Projects database Website.
We selected for specific data fields: grant amount, acres impacted, pesticides transitioned from,
leveraged funds, collaborating organizations, and project type. We sorted each of these data
fields separately using Microsoft Excelฎ. After re-organizing and sorting, we analyzed the data
to determine if the SAI Projects database has the necessary information to track and report on
performance measures. We determined the percentage of incomplete information for the above
selected fields for all 150 projects in the Projects database. We determined the percentage of
incomplete information (of the selected data fields) by region. We also reviewed documents
provided by the SAI national coordinator with proposed changes to the data fields within the
Projects database. For the Collaborations database, we reviewed the types of activities that were
recorded, noting if regions entered or counted similar activities differently.
15
-------
Appendix C
Agency Response
September 17, 2007
Memorandum
Subject: Clarification of OPPTS Response to Draft Report "The Strategic Agricultural
Initiative Needs Revisions to Demonstrate Results" - Assignment No. 2006-01630
(dated July 16, 2007)
From: James B. Gulliford, Assistant Administrator
To: Jeffrey Harris, Director
Cross-Media Issues
Office of Program Evaluation
Office of Inspector General
Thank you for the opportunity to discuss the draft evaluation "The Strategic Agricultural
Initiative Needs Revisions to Demonstrate Results - Assignment No. 2006-01630 (dated July 16,
2007) on September 10, 2007, and to clarify our earlier response. Your evaluation, conducted at
our request, focused on the usefulness of the performance measures of the Strategic Agricultural
Initiative (SAI) program.
OPPTS wholeheartedly agrees with your conclusion that OPP should reassess the need
for the SAI program. This reassessment should focus on SAI's linkage to the Food Quality
Protection Act, the needs and goals of EPA's Pesticide Program, and possible restructuring of the
SAI program, as necessary, to meet the needs identified.
The concerns about methodology that were expressed in our comments do not diminish
our overall agreement with the recommendations. Your draft evaluation highlights the need to
demonstrate the unique role of the SAI program; the need for a stronger link to program missions
and goals; inconsistencies between Headquarters and the Regions; and a concern about the
structure of the program. The draft evaluation also outlines some specific concerns regarding
SAI databases and streamlining grant programs. We agree with these recommendations.
The following is an itemized list of each recommendation and the actions that OPP
intends to take to address each recommendation. Numbers 2-2 through 2-4 obviously hinge on a
positive finding from the first recommendation, that the need for a stewardship program still
exists to assist growers as identified in the FQPA.
16
-------
2-1 The draft report recommends that OPPTS perform a needs assessment for the SAI
program to establish to what extent SAI fulfills a unique role for meeting FQPA
requirements. This assessment should also illustrate how program resources are
used to fulfill specific needs of the FQPA. If the assessment results show that the
SAI program does not fulfill a unique role in meeting FQPA requirements, OIG
recommends that the Agency phase out the SAI program and shift funds to other
OPP priorities.
We agree with your recommendation to conduct a needs assessment, which would include
gathering and assessing information on risk reduction needs from a variety of key stakeholders
and our counterparts in USDA who share responsibilities under the FQPA. By the end of
October 2007, we propose to develop a methodology, workplan, and schedule for the completion
of a needs assessment for the mission of SAI. This needs assessment would identify needs based
on FQPA and EPA/OPP's strategic goals and would further identify an optimum structure for
meeting those needs. We anticipate that the assessment will be complete in the first quarter of
2008.
2-2 If a valid need is identified by the above assessment, OPP should create a strategic
plan, with:
a national set of goals, objectives and measures,
clear, strategic priorities with a focus on a transition to reduced-risk practices,
links to projects to EPA and OPP Strategic Plan goals, and performance
measures,
internal measures used to gauge project and program success with the database
information,
a roadmap that articulates how each type of project (research, demonstration,
implementation, education) directly contributes to program outcomes, and
a logic model that adheres to OPEI guidelines and accurately reflects outputs,
and short-, intermediate-, and long-term outcomes of the program.
If the SAI needs assessment demonstrates a unique need for SAI then OPP will begin to look at
recommendations 2-2 thru 2-4. OPP, in coordination with the ten EPA Regions, will develop a
strategic plan, based on logic modeling, that will address the issues identified in this evaluation
and the SAI needs assessment.
If the needs assessment does demonstrate a unique need for a non-regulatory approach to
pesticide risk reduction, such as SAI, we recognize that there may be a need for restructuring of
the program as it exists today. Any such restructuring would be done in order to contribute to
the successful achievement of EPA/OPP national goals, to Regional goals, and to significantly
improve pest management practices for growers. This would ultimately reduce the risks of both
pests and pest management practices for the American people.
17
-------
2-3 Develop an overarching RFP that aligns national goals of the SAI program, and also
allows for Regional priorities.
o OPP agrees with this recommendation and will utilize a single national RFP in
future grant cycles.
2-4 Create guidance documents for the use of the Projects and Collaborations
databases. These documents should:
include a clear and consistent data dictionary for all fields,
provide a quality control/quality assurance check for data entry,
specify how the data will be used for reporting and analysis by the SAI
National and Regional Coordinators, and
make SAI data and results accessible to grantees and other stakeholders.
OPP agrees that these are necessary components and will include them in any projects and
collaboration databases developed for the SAI program.
18
-------
Appendix D
Distribution
Office of the Administrator
Assistant Administrator, Office of Prevention, Pesticides, and Toxic Substances
Principal Deputy Assistant Administrator, Office of Prevention, Pesticides, and Toxic Substances
Director, Office of Pesticide Programs
Deputy, Office of Director for Programs, Office of Pesticide Programs
Special Assistant to the Office Director, Office of Pesticide Programs
Director, Biopesticides and Pollution Prevention Division, Office of Pesticide Programs
Branch Chief, Environmental Stewardship Branch, Office of Pesticide Programs
Office of General Counsel
Agency Followup Official (the CFO)
Agency Followup Coordinator
Associate Administrator for Congressional and Intergovernmental Relations
Associate Administrator for Public Affairs
Audit Followup Coordinator, Office of Prevention, Pesticides, and Toxic Substances
Acting Inspector General
19
------- |