^&STA%
* o %
IŪ I
OFFICE OF INSPECTOR GENERAL
Catalyst for Improving the Environment
Evaluation Report
Measuring the Impact of the
Food Quality Protection Act:
Challenges and Opportunities
Report No. 2006-P-00028
August 1, 2006
-------
Report Contributors: Jerri Dorsey
Gabby Fekete
Alice Fong
Jeffrey Harris
Abbreviations
EPA U.S. Environmental Protection Agency
FQPA Food Quality Protection Act
GAO Government Accountability Office
OIG Office of Inspector General
OPP Office of Pesticide Programs
USD A U.S. Department of Agriculture
Cover photo: The Food Quality Protection Act emphasizes the need to protect children
from pesticides (EPA OIG photo).
-------
^tDsr%
o
3s
a VM/^ *
V 0<-°
PRO"*4-
U.S. Environmental Protection Agency
Office of Inspector General
At a Glance
2006-P-00028
August 1, 2006
Catalyst for Improving the Environment
Why We Did This Review
We initiated this review to
determine the ability of the
U.S. Environmental Protection
Agency's (EPA's) Office of
Pesticide Programs (OPP) to
measure its performance in
meeting the mandates of the
Food Quality Protection Act
(FQPA). We sought to
determine the strengths and
weaknesses of OPP's current
measuring system, how OPP
can use existing data to
measure, and what impact
FQPA had on mitigating
dietary pesticide exposure risk
on children's health. This is
the third in a series of three
reports on FQPA.
Background
The goal of EPA's pesticide
program is to protect public
health and the environment by
ensuring pesticides are used
safely. FQPA changed the
way OPP regulates pesticides
and emphasizes protecting
children's health.
Measuring the Impact of the Food Quality
Protection Act: Challenges and Opportunities
What We Found
Although EPA has made progress in implementing the requirements of the FQPA,
we found that OPP has primarily measured its success and the impact of FQPA by
adherence to its reregistration schedule rather than by reductions in risk to
children's health. The measures used by OPP generally indicate actions taken,
instead of environmental or human health outcomes achieved. OPP lacks outcome
measures to assess the specific impact of those actions on the health of children
and others. OPP has recently taken steps to develop outcome measures, but
significant challenges remain.
By integrating existing data into a suite of performance measures, OPP can better
track the effectiveness of regulatory decisions and program performance. We
identified several pools of quantitative data available for use as a suite of
performance indicators, but coordination efforts will be needed. OPP can better
utilize a number of data and measurement sources, including the National Health
and Nutrition Examination Survey and the U.S. Department of Agriculture's
Pesticide Data Program, to track health-based indicators of children's health risks.
EPA can measure the impact of FQPA on children's health more efficiently
through the examination of pesticide exposure data, and changes in usage patterns,
substitutions, and import trends. We used the U.S. Department of Agriculture's
Pesticide Data Program data to illustrate dietary risk changes since the passage of
FQPA in toxicity risks on the foods commonly consumed by children.
What We Recommend
For further information,
contact our Office of
Congressional and Public
Liaison at (202) 566-2391.
To view the full report,
click on the following link:
www.epa.qov/oiq/reports/2006/
20060801 -2006-P-00028.pdf
To view a supplemental report
with additional details, click on:
www.epa.qov/oiq/reports/2006/
20060801-2006-P-00028A.pdf
We recommend that OPP work to move away from primarily using outputs as
performance measures, and implement a suite of output and outcome measures to
assess the human health and environmental impacts of its work. We also
recommend that OPP pursue revision of EPA's goal structure as appropriate, and
work with other EPA program offices and other Federal agencies to obtain needed
data. EPA generally agreed with the recommendations, and expressed its
appreciation for our findings. We made changes where appropriate.
-------
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
OFFICE OF
INSPECTOR GENERAL
August 1, 2006
MEMORANDUM
SUBJECT: Measuring the Impact of the Food Quality Protection Act:
Challenges and Opportunities
Report No. 2006-P-00028
TO: Jim Jones
Director, Office of Pesticide Programs
Lyons Gray
Chief Financial Officer
This is our report on the subject evaluation conducted by the Office of Inspector General (OIG)
of the U.S. Environmental Protection Agency (EPA). This report contains findings that describe
the problems the OIG has identified and corrective actions the OIG recommends. This report
represents the opinion of the OIG and does not necessarily represent the final EPA position.
Final determinations on matters in this report will be made by EPA managers in accordance with
established resolution procedures.
The estimated cost of this report - calculated by multiplying the project's staff days by the
applicable daily full cost billing rates in effect at the time - is $227,099.
Action Required
In accordance with EPA Manual 2750, you are required to provide a written response to this
report within 90 calendar days. You should include a corrective actions plan for agreed upon
actions, including milestone dates. We have no objections to the further release of this report to
the public. This report will be available at http://www.epa.gov/oig.
^t0SrX
* O \
ijfe*
-------
If you or your staff has any questions, please contact me at (202) 566-0847 or
roderick.bill@epa.gov, or Jeffrey Harris, Product Line Director for Cross Media Issues,
at (202) 566-0831 or Harris.ieffrev@epa.gov.
Sincerely,
Acting Inspector General
cc: James B. Gulliford
Assistant Administrator, Office of Prevention, Pesticides, and Toxic Substances
-------
Measuring the Impact of the Food Quality Protection Act: Challenges and Opportunities
Table of C
Chapters
1 Introduction 1
Purpose 1
Background 1
Scope and Methodology 2
Prior Reviews 3
Results of Review 4
2 Improvements Needed in OPP's Measurement System 5
OPP's Current Measurement System Focuses on Outputs 5
Outcome Measures under Development 7
OPP Faces Challenges in Creating Effective Performance Measures 7
FQPA Logic Model Can Improve Performance Measuring 8
EPA Should Consider Revising Goal Structure 10
Recommendations 11
Agency Response and OIG Evaluation 11
3 Suite of Performance Measures Can Help OPP Better Measure Impact 12
Using a Suite of Performance Measures Can Help OPP Track Progress 12
Five Examples Demonstrate Use of Suite Approach 15
Recommendation 16
Agency Response and OIG Evaluation 16
4 OPP Decisions on Pesticide Dietary Exposure Risk Had Positive Impact 17
FQPA Actions Decreased Dietary Pesticide Risks to Children 17
Reduced Risk Attributable to Two Major EPA Actions 19
Method Could Help OPP 21
Recommendation 22
Agency Response and OIG Evaluation 22
Status of Recommendations and Potential Monetary Benefits 23
Appendix
A Agency Response 24
B OPP's Proposed Goal-Related Measures 30
C Distribution 31
-------
Chapter 1
Introduction
Purpose
We initiated this review to evaluate the U.S. Environmental Protection Agency's
(EPA's) activities to implement the Food Quality Protection Act (FQPA) of 1996.
Our overall objective was to determine the impact of FQPA on Agency practices,
data requirements, and children's health. The primary goal of FQPA is to protect
children and infants from pesticide exposures. In this report, our primary
objective was to evaluate the effectiveness of EPA's Office of Pesticide Programs
(OPP) in measuring the overall impact of FQPA implementation activities. We
specifically sought to determine:
What are the strengths and weaknesses of OPP's current measurement
system in tracking FQPA objectives, and how could it be improved, if
necessary?
What existing data can OPP use to assess its performance under FQPA
and measure the impact of its regulatory actions?
What impact did FQPA have on mitigating dietary pesticide exposure risk
and on children's health?
Background
Congress unanimously passed the FQPA in 1996, due in large part to a 1993
National Academy of Sciences report, Pesticides in the Diets of Infants and
Children. According to this report, the then-current scientific and regulatory
approaches did not adequately protect infants and children from pesticides.
Children are uniquely susceptible to the health threats posed by pesticides, in both
household chemicals and food. Children generally consume more fresh produce
and drink more water per pound of body weight than adults. Additionally, a
child's exposure to pesticides can occur as early as the prenatal phase, or during
infancy through breast-feeding. Children have higher rates of metabolism, less
mature immune systems, unique diets, and distinct patterns of activity and
behavior when compared with adults.
The Government Performance and Results Act of 1993 requires Federal agencies,
including EPA, to prepare performance plans containing annual performance
goals and measures to help move them toward managing for results. Performance
measurement is the monitoring and reporting of program accomplishments,
particularly progress toward pre-established goals. Performance measures may
1
-------
address the type of program activities conducted, the direct products and services
delivered by a program (outputs), and the results of those products and services
(outcomes). Effective performance measurement enables an agency to establish
baselines; identify and prioritize problems; and evaluate, manage, and improve
programs. Table 1.1 further defines performance measurement terms.
Table 1.1: Performance Measurement Terminology
Term
Definition
Inputs
Personnel, funds, and other resources that contribute to an activity
Outputs
Quantitative or qualitative measures of activities, work products, or
actions
Intermediate
Outcomes
Changes in knowledge, behavior, or conditions that result from
program activities and are needed to achieve the end outcome
End
Outcomes
The ultimate outcomes of program activities - the results compared
to their intended purpose
EPA's strategic plan outlines the Agency's five long-term goals and guides in
establishing the annual goals that must be met along the way. To fulfill its five
strategic goals, the plan includes a series of more specific goals in the form of
objectives and sub-objectives. Each of these objectives has associated
performance measures designed to demonstrate progress in achieving the
objective and, eventually, the strategic goal. The annual performance plan defines
the Agency's budget and associated goals and objectives in greater detail and ties
the annual budget to the 5-year strategic plan.
The mission of OPP is to protect human health and safeguard the environment
from unreasonable adverse effects resulting from the use of pesticides. OPP is
responsible in part for implementing the FQPA. To successfully implement
FQPA, OPP needs to use new tools to reduce pesticide exposures and resultant
risks for children. OPP is responsible for using performance measures and goals
to assess the impact of its actions.
Scope and Methodology
We generally performed our evaluation in accordance with Government
Auditing Standards, issued by the Comptroller General of the United States.
We performed our field work from July 2005 through January 2006.
To determine the strengths and weaknesses of OPP's current FQPA-related
measurement system, we reviewed internal OPP documents, EPA Office of the
Chief Financial Officer reports and plans, and Office of Management and Budget
documents. We reviewed EPA's 2003 - 2008 Strategic Plan: Direction for the
Future, fiscal 2004 and 2005 annual performance plans, and the Agency's fiscal
2004 and 2005 annual reports. We interviewed internal program staff, and
2
-------
internal and external stakeholders. We reviewed reports issued by the
Government Accountability Office (GAO). We reviewed Florida State
University's Program for Environmental Policy and Planning Systems' Chemical
and Pesticide Results Measures project, a cooperative agreement with OPP, and
interviewed the author of a report prepared as a result of the project. We also
interviewed other members of academia.
To determine what additional data sources and measures OPP could use and other
ways it can use existing data sources, we consulted the U.S. Department of
Agriculture, U.S. Department of Health and Human Services, Office of
Management and Budget, and EPA's Office of the Chief Financial Officer.
In assessing the overall impact of OPP's actions on children's health, we
interviewed internal program staff, and internal and external stakeholders, to
identify measures, additional data, and trends in children's health. We reviewed
other entities' research on potential human health indicators related to pesticide
exposures, dietary risk, and reductions in risk due to EPA action. We conducted
an analysis of publicly available toxicological and residue data supporting EPA
dietary risk assessments to assess the impact of FQPA on dietary pesticide risks
from 1994 through 2003.1 A detailed discussion of the methodology for this
analysis, which can be used by OPP to perform its own analyses, is in a
supplemental report.
Our review focused on existing data and interviews, and we did not examine
internal controls. We evaluated OPP's compliance with the Government
Performance and Results Act, FQPA, and other regulations as appropriate.
Prior Reviews
This report is the last in a series of three EPA Office of Inspector General (OIG)
reports on the Agency's FQPA implementation efforts. The prior two reports are:
EPA OIG Report No. 2006-P-00009, Opportunities to Improve Data
Quality and Children's Health through the Food Quality Protection Act,
January 10, 2006
EPA OIG Report No. 2006-P-00003, Changes Needed to Improve Public
Confidence in EPA's Implementation of the Food Quality Protection Act,
October 19, 2005
In addition, we reviewed the following GAO reports that addressed performance
measures:
1 Some of the analysis work was conducted through a contract with Benbrook Consulting Services, Sandpoint,
Idaho.
3
-------
GAO-05-52, Environmental Indicators: Better Coordination Is Needed to
Develop Environmental Indicator Sets That Inform Decisions,
November 17, 2004
GAO-02-372, Performance Reporting: Few Agencies Reported on the
Completeness and Reliability of Performance Data, April 26, 2002
GAO Letter Report B-285312, Managing for Results: Assessing the
Quality of Program Performance Data, May 25, 2000
GAO/RCED-OO-77, Managing for Results: EPA Faces Challenges in
Developing Results-Oriented Performance Goals and Measures,
April 2000
Further, we reviewed a report issued by the Florida State University's Program
for Environmental Policy and Planning Systems in February 2003, Chemical and
Pesticide Results Measures II. The report was based on a joint effort by EPA and
the university to develop a national set of chemical, pesticide, and pollution
prevention indicators to describe and understand environmental trends and
conditions concerning chemical and pesticide issues.
We also reviewed reports issued by the Office of Management and Budget on its
Program Assessment Rating Tool assessments of EPA. The Office gave OPP's
pesticide registration and reregistration programs "results not demonstrated"
ratings in fiscal 2003, but improved the rating for the registration program to
"adequate" in fiscal 2004.
Results of Review
OPP needs to move to a better mix of output and outcome measures to assess its
performance in achieving FQPA's mandate of protecting children from pesticide
exposure risks. Although EPA has made progress implementing the requirements
of FQPA, we found that OPP has primarily measured its success and the impact
of FQPA by adherence to its reregistration schedule rather than by improvements
in children's health. While OPP has recently taken steps to develop more
outcome measures, significant challenges remain. We identified opportunities for
OPP to utilize existing data in different ways to track the effectiveness of its
pesticide regulatory decisions and program performance. We conducted an
analysis of the dietary pesticide residue data from the U.S. Department of
Agriculture's Pesticide Data Program and found that EPA's regulatory actions
had a significant impact in reducing pesticide exposure risk on domestic foods
commonly eaten by children. The Agency concurred with our recommendations.
We summarized the comments and provided our evaluations at the end of each
chapter. The full text of EPA's memorandum and comments is in Appendix A.
4
-------
Chapter 2
Improvements Needed in OPP's Measurement System
OPP launched an internal workgroup in May 2005 to develop better measures, but
none of the proposed measures have been implemented. OPP faces a number of
challenges in creating effective measures, including: the complexity of
environmental problems, external factors, cost, and an historical reliance on
output measures. Since 1996, OPP has tracked success in meeting statutory
deadlines and progress toward mandated FQPA goals. FQPA required OPP to
complete the reassessment of all 9,721 food-related pesticide tolerances by
August 2006. While important and required, these output measures, which
comprise the majority of OPP's performance measurement system, do not
measure impact. Because it lacks measures on the impact of actions on the health
of infants, children, and the overall human population, OPP cannot state the
impact of its FQPA efforts. Several opportunities for OPP to improve
performance measurement are discussed in this chapter and Chapters 3 and 4.
OPP's Current Measurement System Focuses on Outputs
A good performance measurement and reporting system ensures transparency and
holds an organization accountable. OPP adopted transparency and accountability
as goals, and further noted that accountability is paramount to the development of
effective performance measures.
OPP uses the graphic in Figure 2.1
to illustrate its view on the
importance of performance
measures as accountability tools
for a number of purposes. The
measures ensure that OPP provides
stakeholders and the Agency with a
cohesive display of the program.
OPP intends to use performance
measures in most if not all of the
areas indicated in the figure. OPP
uses performance measures as part
of EPA's strategic planning. In the
Agency's overall structure, OPP's
FQPA-related work falls under
Goal 4: Healthy Communities and
Ecosystems.
Output measures were important to OPP during FQPA implementation in terms of
meeting congressionally mandated deadlines and timeframes. While output
Figure 2.1: Performance Measures for Accountability
e ports
Measures
Office-wide
Work Plans
COUNTAglLITY
Guidance to
Regions
Strategic
Plan OMB
t0 Part
Congress
Budget
/Performance
/ Accountability/pprlhrTnanrp \Operating Plar
Reports
Employee
Standards
PARS Guidance
to States
and
Tribes
Qni ima- OPP
5
-------
measures are needed, they comprised all but two of OPP's performance measures
in the Fiscal Year 2005 annual report. The lack of outcome-based measures
impedes OPP's ability to assess the impact of its actions.
For Fiscal Year 2005, OPP used the following output measures to assess
programs:
Ķ Cumulative number of safer chemicals/biopesticides registered.
Ķ Cumulative percentage of the 9,721 tolerances required to be reassessed over
10 years that have already been reassessed.
Ķ Cumulative number of new uses.
Ķ Number of inert ingredients in pesticide products reregistered.
Ķ Cumulative number of new chemicals registered.
Ķ Cumulative percentage of Reregistration Eligibility Decisions completed.
Ķ Percentage of tolerance reassessments issued for the "Top 20" foods eaten by
children.
Ķ Children's exposure data and tools for assessing aggregate exposure to
residential use pesticides.
Ķ Number of product reregistrations.
One of the strengths of OPP's current output measures is that the majority are
based on actual counts, not on modeling or predictions. Staff can easily collect,
compute and analyze the data. However, one of the major weaknesses is that OPP
staff and other data users cannot draw conclusions, trends, or significant analyses
from the measures about the impact of actions. While output measures are
important as an internal program management tool, their value in illustrating
actual programmatic success and impact is severely limited. Counts of
registration and reregistration numbers alone do not provide evidence that
children's health benefited from a reduction in pesticide exposure risk.
OPP's two outcome-based measures for FY 2005 were:
1. The reduction of detections on a core set of 19 foods eaten by children relative
to detection levels for those foods reported in 1994-1996; and
2. The percentage of acre treatments with reduced risk pesticides.
Both were low-level outcomes in the hierarchy of measures,2 but nonetheless
provided a clearer picture to OPP of the impact of its FQPA-related actions than
the output structure.
2 Performance measures can be categorized along a "hierarchy" of measures. Levels 1 and 2 measure administrative
actions and program activities (outputs), while levels 3 through 5 represent intermediate outcomes and level 6
represents long-term outcomes. As the measures progress from levels 3 to 6, the association of the indicator to
environmental outcomes strengthens. While level 6 outcomes are important, they are less feasible for measurement
than levels 3 through 5.
6
-------
OPP staff said the greatest outcome from implementing FQPA is the reduction in
household poisonings. By removing household uses of many toxic pesticides,
OPP reduced the potential for children's exposure to those pesticides. However,
OPP does not have measures to show reduction in household poisonings or use
patterns. In its 2004 annual report, OPP cited two success stories that lacked
measures with which to capture their impact. One involved the elimination of a
product (thiram) from almost all residential uses. OPP also noted that EPA
pesticide reregistration decisions in response to FQPA resulted in removal of
15 million to 20 million pounds of organophosphates from use in and around
homes annually. These successes, however, cannot be inferred by the general
public and stakeholders from any of OPP's performance measures.
Outcome Measures under Development
In May 2005, OPP established an internal workgroup to develop results indicators
for its program as a whole. OPP has had one full time person working on
performance measures since the inception of the workgroup, but other staff
members have devoted significant amounts of time in developing the initiative.
The workgroup has proposed measures (see Appendix B), but these measures
have not yet been implemented. Under this performance measurement project,
OPP has identified the following overall strategic measures for reducing risk to
the general public:
Ķ Reduce the number of acute poisoning incidents from pesticides in and around
the home.
Ķ Reduce the level of currently registered pesticides in the general population.
Ķ Reduce pesticide residues in the 20 foods most commonly eaten by children
using Pesticide Data Program residue data.
OPP Faces Challenges in Creating Effective Performance Measures
OPP faces a number of challenges in creating effective, outcome-based
performance measures. These include: the complexity of environmental
problems, external factors, data housed in other Federal agencies, the expense of
collecting new human health data, and the historical reliance of the organization
on output-based measures.
OPP, like EPA as a whole, faces significant performance measurement challenges
related to environmental problems. Data on environmental conditions and health
effects of pollutants are limited. Further, there are difficulties in linking a
program's activities and the resulting changes in the environment. Numerous
factors beyond EPA's control, such as technological change and socioeconomic
factors, play a role. Although EPA has volumes of data on individual pesticides,
OPP is often prohibited from releasing it due to confidential business information.
7
-------
OPP needs to obtain a great deal of its data from other Federal agencies, and thus
is reliant on the type of data those other agencies collect. The U.S. Department of
Agriculture (USDA) provides data on food consumption, food commodity, and
pesticide residue through such databases as the Food Commodity Intake
Database. FQPA contains specific provisions for cooperative activities between
EPA and USDA. Since 1999, USDA integrated its food intake survey with
another large survey - the National Health and Nutrition Examination Survey -
conducted by the U.S. Department of Health and Human Services. The main data
source for dietary residues is the monitoring study conducted by the Pesticide
Data Program at the USDA Agricultural Marketing Service. This program has
generated extensive pesticide residue data on over 50 foods eaten daily, including
data on pesticide residues in fruits, vegetables, grains, dairy products, and meats.
OPP recognizes that it can obtain valuable data from such sources as the National
Health and Nutrition and Examination Survey and Pesticide Data Program, and it
needs to maximize its opportunities to obtain and use this information.
OPP's mission is not one of zero risk or zero exposure, which makes it difficult to
set ambitious, aggressive goals. This is compounded by the legality of pesticides;
they are not "end of the pipe" pollution, but substances legally in the
environment. OPP must balance its dual mission of providing a gateway to the
marketplace for pesticide products with the protection of the public from harmful
pesticide exposures. Further, determining attribution for changes in pesticide
residue in humans as a result of EPA actions taken is problematic.
FQPA Logic Model Can Improve Performance Measuring
Logic models and performance indicators are tools to provide better performance
measures and, thus, program management decision making. Our prior January
10, 2006 FQPA report (2006-P-00009) addressed how logic models distinguish
outputs and outcomes in program design. OPP developed logic models for
individual programs during its current performance measurement initiative, but
did not develop a logic model for the pesticide program as a whole.
The logic model in Figure 2.2, prepared by the EPA OIG, provides an overarching
picture of OPP's FQPA-related activities and potential outcomes. Those
measures, activities, and resources in orange boxes are currently used by OPP.
Those in green boxes are proposed and/or prospective measures. Some of the
measures are proposed by OPP (shown in Appendix B); others come from
literature related to pesticide performance measurement. The logic model
illustrates the potential flow from activities through long-term outcomes from
OPP actions.
8
-------
Figure 2.2: Logic Model on FQPA Implementation and Children's Pesticide Exposure Health Risk Mitigation
Resources
OPP
Risk Assessors
Risk Managers
EPA Offices
External
Stakeholders
(USDA3CDC,
FDA)
Activities
Risk Assessments
Risk Management
Decisions
Exposures Research
& Toxicity Testing
Data collection on
Children's Food
Intake & Activities
Food Pesticide
Residue Testing
Strategic Planning
Chemical
Nominations for
Biomonitoring
Development of
work plans
Outreach and
communication
Outputs
Level 1 & 2 Indicators
Cumulative number safer chemicals/
biopesticides registered
Cumulative percentage of the 9721 tolerances
required to be reassessed over ten years
reassessed
Cumulative number of new uses
Number of inerts reregistered
Cumulative number new chemicals registered
Cumulative percentage of Reregistration
Eligibility Decisions (REDs) completed
Percentage tolerance reassessments
completed for the Top 20 foods eaten by
children
Number of assays standardized and validated
Children's exposure data and tools for
assessing aggregate exposure to residential
use pesticides
Efficiency: Reduction in decision times for
reduced risk chemicals, new conventional
Product reregistrations
Number of public awareness campaigns about pesticide hazards and safe application and use;
Amount of public and professional education about symptoms of low-level pesticide
exposure;
Number of jurisdictions in which "organic" foods are available
Immediate Outcomes
Level 3 Indicators
Percentage of acre-
treatments with reduced
risk pesticides
Percentage of occurrences
of residues on a core of
19 foods eaten by children
relative to occurrence
levels for those foods
reported in 1994-1996
Percentage increased
market shares of reduced
risk pesticides
Percentage of crops and/or
acres in transition to
Integrated Pest
Management and
reduced-risk pesticides
Average toxicity of
pesticide active ingredient
applied per acre
Number of agricultural
acres treated with
biopesticides
Annual pesticide use on
select field crops by
pesticide product signal
word
U.S. annual volume of
pesticide usage by type of
active ingredient
Intermediate Outcomes
Level 4 and 5 Indicators
Long-Term Outcomes
Level 6 Indicators
Percentage reduction in children with
detectable pesticide levels in the body
Percentage reduction in pesticide
concentrations in environmental
media
95th percentile blood and urine
concentration levels for biomarkers
of exposure to pesticides
Percentage reduction in children's
systemic poisoning incidences at
homes, schools or public places
Percentage reduction in foods
sampled with detectable pesticide
residues
Percentage reduction in foods
sampled with pesticide residues that
violated or were presumed to violate
tolerances
Incidence of pesticide-related
poisonings and illnesses in pesticide
workers
Number of non-occupational
pesticide-related poisoning and
illness
Reduce the number of acute
poisoning incidents from pesticides in
and around the house
Percentage risk reduction to children
from:
> Pesticides in ambient air
> Pesticide residues in drinking
water and in food
> Pesticide used in the home
Percentage
reduction in levels
of currently
registered
pesticides found
in children
Percentage
reductions in the
level of currently
registered
pesticides in the
general population
Percentage
reduction in
vector borne
disease in children
Improvements in
children's and
public health
Production of
safer pesticidal
chemicals
Percentage
reduction in
pathologies in
children caused by
pesticide exposure
Customers
Congress, Office of Management and Budget, internal stakeholders, other federal agencies, State/Tribal environmental agencies, universities/academic institutions,
nonprofit research institutions, industry, agri-community, nongovernmental environmental groups, and public health professionals
OIG staff developed this logic model based on data collected for this evaluation.
9
-------
EPA Should Consider Revising Goal Structure
The structure of EPA's Goal 4 is currently not set up to accomplish or measure
human health or environmental outcomes. OPP has the potential to measure
ambient conditions, body burdens, quantities and toxicities of pesticides sold, and
residue levels, but cannot currently tell the public anything about these elements
because the current goal structure does not reflect the fact this is part of its
mission. OPP said that the existing structure is highly output-oriented, and does
not correlate well with OPP's three major mission areas (see Figure 2.3).
OPP has proposed a new goal structure for its portion of Goal 4 that outlines
changes for three major mission areas - human health, the environment, and other
benefits - as shown in Figure 2.3:
Figure 2.3: Existing and Proposed Goal 4 Structure for EPA Strategic Architecture
Goal 4: Healthy communities and ecosystems
Existing Structure
Proposed Structure
4.1 Chemical, organism, and pesticide
risks
4.2.1 Protect human health from
pesticide risk
4.1.1 Reduce exposure to toxic
pesticides
4.2.2 Protect the environment from
pesticide risk
4.1.2 License pesticides meeting safety
standards
4.2.3 Realize the benefits from pesticide
use
OIG staff developed figure based on data collected during evaluation.
The proposed strategic plan structure is more outcome-oriented, and better
addresses OPP's strategic goals. OPP wants the goal structure to acknowledge
OPP's dual role as a gateway to the market for pesticide products and as a steward
of human health and the environment. The proposed goal structure reflects this
dual role. OPP officials have expressed this concern to the Office of the Chief
Financial Officer in the past. The 2006-2011 EPA Strategic Architecture drafted
by the Office of the Chief Financial Officer incorporated OPP's proposed
structure and strategic goals, and allows for the development and use of outcome-
based performance measures.
One problem with the existing goal structure is that the reregistration program
will be ending in August 2006, turning into the registration review process, as
outlined under FQPA. Although the registration and reregistration processes
provide support for what OPP does, they are activities, not outcomes. OPP's
performance measures and budget will be merged by the end of this process, and
the proposed goal structure better allows for the merging of measures, goals, and
budgeting.
10
-------
Recommendations
We recommend that the Director, Office of Pesticide Programs:
2-1 Continue to move away from focusing primarily on outputs when
examining the human health and environmental impacts of work by
continuing the current measurement initiative.
2-2 Implement the following three human health performance measures in the
next round of strategic planning:
Ķ Reduce the number of acute poisoning incidents from pesticides in
and around the home.
Ķ Reduce the level of currently registered pesticides in the general
population.
Ķ Reduce pesticide residues in the 20 foods most commonly eaten by
children using Pesticide Data Program residue data.
2-3 Work with other EPA program offices and other Federal agencies to
ascertain supporting data for new measures of FQPA results.
We recommend that the Chief Financial Officer:
2-4 Revise the Goal 4 structure in the next round of strategic planning to
create a more outcome-oriented goal structure and to acknowledge OPP's
dual role as a gateway to the market for pesticide products and as a
steward of human health and the environment.
Agency Response and OIG Evaluation
The Agency agreed with the recommendations provided in Chapter 2, and
believes that many of our suggested measures support its current system of
measurement. Appendix A provides the full text of the Agency's response.
11
-------
Chapter 3
Suite of Performance Measures Can Help OPP
Better Measure Impact
By integrating data into a suite of performance measures, OPP can better utilize
existing data to track the effectiveness of pesticide regulatory decisions and
program performance. We identified several pools of quantitative data available
for use as performance indicators. To manage FQPA performance indicators
effectively, coordination between OPP staff, other EPA offices, and external
entities is needed. Developing a suite of performance measures will facilitate
such an approach.
Using a Suite of Performance Measures Can Help OPP Track Progress
While OPP has limited in-house data on human health effects, there are numerous
databases and data sources external to EPA that can provide information for
developing health-based outcome measures. Integrating this data into a suite of
performance measure can better enable OPP to measure progress.
For example, biomonitoring data are helpful in tracking trends and extremes of
pesticide exposure. Biomonitoring is the term given to the analysis of biological
samples (e.g., human blood, bodily tissues and fluids, and breast milk) to identify
the presence and levels of specific substances in the body. By comparing levels
in one individual to normal levels in the general population, and then comparing
them to levels recognized by the medical community to cause harm, scientists and
public health officials are able to make more accurate and effective decisions to
prevent illness and protect the public. However, biomonitoring data alone do not
constitute a complete pesticide exposure risk assessment, since exposure to
multiple pesticides from many sources rather than just one pesticide from one
source is a routine part of life for children.
Integrated environmental monitoring and health tracking systems that employ
well-validated performance indicators can enable EPA officials to scan for
potential causes or triggers of changes in trends and patterns. The following is a
list of existing data sources from which OPP can pull integrated outcome
performance measurement information:
12
-------
The National Report on Human Exposure to Environmental Chemicals, from
National Health and Nutrition Examination Surveys
Pesticide residual data from the:
o Pesticide Data Program managed by USDA's Agricultural Marketing
Service
o Total Diet Study, sometimes called the Market Basket Study, by the
Center for Food Safety and Applied Nutrition of the Food and Drug
Administration within the Department of Health and Human Services
The Agricultural Health Study
Research data from the various EPA-funded Centers for Children's
Environmental Health and Disease Prevention Research
Poison Control Centers Data
Pesticides Industry Sales and Usage Data
National Home & Garden Pesticide Use Survey
OPP is currently looking for new opportunities to gather data for non-
occupational exposures, including what can be targeted in future rounds of the
National Health and Nutrition Examination Surveys. OPP plans to target groups
of chemicals by class to provide a historical, retrospective picture of OPP impact.
In addition, OPP plans to do more collaborative work with the other EPA offices,
such as the Office of Water, to ascertain whether there is any additional data
in-house that OPP could use in developing FQPA performance indicators.
Although these data sources are available to OPP, we found little evidence to
suggest OPP mines datasets to uncover less-obvious human health effects, such as
developmental health risks or long-term diseases. Although some data from the
list above could be used as stand-alone performance indicators, others should be
grouped during analysis to uncover effects from regulatory decisions or complex
interactions requiring regulatory interventions. This grouping of measures will
provide a "suite" of measures from which OPP can garner trends and impact.
Using a suite of FQPA performance measures would demonstrate OPP
performance in mitigating children's pesticide exposure risk and enhancing public
health from reduced risk pesticide usage.
In Table 3.1, we outline potential FQPA performance measures from existing data
sources and how these measures can be used in a suite. In the absence of ideal
end outcome measures of human health, the Agency could employ a suite of
surrogate measures. For example, dietary consumption data from the National
Health and Nutrition Examination Survey can be paired with pesticide residue
data analyzed by the USDA's Pesticide Data Program and the Food and Drug
Administration's Total Diet Study to determine pesticide exposures from
ingestion. Examining these measures along with information on the frequency of
foods consumed with nonviolative pesticide residues could confirm effectiveness
of OPP's pesticide tolerance determinations.
13
-------
Table 3.1: Examples of Suites of Measures to Illustrate Children's Pesticide Exposure Levels,
Body Burdens, and Possible Health Effects
Data
Sources
Possible Measures
(Using Existing Data)
Potential Measures
(No National Data
currently exists)
Indicators of:
Protection of
children's health
Impact of OPP
actions on use
Impact of FQPA
on behavior
Pesticide Residual Data
from Agricultural
Marketing Service,
USDA's Pesticide
Data Program
Dietary risk trends from
domestic vs. imported foods.
Pesticide exposure risk from
drinking water and water
treatment processes.
Effects of regulatory decisions
in mitigating risk sources.
Risk-drivers from chemical
trading or substitutions.
Meta measures: Food
and water consumption
data linked to pesticide
residue analysis and
violations.
V
Dietary: Total Diet Study
& Food Market Basket
Surveys
Compliance with tolerances.
National Health and
Nutrition Examination
Survey
Measure of contaminants and
body burdens.
Pesticide residue levels in
drinking water quality.
Residential dust/ambient
exposures.
Dietary consumption
information matched with
biomarker and pesticide
residue information.
Pesticides and their
metabolite levels in
Human umbilical cord
blood.
Birth outcome (weight
and length).
Paraoxonase 1 (PON1)
activity.
Human breast milk.
V
V
V
Poison Control Centers
Data
Acute illnesses associated
with pesticide exposures at
schools and municipal parks.
Schools and municipal
parks participating in
Integrated Pest
Management programs.
V
V
Farm Family Pesticide
Exposure Data: Farm
Family Pesticide
Exposure Study and
Studies in the
Agricultural Health
Study
Health effects among farmers'
wives at reproductive age.
Birth weights, birth defects,
and asthma/chronic
respiratory diseases among
farmers' children.
Spray drift and acreage
treatment data.
Birth outcome data.
Asthma data.
V
V
Pesticides Industry
Sales and Usage Data
Risk-drivers from chemical
trading or substitutions.
Economic profile/regulatory
impacts on industry.
Market share of reduced
risk pesticides.
V
National Home &
Garden Pesticide Use
Survey
Trends from self-reported
home and garden pesticide
use data
Trends in residential usage,
reduced risk pesticide sales,
and acute poisoning data.
Self reported residential
usage data.
V
V
OIG staff developed figure based on data collected during evaluation.
14
-------
Five Examples Demonstrate Use of Suite Approach
The following five examples show how a suite of performance measures using
external sources could help OPP track regulatory progress and demonstrate
effectiveness of FQPA-relevant decisions. Each example focuses on observable,
quantifiable exposure and offers a long-term strategic picture of the gap between
regulatory actions and outcomes of children's health. Collectively, they offer
broad measures of the effectiveness of the pesticide programs' aggregate effects
on children's health. In addition, some of these examples focus on the type of
developmental and exposure data that may be necessary for future pesticide
registrations.
Example 1: Research results reported by the Centers for Children's
Environmental Health and Disease Prevention Research3 illustrate the importance
of using a suite of measures to highlight regulatory pesticide exposure risk
mitigation efforts by EPA. Three recent studies4 appearing in the journal
Environmental Health Perspectives examined the relationship between pregnant
women's exposures to selected pesticides and birth outcomes. Participants in
each study were likely exposed to many classes of pesticides as well as other
environmental chemicals, but the focus of these studies was on organophosphorus
pesticides.
Example 2: One major advantage of using biomarker data from the National
Health and Nutrition Examination Survey is that the data provide an ongoing
assessment of the U.S. population's exposure to environmental chemicals using
biomonitoring. Centers for Disease Control and Prevention scientists measure
chemicals or their metabolites (breakdown products) in blood and urine samples
from selected participants in each survey, and the chemical analysis of
environmental monitoring samples collected during each survey. Analytical
results are published in the National Reports on Human Exposure to
Environmental Chemicals. Human exposure data from the survey would allow
EPA officials to track children's body burden trends and determine whether past
and current regulatory actions are effective in reducing or mitigating pesticide
exposure risks for children and women of childbearing age.
Example 3: In Chapter 4, we noted the use of an empirical approach to examine
and document OPP's performance in mitigating dietary pesticide exposure risks
for children from domestic foods that they consume frequently. If OPP carries
our analysis one step further, it could track actual consumption risks by
3 Funded by EPA in partnership with the National Institute of Environmental Health Sciences and the Centers for
Disease Control and Prevention.
4 Berkowitz, Gertrud S et al. In Utero Pesticide Exposure, Maternal Paraoxonase Activity, and Head Circumference.
Environmental Health Perspectives 112:388-391, 2004. Eskenazi, Brenda et al. Association of in Utero
Organophopshate Pesticide Exposure and Fetal Growth and Length of Gestation in an Agricultural Population.
Environmental Health Perspectives 112:116-1124, 2004. Whyatt, Robin M et al. Prenatal Insecticide Exposures
and Birth Weight and length among an Urban Minority Cohort. Environmental Health Perspectives 112:1125-1132,
2004.
15
-------
integrating food and water pesticide residue measures from USDA's Pesticide
Data Program with the National Health and Nutrition Evaluation Survey data on
children's actual food consumption amounts and urinary biomonitoring.
Example 4: External stakeholders continue to express concern over the health of
farm workers and their families from pesticide exposures. Two pools of data are
available to EPA for using real-world pesticide exposure information on farmers
and their families to track progress in pesticide exposure risk management: the
Farm Family Pesticide Exposure Study and the substudies of Agricultural Health
Study. Such studies are epidemiological-based and use health outcomes, along
with biomonitoring of pesticides in urine, to quantify the exposure of
farmers/applicators and their families. Also, the Centers for Children's
Environmental Health and Disease Prevention Research conduct research on farm
families and their children.
Example 5: EPA collects information about pesticide poisonings by a variety of
mechanisms. In 2004, approximately 70,000 children were accidentally exposed
to or poisoned by pesticides. OPP has used Poison Control Centers' data to show
information on how EPA is improving protection from acute adverse effects of
pesticide exposure. OPP receives mandated reports of adverse effects from
manufacturers, and periodically reviews both the Toxic Exposure Surveillance
System data maintained by the American Association of Poison Control Centers
and aggregated data from State-based surveillance programs. A 2005 study was
published in the Journal of the American Medical Association that illustrated the
importance of tracking risks of pesticide use in and around the nation's schools.
We believe EPA can further analyze available data from State and toxic exposure
surveillance systems to extract pertinent pesticide poisonings trends in schools
and childcare centers.
Recommendation
We recommend that the Director, Office of Pesticide Programs:
3-1 Evaluate the use of suites of performance measures to more
comprehensively assess OPP's FQPA implementation performance and
impacts on children's health.
Agency Response and OIG Evaluation
The Agency agreed in general with the recommendation provided in Chapter 3.
OPP noted that measures are only as good as the data upon which they are based,
and have concerns that such data do not exist for some of our recommended
measures. OPP additionally stated that it will consider the measures provided and
further evaluate the recommendation as they make progress in implementing
FQPA performance measures. Appendix A provides the full text of the Agency's
response.
16
-------
Chapter 4
OPP Decisions on Pesticide Dietary Exposure Risk
Had Positive Impact
OPP has not yet stated the impact of its performance under FQPA on children's
health. As discussed in Chapter 2, this occurred because OPP did not have
adequate measures. To get a general idea on the impact of OPP's actions on
children's health based on the FQPA, we examined pesticide dietary risk. Dietary
risk is based on both the exposure to a pesticide and the pesticide's toxicity. We
conducted an analysis of the dietary pesticide residue data from USDA's Pesticide
Data Program and found that EPA's regulatory actions had a significant impact in
reducing pesticide exposure risk. Risks associated with 16 foods commonly eaten
by children declined almost 50 percent. However, while we noted positive
changes for domestic foods, there has been a shift of risk to imported foods.
FQPA Actions Decreased Dietary Pesticide Risks to Children
Using USDA's Pesticide Data Program data, we found that risks associated with
16 foods commonly eaten by children declined by almost 50 percent.
Specifically, as illustrated in Figure 4.1, the total dietary risk index amount
(domestic and imported combined) decreased from 3,170 in 1994 to 1,532 in
2003.5
Figure 4.1: Total Dietary Risk Index Scores for Selected Children's Foods6
( I
Total Dietary Risk Index- Combined
3500
8
Ķo 3000
c
ĶĢ 2500
(A
Ģ
2*
ĶĢ 1500
2000
Id
Q
1000
15
500
& J? c# r# A?
Year
W
5 2003 is currently the last year for which data is available through USDA's Pesticide Data Program.
6 Certain years included in our sample lacked data. For these years, the trend data and data points are extrapolated.
Details on dietary risk index values for selected foods are in the supplemental report.
17
-------
To evaluate the impact of the FQPA on dietary pesticide exposure risk for
children, we analyzed data on food intake and exposure to food pesticide residues.
The methodology we developed is discussed in our supplemental report. The
Dietary Risk Index is a basic unit of measure the OIG team used to track pesticide
dietary risks for food commonly consumed by children. Our index values are
based on risk assessment methods and publicly available toxicological data
supporting EPA dietary risk assessment.
We found risks have declined by about two-thirds in domestically grown foods in
16 important children's foods7 included in our analysis. Figure 4.2 illustrates that
between 1996 (when FQPA was implemented) and 2003, the average Dietary
Risk Index values across the 16 domestically produced foods declined from
175 to 65, or about 63 percent.
Figure 4.2: Average Total Dietary Risk Index Values for Selected Foods
~ Domestic
Ķ Import
~ Combined
1994 1995 1996 1997 1998 1999 2000 2001 2002 2003
Years
A similar analysis was conducted on pesticide residues in imported samples.
Although there was a decline in the total imported food dietary risk index, the
decline was not nearly as significant as it was for domestic foods. On a serving-
compared-to-serving basis, dietary risks in 1996 were roughly comparable for
imported and domestic foods. By 2003, however, total pesticide residual risk for
imported foods were nearly four-times higher than those of the domestic scores
(see Figure 4.3).8 Breakdowns for specific domestic and imported food
commodities are in the supplemental report.
7 Apple juice, apples, broccoli, cantaloupe, carrots, celery, cucumbers, grapes, green beans, lettuce, oranges,
peaches, potatoes, spinach, sweet bell peppers, and tomatoes. These foods were selected based on the amount of
data points in the Pesticide Data Program database, and the frequency with which they are consumed by children.
8 The word "total" in our analysis refers to the "total" summed values of the risks we assessed using the Pesticide
Data Program data. It is important to note that USDA's sampling is limited. Therefore, "total" does not reflect all
foods, but simply the sum of the risk from our analysis and USDA's sampling.
18
-------
Figure 4.3: Comparisons of Total Pesticide Dietary Exposure Risks in Domestic
and Imported Foods
2000-1
-------
Chlorpyrifos was one of the most widely-used insecticides in the United
States, and was commonly found in many home-and-garden insecticides.
Additionally, it was used in some termite treatments and on agricultural
crops. In June 2000, EPA released a revised risk assessment and
announced an agreement with registrants to phase out and eliminate
certain uses of chlorpyrifos. This action eliminated home, lawn, and
garden uses by the end of the 2000, as well as all termite-control uses in
existing homes and for new home-and-building construction by the end of
2004. Additionally, the use of chlorpyrifos for all sensitive areas, such as
schools, day cares, parks, hospitals, nursing homes and malls, were
eliminated by the end of the 2000. EPA also canceled the use of
chlorpyrifos on tomatoes and restricted the use on apples.
Looking at Pesticide Data Program data, we found a single pesticide like
chlorpyrifos accounted for 70 percent or more of total dietary risk contributions in
apples grown in the United States between 1999 through 2002. As Figure 4.4
illustrates, risk associated with chlorpyrifos on apples has declined dramatically.
Figure 4.4: Changes on Dietary Exposure Risk from Chlorpyrifos for Apples
Chlorpyrifos
140 -
120 -
100 -
Table 4.1 illustrates that the contribution of chlorpyrifos to total dietary pesticide
risk decreased to only 8 percent in domestically grown apples, with the remaining
risk coming from newer, lower-risk pesticides.
20
-------
Table 4.1: Chlorpyrifos as a Single-Risk Driver in Domestically Grown Apples
Year
Total Pesticides
Dietary Exposure Risk
in Apples
Dietary Exposure Risk
Chlorpyrifos/Apple
Combination
%of
Total
(Dietary Risk Index Values)
1999
180.2
140.2
78%
2000
144.3
112.0
78%
2001
144.1
105.2
73%
2002
43.7
3.6
8%
In Table 4.2, we show that EPA's August 1999 regulatory actions against the
parathions resulted in a drop of about 83 percent of the total impact in domestic
foods triggered. Changes in chlorpyrifos tolerances reduced total Dietary Risk
Index values by another 241 points. Taken together, tolerance revocations and
reductions imposed on 8 uses of the parathions and chlorpyrifos accounted for
98 percent of the total impact of EPA actions to date on a set of 30 of the most
serious domestic "risk drivers." Risk drivers are high risk pesticide-food
combinations accounting for the majority of summed pesticide dietary exposure
risk. Details are in the supplemental report.
Table 4.2: EPA Regulatory Actions Decrease Dietary Pesticide Exposure Risks
Pesticide
Aggregate Dietary Risk Index/
Year before EPA Action
% Reduction
(For Domestic Food Samples)
Parathions
1,369
83%
Chlorpyrifos
241
15%
Total
1,610
98%
Method Could Help OPP
The Dietary Risk Index described in this chapter is one method for OPP to
consider in assessing the impact of its performance. This index could be used in
combination with the proposed measures and suites of measures recommended in
Chapters 2 and 3. We recognize that there are limitations, including cost and data
collection constraints, to OPP's use of this methodology. Nonetheless, we believe
it is important for OPP to take action to improve its ability to document impacts
of the FQPA on dietary risk levels for foods that contribute significantly to the
diets of infants and children.
21
-------
Recommendation
We recommend that the Director, Office of Pesticide Programs:
4-1 Develop a cost-effective and scientifically defensible method to assess its
performance in decreasing dietary pesticide exposure risks to children
from its tolerance assessments and reassessments.
Agency Response and OIG Evaluation
The Agency agreed in general with the recommendation provided in Chapter 4.
OPP noted that under the Federal Food, Drug, and Cosmetic Act, it has regulatory
authority to reduce a tolerance only if dietary exposure risks do not meet the
"reasonable certainty of no harm" standard established by FQPA. However, OPP
also stated that an exposure-based measure utilizing USDA's Pesticide Data
Program data offers the best alternative at this time, and that it is always
interested in exploring ways to measure the human health and environmental
impacts of our work based upon risk-based measurements.
Appendix A provides the full text of the Agency's response.
22
-------
Status of Recommendations and
Potential Monetary Benefits
RECOMMENDATIONS
POTENTIAL MONETARY
BENEFITS (In $000s)2
Rec.
No.
Page
No.
Subject
Status1
Action Official
2-1
2-2
2-3
Continue to move away from focusing primarily on
outputs when examining the human health and
environmental impacts of work by continuing the
current measurement initiative
Implement the following three human health
performance measures in the next round of
strategic planning:
Reduce the number of acute poisoning
incidents from pesticides in and around the
home
Reduce the level of currently registered
pesticides in the general population
Reduce pesticide residues in the 20 foods
most commonly eaten by children using
Pesticide Data Program residue data
Work with other EPA program offices and other
Federal agencies to ascertain supporting data for
new measures of FQPA results
Director, Office of
Pesticide Programs
Director, Office of
Pesticide Programs
Director, Office of
Pesticide Programs
Planned
Completion
Date3
Claimed
Amount
Agreed To
Amount
TBD
TBD
TBD
2-4 11 Revise the Goal 4 structure in the next round of
strategic planning to create a more outcome-
oriented goal structure and to acknowledge OPP's
dual role as a gateway to the market for pesticide
products and as a steward of human health and the
environment
The Chief Financial Officer
TBD
3-1 16 Evaluate the use of suites of performance
measures to more comprehensively assess OPP's
FQPA implementation performance and impacts on
children's health
Director, Office of
Pesticide Programs
TBD
4-1 22 Develop a cost-effective and scientifically
defensible method to assess its performance in
decreasing dietary pesticide exposure risks to
children from its tolerance assessments and
reassessments
Director, Office of
Pesticide Programs
TBD
1 0 = recommendation is open with agreed-to corrective actions pending;
C = recommendation is closed with all agreed-to actions completed;
U = recommendation is undecided with resolution efforts in progress
2 Identification of potential monetary benefits was not an objective of this evaluation.
3 In accordance with EPA Manual 2750, the Agency is required to provide a written response to this report within 90 calendar days that will include a
corrective actions plan for agreed upon actions, including milestone dates.
23
-------
Appendix A
Agency Response
MEMORANDUM
SUBJECT:
OPP's Comments on the OIG's Draft Evaluation Report
"Measuring the Impact of the Food Quality Protection Act: Challenges
and Opportunities" (Assignment No. 2005-001034)
FROM:
Jim Jones, Director
Office of Pesticide Programs
TO:
Jeffrey K. Harris, Director for Program Evaluation, Cross Media
Office of Inspector General
This memorandum is in response to the recommendations made by the Agency's Office of
Inspector General (OIG) in its April 13, 2006, report evaluating EPA's implementation of the
Food Quality Protection Act (FQPA). The report, the final in a series of three planned reports,
comments on OPP's ongoing work to develop better performance accountability measures.
The EPA's OIG report, "Measuring the Impact of the Food Quality Protection Act: Challenges
and Opportunities," focuses upon the following issues:
What are the strengths and weaknesses of OPP's current measurement system in tracking
FQPA objectives, and how could it be improved, if necessary?
What impact did FQPA have on mitigating dietary pesticide exposure risk and on
children's health?
Our response is organized as follows: The first section summarizes our responses to the OIG's
specific recommendations. The second section contains more detailed comments on the text of
the report.
24
-------
1. OPP's Responses to the OIG's Recommendations
OIG Recommendations
Chapter 2 - "Improvements Needed in OPP's Measurement System."
OIG Recommendation 2-1: Continue to move away from focusing primarily on outputs when
examining the human health and environmental impacts of work by continuing the current
measurement initiative.
OPP Response: OPP agrees with this recommendation. OPP agrees that our performance
measures in the past have primarily been output measures. Although we recognize that there will
always be a need to track such outputs, we are striving to develop more direct and effective ways
to assess the human health and environmental impacts of our decisions.
OPP recently conducted a multi-year effort with extensive stakeholder input to develop a
comprehensive suite of outcome performance accountability measures. The final product of this
effort will be the establishment of a comprehensive and consistent set of realistic, meaningful,
and supportable outcome performance measures, and steps to ingrain those measures in the
program's activities. OPP is working to include these new measures in EPA's strategic plan, and
to begin implementation of measures and collection of baseline data.
OIG Recommendation 2-2: Implement the following three human health performance
measures in the next round of strategic planning:
reduce the number of acute poisoning incidents from pesticides in and around the home;
reduce the level of currently registered pesticides in the general population; and
reduce pesticide residues in the 20 foods most commonly eaten by children using
Pesticide Data Program residue data.
OPP Response: OPP agrees with this recommendation.
The recommended measures comport with the measures OPP is recommending for the Agency's
draft strategic plan. However, with regard to the first measure (to reduce incidents in and around
the home) OPP's interest is in reducing all incidents - both human and environmental - from all
pesticide uses, not just those in and around homes.
OIG Recommendation 2-3: Work with other EPA program offices and other Federal Agencies
to ascertain supporting data for new measures of FQPA results.
OPP Response: OPP agrees with this recommendation. For many years, OPP has worked
extensively with other program offices within EPA, as well as other Federal (e.g., USDA,
DHHS/CDC) and state agencies, to use the results of their studies and data collection as
indicators in measuring results of FQPA implementation. EPA plans to continue such
collaboration.
25
-------
OIG Recommendation 2-4:
We recommend that the Director, Office of the Chief Financial Officer:
Revise the Goal 4 structure in the next round of strategic planning to create a more outcome-
oriented goal structure and to acknowledge OPP's dual role as a gateway to the market for
pesticide products and as a steward of human health and the environment.
OPP Response: OPP is in discussion with the Office of the Chief Financial Officer regarding
our dual role as gateway to the market and steward of human health and the environment as it
relates to Goal 4.
Chapter 3 - "Suite of Performance Measures Can Help OPP Better Measure Impact."
OIG Recommendation 3-1: Evaluate the use of suites of performance measures to more
comprehensively assess OPP's FQPA implementation performance and impacts on
children's health.
OPP Response: OPP appreciates OIG's suggestions for developing a proposed suite of
measures. The list of potential performance measures and examples given in Table 3.1 presents
thoughtful examples of approaches using different data sources to measure the impacts of FQPA
implementation. OPP has had a history of using information from listed and other data sources
to make more informed science and regulatory decisions. OPP has also given significant
consideration to using these data sources to formulate meaningful measures but has recognized
that data sources often have limitations (e.g., quality, quantity or focus of data) which may affect
our use of those data for valid and desirable measurements. OPP provides further discussion on
this issue below under Technical Comments.
OPP generally supports the concept of using a suite of measures to evaluate impacts. However,
measures are only as good as the data upon which they are based. In general, the data need to be:
1) of high quality, statistically representative of the population and directly related to the
measure; 2) collected over a time frame sufficient to show trends; and 3) collected in a consistent
manner to permit meaningful comparisons across different collection intervals. For the most part,
such data sources are very limited. The exceptions appear to be PDP and FDA compliance data;
Poison Control Center (PCC) data; NHANES data; and possibly pesticide sales and usage data.
A group of similar data can be pooled to provide a "snap-shot" of the impact of OPP's actions
upon public health. While a snap-shot of data cannot be used to provide information on trends, it
can be used for identifying potential new measures.
Again, OPP appreciates OIG's thoughtful development of a proposed suite of measures, and will
consider them and further evaluate the recommendation as we make progress in implementing
FQPA performance measures. We welcome new ideas and studies for finding appropriate
measures of our performance.
26
-------
Chapter 4 - OPP Decisions on Pesticide Dietary Exposure Risk Had Positive Impact
OIG Recommendation 4-1: Develop a cost-effective and scientifically defensible method to
assess its performance in decreasing dietary pesticide exposure risks to children from its
tolerance assessments and reassessments.
OPP Response: OPP generally agrees with this recommendation. It should be noted that under
FFDCA, OPP has regulatory authority to reduce a tolerance only if dietary exposure risks do not
meet the "reasonable certainty of no harm" standard established by FQPA.
OPP believes that an exposure-based measure utilizing USDA's PDP data offers a good data
source for dietary exposures. In addition, OPP is always interested in exploring ways to measure
the human health and environmental impacts of our risk based upon risk-based decisions.
1. Technical comments
Page 7: "Although EPA has volumes of data on individual pesticides, OPP is often
prohibited from using it due to confidential business information conflicts."
OPP believes this sentence requires correction. The real issue is not that we are "prohibited"
from using confidential business information, but that we are prohibited from releasing CBI
without the owner's permission due to statutory restrictions on the disclosure of CBI. In
addition, OPP may release reports with the CBI information redacted. In any case, the CBI
material is used in decision making, but ordinarily cannot be made publicly available due to
statutory restrictions on the disclosure of CBI. Further, this sentence is misleading because it
ignores the fact that safety data on registered or previously registered pesticides are explicitly
excluded from CBI protection. 21 USC 136h(d)(l).
Page 9, Figure 2.2:
OPP appreciates OIG's work and thoughtfulness in developing the "Logic Model on FQPA
Implementation and Children's Pesticide Exposure Health Risk Mitigation" Figure 2.2. OPP
does have an exposure measure for the foods most commonly consumed by infants and children
that we have been working to refine. However, OPP agrees that logic modeling can be a useful
tool in measures development, and we like the concept of a broader children's health measure.
Generally, OPP will consider the OIG's children's exposure/risk logic model as we continue to
refine measures of human health and environmental impacts of our work.
OPP has a few additional comments on the suggested logic model. Specifically, the OIG could
have included many other critical resources, such as other Federal agencies (USDA for PDP,
FDA for tolerance monitoring, CDC for NHANES), state lead agencies who have been
supportive of our measurement effort, industry, public interest groups, and other offices within
EPA, such as the Office of Water. Another key resource, especially in the area of reducing
exposure to pesticides, and mitigation of exposure risks, is OPP's outreach and communication
activities to educate the public. OPP's outreach and communication efforts should be included
under the heading of "activities."
27
-------
Page 12:
The leading sentence of the second paragraph states "While OPP has limited in-house data on the
health effects of pesticides..implies our data base on human health effects from pesticides is
moderate to inadequate and therefore understates the scope and breadth of information OPP
actually has and uses to characterize potential human health effects from pesticides. This also
discounts OPP's use of published studies. OPP recommends omitting the word "limited."
Page 12/13:
OPP agrees that several of the studies referenced in this section contain data useful for measuring
the impacts of EPA's pesticide regulatory program. For example, OPP intends to develop
measures based on the NHANES and PDP data. Data from other sources, while informative,
cannot be used to form the basis of a measure for tracking OPP's impact because they do not
collect data over different time periods that may be statistically analyzed to show changes and
trends (e.g., National Home and Garden Use Study.)
Page 14, Table 3.1:
For the data source "Family Farm Pesticide Exposure Data..In order to have an indicator of
the "Impact of FQPA on Behavior" there should be a commensurate "Impact of OPP Actions on
Use." Therefore, all three boxes should be checked in this row.
Page 15, Example 1:
Note: The three studies cited are of interest for identifying a snap-shot of success or a potential
problem area. However, because they are one-time studies, they are not useful as a measure of
long-term success. In addition, it is important to note that these studies do not provide data
demonstrating causation or information about trends, which are particularly important for
tracking progress that may be used as a meaningful measurement.
Page 15, Example 2:
OPP is currently developing an NHANES measure and we are amenable to examining whether
or not the National Report on Human Exposure to Environmental Chemicals may be useful for
such a measure.
Page 15, Example 3:
OPP has a measure using PDP data and is developing a measure using CDC's NHANES data.
OPP is interested in developing a more accurate and reliable measure using these data. We are
also interested in moving from exposure measures to risk-based measurement. To date, all risk
measures considered would require direct comparisons of different health effects.
Page 15, Example 4:
OPP supports the creation of a farm health measure and we are aware of the studies referenced in
this example. However, we are unaware of data sources that are regularly collected and
statistically robust to provide accurate and meaningful measures.
Page 15, Example 5:
As data that are regularly collected, PCC data can provide a basis for a long-term measure.
However, the PCC data has some significant limitations; most notably the number of incidents
28
-------
appears to fluctuate with funding levels of PCC's across the country. If there is less funding,
there will be fewer poison control centers, with shorter hours and less data compilation. On a
limited basis, other data can be used to augment these data.
Chapter 4, pages 17-22:
OPP believes that it has positively affected the health of children through its implementation of
FQPA. We agree that the two specific decisions mentioned were among many that contributed
to the improvement of children's health. We are also eager to develop and use a measure that
could show how our regulatory programs are affecting public health. Despite considerable
effort, OPP has been unable to develop a scientifically sound measure that uses available data
and can be implemented efficiently. At present, the most scientifically defensible approach for
assessing changes in risk appears to be repetition of the kinds of dietary risk assessments for
pesticides that OPP has conducted during the tolerance reassessment program. The repeat risk
assessments would need to use updated PDP or FDA residue data that were collected after EPA
had taken regulatory actions affecting pesticide use. At present, such an effort appears
impractical given OPP's resource levels and its statutory mandates.
OPP is in agreement that a scientifically sound, risk-based measure would provide a more
accurate and effective indicator than an exposure measure, and is conceptually preferable. Once
a draft measure is developed, it should undergo external scientific peer review and public
comment.
OPP questions the validity of the OIG risk measure used to draw the conclusions in this chapter
and the Supplemental Report. One of OPP's fundamental concerns with the "risk index"
developed by OIG is that it makes no distinction among the potential adverse effects prevented
by OPP's actions. For example, this approach could actually give a higher weight to a quickly
reversible skin irritation than to a severe birth defect. OPP believes there would be value to a
more in-depth review of the index and plans to schedule a meeting for our scientists to talk with
the creators of the proposed index.
Page 21:
The sentence: "We recognize that there are limitations to OPP's use of this methodology"
should be clarified to explain the limitations.
Supplemental Report: Details on Dietary Risk Data:
OPP appreciates OIG's time and effort in developing the proposed measures in the supplemental
report. We will consider these suggestions as we continue with development of meaningful
measures.
29
-------
Appendix B
OPP's Proposed Goal-Related Measures
ID#
Measure
Goal
Human Health
HH1
Reduce the concentration of specific
pesticides detected in the general population
by 50% by 2011
Provide aggregate picture of
effectiveness of OPP actions on
U.S. population
HH2
Measure concentration in drinking water and
raw water over time as a result of mitigation
Show impact of regulations and
mitigation on pesticides in drinking
water over time
HH3
Reduce pesticide residues in 20 foods most
commonly eaten by children using Pesticide
Data Program residue data
Show how combined efforts within OPP
have reduced pesticide exposure to
children
HH4
Ensure efficacious public health antimicrobial
products in the marketplace
Assure that antimicrobial products
registered and on market efficacious
HH5
Reduce number of acute poisoning incidents
from pesticides in and around the home
Capture reductions in number of
adverse effects associated with
residential exposure
Worker Safety
WS1
Survey of agricultural workers' awareness of
Worker Protection Standard provisions
Increase awareness of agricultural
worker protection provisions for
agricultural employees
WS2
Increase in percent-change in Worker
Protection Standard planned use compliance
ratio over time (# establishments in
compliance with Worker Protection Standard/
# inspections)
Have all employers in compliance with
Worker Protection Standard
WS3
Percent-change in number of Worker
Protection Standard violations for each
category over time (# Worker Protection
Standard violations/# inspections)
Have agricultural employers and
employees use knowledge and skills to
reduce pesticide risk in workplace
WS4
Protect those occupationally exposed to
pesticides by improving or sustaining the
extremely low rate of 3.5 or less incidents per
100,000 potential risk events.
Improve health of occupational
pesticide workers
WS5
Reduce number of certified applicators with
repeated enforcement violations
Change behavior of certified applicators
out of compliance with pesticide laws
and label directions
WS6
Improve the health of those who work in or
around pesticides by reaching a 50% targeted
reduction in moderate to severe incidents for
six acutely toxic agricultural pesticides with
the highest incident rate.
Improve health of occupational
pesticide workers
30
-------
Appendix C
Distribution
Office of the Administrator
Assistant Administrator for Prevention, Pesticides, and Toxic Substances
Assistant Administrator for Research and Development
Chief Financial Officer
Director, Office of Pesticide Programs
Acting Director, Office of Children's Health Protection
Associate Director, Field and External Affairs, Office of Pesticide Programs
Agency Followup Official (the CFO)
Agency Followup Coordinator
Audit Followup Coordinator, Office of Prevention, Pesticides, and Toxic Substances
Audit Followup Coordinator, Office of Research and Development
Audit Liaison, Office of Pesticide Programs
General Counsel
Associate Administrator for Congressional and Intergovernmental Relations
Associate Administrator for Public Affairs
Acting Inspector General
31
------- |