Ss>
i C \OIG
% ~
% PR0^S°	OFFICE OF INSPECTOR GENERAL
Catalyst for Improving the Environment
Memorandum Report
Selected Children's Health
Annual Performance Measures
for Goal 8, Sound Science
Report No. 2003-M-000017
September 26, 2003

-------
Report Contributors: Jerri Dorsey
Laurie Adams
Abbreviations
APM	Annual Performance Measure
EPA	Environmental Protection Agency
FY	Fiscal Year
GPRA	Government Performance and Results Act
OIG	Office of Inspector General
ORD	Office of Research and Development

-------

2
<
5o
o
v
^ PBO^0
ro
LU
a
&
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
OFFICE OF
INSPECTOR GENERAL
September 26, 2003
MEMORANDUM
SUBJECT:
Selected Children's Health Annual Performance Measures
for Goal 8, Sound Science
Report Number 2003-M-000017
FROM:
TO:
Jeffrey K. Harris /s/
Director for Program Evaluation, Cross-Media Issues
Paul Gilman
Assistant Administrator, Office of Research and Development
This is our final memorandum report on the subject review conducted by the Office of Inspector
General (OIG) of the U.S. Environmental Protection Agency (EPA). This memorandum report
contains findings that describe the problems that the OIG has identified and corrective actions
that the OIG recommends. This memorandum report represents the opinion of the OIG and the
findings contained in this report do not necessarily represents the final EPA position. Final
determinations on matters in this report will be made by EPA managers in accordance with
established resolution procedures.
Action Required
EPA's Office of Research and Development (ORD) provided a corrective action plan for agreed
upon actions, including milestone dates. All corrective actions have been completed. Therefore,
we are closing the memorandum report on issuance. We have no objections to the further release
of this report to the public. For your convenience, this report will be available at
http://www.epa.gov/oig/
If you or your staff have any question, please contact me at (202) 566-0831 or Jerri Dorsey at
(919) 541-3601.
1

-------
Purpose
We have reviewed a subset of EPA's Government Performance and Results Act (GPRA)
measures related to children's environmental health research. We considered the question:
To what extent does EPA accurately communicate the status of its Congressional GPRA
measures of children's health research? This was the first of a series of reviews we have
planned on environmental issues related to children's health.
Results in Brief
The status of the annual performance measures developed by ORD for Fiscal Year (FY) 2000
Annual Performance Goal 60, "develop risk assessment guidance and regional assessments
concerning risks to children exposed to environmental contaminants," was not reported
accurately. This generally occurred because personnel had varying interpretations of what
constituted completion of measures.
Background
At the time of our review, EPA had 10 long-term strategic goals1 that identified environmental
results the Agency is working to achieve. Under Goal 8, Sound Science, the Agency is charged
to develop and apply the best available science for addressing current and future environmental
hazards, as well as new approaches toward improving environmental problems. Under the
Agency's current goal structure, there are several objectives under Goal 8. One objective is to
improve the scientific basis for managing environmental exposures and risks, and developing
new models and methodologies to integrate exposure and effects findings.
Each year, as required by GPRA, the Agency prepares an annual plan that translates long-term
goals into specific actions to be taken and resources to be used during the fiscal year. These
annual plans identify what they intend to accomplish, measure how well they are doing, make
appropriate decisions based on information they have gathered, and communicate information
about their performance to Congress and to the public. At the end of a fiscal year, the Agency
develops an Annual Report that describes the year's achievements and progress in protecting
human health. Agency performance information is reported, per GPRA, in the form of annual
performance goals and associated annual performance measures (APMs). For a fiscal year,
annual performance goals represent the target level of performance expected. The Agency
develops many annual performance goals and related measures for a given year. However only
goals and measures that are identified as Congressional2 are reported in the annual report.
1	EPA's goal structure is currently being revised. The draft strategic architecture establishes five Agency
goals. Under the new goal structure, Research is not a goal but rather an objective under each of EPA's five goals.
2	Beginning with FY 2004, annual performance goals that address the most significant planned
accomplishments are called "External." External are of the level of importance corresponding to Congressional in
previous years.
2

-------
In 1998, ORD initiated the Children's Health program to support research on environmental risks
to children. The primary objective of the program is to conduct the research and provide the
methods to reduce uncertainties in EPA risk assessments for children, leading to effective
measures for risk reduction.
Scope and Methodology
ORD published a Strategy for Research on Environmental Risks to Children in 2000, which
established directions for research in children's risk and documented the rationale for the chosen
program direction. The Children's Health Research Program has been arrayed and tracked under
a series of objectives, subobjectives, and annual milestones for reporting under GPRA.
Congressional annual performance goals and APMs are expected to address the most significant
planned accomplishments, focusing on results. Congressional (externally reported) annual
performance goals and APMs are used in the Agency's Annual Performance Plan to support the
Agency's budget request in the Congressional Justification. Therefore, our review focused on
the performance goal and related measures that were identified by the ORD as Congressional.
Our review specifically focused on the children's health Congressional measures3 reported under
Goal 8, Objective 2 (Table 1).
Table 1: Focus of Review
Goal 8: Sound Science, Improved Understanding of Environmental Risk, and Greater Innovation to Address
Environmental Problems

Objective 2: Improve the scientific basis to identify, characterize, assess, and manage environmental
exposures that pose the greatest health risks to the American public by developing models and
methodologies to integrate information about exposures and effects from multiple pathways.4


Annual Performance Goal 60: Develop risk assessment guidance and regional assessments
concerning risks to children exposed to environmental contaminants



Annual Performance Measures:
1.	Assess pesticide exposure to children in Washington, Minnesota, and Arizona
2.	Report on the use of mechanistic data in developmental toxicity risk
3.	Develop exposure factors handbook for children
To assess the accuracy and presentation of EPA's GPRA measures related to children's health,
we interviewed key officials within ORD, the Office of the Chief Financial Officer, and related
grantees; and reviewed external publications as well as internal planning documents and grants.
We performed our review in accordance with Government Auditing Standards, issued by the
Comptroller General of the United States. We performed our field work from November 18,
2002, through March 28, 2003.
3	In FY 2000, ORD had 13 annual APMs, three of which were identified as Congressional. For FY 2001
and 2002, there were 13 and 10 APMs, respectively, none of which were identified as Congressional.
4	As reported in the Agency's FY 2000 Annual Report.
3

-------
Findings
EPA's Reports Were Inaccurate for Two of the Three Measures
Regarding Annual Performance Goal 60 Performance Measure 2, "Report on the use of
mechanistic data in developmental toxicity risk," ORD indicated this measure was met by
publication in 2000 of the book, Scientific Frontiers in Developmental Toxicology and
Risk Assessment. Based on our review, we concluded that this measure was reported
accurately. However, we determined that information ORD reported for Performance
Measures 1 and 3 was not sufficiently accurate. Details follow.
Annual Performance Goal 60 Performance Measure 1:
Assess pesticide exposure to children in Washington, Minnesota, and Arizona
EPA reported that "assessments of pesticide exposures to children in Washington,
Minnesota, and Arizona, were published in FY 2000." However, we found the reporting
on this APM during FY 2000s to be inaccurate. The Agency awarded three "Science To
Achieve Results" grants to accomplish this measure. Each was awarded in September
1996 and focused on children's exposures in urban and suburban settings. EPA reported
that these assessments were completed in FY 2000. However, we found that the project
period for each grant was extended beyond FY 2000, and supplemental funding was
provided for each grant after FY 2000. As illustrated in Table 2, one final report was
issued in FY 2002 and the other two were issued in FY 2003.
Table 2: Status of "Science To Achieve Results" Grants
Grant Number
Recipient
Original Project
Completion Date
Amended Project
Completion Date
Final
Report Date
R825169-01-0
University of
Arizona
9/30/1999
9/30/2001
9/04/2002
R825283-01-0
University of
Minnesota
9/30/1999
9/30/2002
5/20/2003
R825171-01-0
University of
Washington
9/30/1999
9/30/2002
7/01/2003
In reviewing grants we identified several journal articles and publications related to these
research projects; however according to the grantees and the annual progress reports for
2000, the analysis related to these grants was not completed by the end of FY 2000. For
example, one grantee progress report, covering the period of January 2000 - December
2000, indicated that more detailed analysis for all phases of the study were remaining.
We acknowledge that, over the life of a grant, a series of products, including papers and
5 During FY 2001 and FY 2002, no annual performance goals and related APMs were reported in the
annual report under Goal 8, Objective 2.
4

-------
journal articles, can be published prior to the completion of the grant. However issuance
of such documents does not indicate that the project has been completed. For example, a
grantee may publish papers throughout the entire project period related to different stages
of its analysis.
Annual Performance Goal 60 Performance Measure 3:
Develop exposure factors handbook for children
EPA reported within its FY 2000 Annual Report that, "The Exposure Factors Handbook
was not completed due to the extension of the public comment period. The final
handbook will be released in FY 2001." Subsequently, EPA reported within its 5-year
performance data report (for FY 1999-2003) attached to the FY 2003 Congressional
Justification that the handbook was released in FY 2001. We found these statements to
be inaccurate.
• The reporting of the completion of the handbook in FY 2001 was incorrect. The
external review draft of the handbook was released on EPA's web site in FY 2000.
The interim version of the handbook was placed on EPA's web site January 15, 2003.
According to ORD officials, the Office of the Chief Financial Officer was not made
aware of the delayed issuance of the handbook, which resulted in the inaccurate
information being included in the Five Year Performance review attached to the
FY 2003 Congressional Justification. According to an ORD official, the release of
the final handbook is projected for 2004.
We also found the explanation provided for the delay to be inaccurate. The annual
report stated the handbook was delayed "due to the extension of the public comment
period." However, we found that a public comment period was not held. Rather, a
peer review was conducted at EPA's request. Peer reviews, although an important
component of the review process, are not synonymous with a public comment period.
Generally, a public comment period is open to all issues whereas the peer review is
limited to technical issues.6 ORD recognized that there was no formal public
comment period; however, they provided the American Chemical Council the
opportunity to submit unsolicited comments in addition to the official peer review
that was conducted.
We recognize that there were several reasons for the delay of the handbook, including a
decision to coordinate with guidance being developed by the EPA Risk Assessment
Forum. Nonetheless, the reporting of the completion date and related reasons for delay
were reported inaccurately.
6 The goal of a peer review is to obtain an independent third party review of the product from experts who
have not substantially contributed to its development.
5

-------
ORD Has Initiated Efforts to Improve Measures
Since the development and initial reporting of Annual Performance Goal 60, ORD has examined
its approach to annual performance goals and related APM development. ORD convened a
workgroup to recommend specific improvements for enhancing the quality of performance
information in multi-year plans. The workgroup provided clear definitions for appropriate
annual performance goals and APMs, as well as development guidance. In addition, ORD issued
guidance in October 2002 on multi-year planning. This guidance explains that annual
performance goals and APMs should be quantifiable and measurable. The current guidance
stresses that every effort should be made to ensure that the wording of an APM permits a
determination of when the annual performance goal or APM has been accomplished.
In addition, the ORD guidance states that while the requirement for clearly articulated
performance information applies to both internal and external annual performance goals/APMs,
it is particularly important that externally-reported annual performance goals/APMs
communicate several key pieces of information: (1) the importance of ORD actions to various
stakeholders; (2) the impetus for ORD actions; and (3) how the research outputs enable clients to
achieve outcomes.
Recommendations
We continue to recognize, as we had in previous OIG reports, that it may be difficult to measure
the annual accomplishments of scientific research, since research typically takes several years to
complete. However, since performance information that EPA reports is intended to assist
Agency managers in effectively managing their programs, it is imperative that this information be
reported accurately. Externally reported annual performance goals and APMs communicate the
importance and relevance of research to Congress and other external stakeholders. In addition,
an effective accountability process not only provides feedback on the success of specific
programs, but also introduces a higher level of integrity into planning and budgeting by holding
managers responsible for performance. Further, because annual performance goals and APMs
are used as tools for planning and measurement of progress toward achieving Agency goals,
these goals and measures should be written to clearly show the linkage to the Agency objective
and/or goal.
We recommend that the Assistant Administrator, Office of Research and Development:
1.	Direct staff to consistently follow current guidance when developing annual performance
goals and APMs related to children's health to ensure the wording clearly identifies what
constitutes completion.
2.	Put a system in place to verify the completion of APMs related to children's health before
reporting in the annual report.
6

-------
Agency Response and OIG Evaluation
ORD concurred with both our recommendations and our positions on Performance Measures 2
and 3, but did not concur with the finding related to Performance Measure 1. According to ORD,
the evidence citing specific site visits, annual grant reports, and peer reviewed publications
collectively constituted completion of this APM. A list documenting these publications and
activities was provided. We reviewed pertinent grant files and spoke with ORD and grantee
officials concerning the completion of this APM. The grantee officials advised us that the work
was not completed in FY 2000. Specifically, a grantee official said that only the data collection
was completed by 2000. Consequently, additional funding was requested after 2000 to interpret
and analyze the data. Additionally, another grant's 2000 Annual Performance Report stated that,
". . . the round of sampling should be completed by September 2001."
We acknowledge in our report that, over the life of a grant, a series of products, including papers
and journal articles, can be published prior to the completion of the grant. The Performance
Measure 1 read: "Assess pesticide exposure to children in Washington, Minnesota, and Arizona."
Due to the wording of the APM, the issuance of such documents or the evidence of site visits
would not constitute the completion of the assessment. However, if the intent for completion
was the evidence of a site visit, or publication of a paper or a journal article, the performance
measure should have been written to indicate such. The wording of the Performance Measure 1
did not clearly indicate what constituted completion.
ORD convened a workgroup to recommend improvements for enhancing the quality of
performance information in the multi-year plan. The workgroup reported that APMs should be
written in complete sentences and every effort should be made to ensure that the wording of a
goal or measure permits a determination of when the annual performance goal/APM has been
accomplished. Additionally, ORD's multi-year guidance dated October 2002 clarifies that a
reader should be able to determine what constitutes successful completion of the goal or measure
without further explanation. It further adds that every effort should be made to ensure the
wording of the goal or measure permits determination of when the annual performance
goal/APM has been accomplished.
ORD expressed concern regarding the clarity of the purpose sentence. Consequently, we
modified the purpose sentence to more clearly articulate the focus of the review. ORD agreed
with the report's emphasis on the importance of performance indicators in effective management
and accountability, and commended the OIG staff for emphasizing this point. ORD concurred
with the report's recommendations, for which they have already implemented and completed
corrective actions. In 2002, ORD completed a draft multi-year plan for Human Health Research
that established long-term goals, annual performance goals, and APMs for the Children's Health
Research program.
The text of ORD's memorandum responding to our report is in Appendix A; the three
attachments to that memorandum are available upon request.
7

-------
Appendix A
Auditee's Response
i Q \
13S, ?	UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
^	WASHINGTON, D.C. 20460
PRO"**"
OFFICE OF
RESEARCH AND DEVELOPMENT
August 26, 2003
MEMORANDUM
SUBJECT:	Response to the Office of Inspector General (OIG)
Draft Memorandum Report "Review of Selected Annual Performance
Measures Reported for Goal 8: Sound Science, Objective 2" (No. 2002-
000650)
FROM:	Paul Gilman /s/ Paul Gilman
Assistant Administrator (8101R)
TO:	Jeffrey K. Harris
Director for Program Evaluation,
Cross-Media Issues
Office of Inspector General (2460T)
I cannot concur with the Findings section or with its Results in Brief section of this draft
report because of problems with factual accuracy. These problems persist, despite clear and
significant evidence already provided by ORD which demonstrates that information and
conclusions presented in the draft report should be revised before the report is issued in final
form.
In the Findings section, the report asserts that Annual Performance Goal 60, Performance
Measure 1 was not completed during FY 2000, despite evidence already provided by ORD citing
specific site visits, annual grant reports, and peer reviewed publications that collectively
constitute completion of the Annual Performance Measure. A list of this evidence is provided in
Attachment 3. Although the draft report acknowledges these publications, it concludes that
"issuance of such documents does not indicate that the project has been completed. For example,
a grantee may publish papers throughout the entire project period related to different stages of
8

-------
their analysis." This statement is factually incorrect. It indicates that the draft OIG report
confuses completion of the publication of the scientific knowledge with completion of the period
of performance of the assistance agreement. It is the former event which ensures achievement of
the performance measure, not the latter event. Correcting this factual inaccuracy will alter the
conclusion now presented in both the Findings section and the Results in Brief section. The
revised conclusion should be that "EPA's reports did not communicate accurately about one of
three performance measures that contribute to accomplishing Annual Performance Goal 60 in
fiscal year 2000."
We are also concerned that there appears to be a disconnect between the purpose and the
findings of the report. The purpose of the report states that the OIG, "considered the question:
To what extent do EPA's Congressional GPRA measures of children's health research progress
accurately represent the Agency's efforts?" The first sentence of the purpose statement makes
clear that OIG was only reviewing a subset of measures related to children's health research. In
addition, the findings of the report make clear that OIG only examined the reporting on this
subset of measures. I suggest that the purpose statement be modified, as follows, to more clearly
represent the focus of the report: "We considered the question: To what extent does EPA
accurately communicate the status of its Congressional GPRA measures of children's health
research?"
However, the draft report underlines the importance of performance indicators for
effective management and accountability. I agree with this point and I commend your staff for
emphasizing it. In addition, I concur with both of the report's recommendations, for which ORD
has already implemented and completed corrective actions (Attachment 2). The draft report
recognizes that ORD issued guidance in October 2002, that helps members of ORD, and its
cross-agency Multi-Year Planning Teams, improve communication about, and enhance the
quality of, its performance indicators. This guidance is directed to all of our Multi-Year
Research Plans and thus is even broader in its extent than the OIG recommendations included in
this draft report.
Attachment 1 is our consolidated, page-specific comments which we offer for
incorporation in the final report. As noted, we have also included our corrective action plan to
the draft's recommendations. Since ORD has already completed the required actions, we believe
that the final report should state this and be closed out upon issuance.
Thank you for the opportunity to review and respond to this draft report. Should your
staff have questions or require further information, please have them contact Arnold Bloom at
(202) 564-6687.
Attachments (3)
9

-------
Appendix B
Distribution
Assistant Administrator, Office of Research and Development (8101R)
Comptroller (2731 A)
Agency Followup Official (271 OA)
Agency Followup Coordinator (2724A)
Audit Followup Coordinator, Office of Research and Development (8102)
Associate Administrator for Congressional and Intergovernmental Relations (1301 A)
Associate Administrator for Communications, Education, and Media Relations (1101 A)
Inspector General (2410)
10

-------