Ss> i C \OIG % ~ % PR0^S° OFFICE OF INSPECTOR GENERAL Catalyst for Improving the Environment Memorandum Report Selected Children's Health Annual Performance Measures for Goal 8, Sound Science Report No. 2003-M-000017 September 26, 2003 ------- Report Contributors: Jerri Dorsey Laurie Adams Abbreviations APM Annual Performance Measure EPA Environmental Protection Agency FY Fiscal Year GPRA Government Performance and Results Act OIG Office of Inspector General ORD Office of Research and Development ------- 2 < 5o o v ^ PBO^0 ro LU a & UNITED STATES ENVIRONMENTAL PROTECTION AGENCY WASHINGTON, D.C. 20460 OFFICE OF INSPECTOR GENERAL September 26, 2003 MEMORANDUM SUBJECT: Selected Children's Health Annual Performance Measures for Goal 8, Sound Science Report Number 2003-M-000017 FROM: TO: Jeffrey K. Harris /s/ Director for Program Evaluation, Cross-Media Issues Paul Gilman Assistant Administrator, Office of Research and Development This is our final memorandum report on the subject review conducted by the Office of Inspector General (OIG) of the U.S. Environmental Protection Agency (EPA). This memorandum report contains findings that describe the problems that the OIG has identified and corrective actions that the OIG recommends. This memorandum report represents the opinion of the OIG and the findings contained in this report do not necessarily represents the final EPA position. Final determinations on matters in this report will be made by EPA managers in accordance with established resolution procedures. Action Required EPA's Office of Research and Development (ORD) provided a corrective action plan for agreed upon actions, including milestone dates. All corrective actions have been completed. Therefore, we are closing the memorandum report on issuance. We have no objections to the further release of this report to the public. For your convenience, this report will be available at http://www.epa.gov/oig/ If you or your staff have any question, please contact me at (202) 566-0831 or Jerri Dorsey at (919) 541-3601. 1 ------- Purpose We have reviewed a subset of EPA's Government Performance and Results Act (GPRA) measures related to children's environmental health research. We considered the question: To what extent does EPA accurately communicate the status of its Congressional GPRA measures of children's health research? This was the first of a series of reviews we have planned on environmental issues related to children's health. Results in Brief The status of the annual performance measures developed by ORD for Fiscal Year (FY) 2000 Annual Performance Goal 60, "develop risk assessment guidance and regional assessments concerning risks to children exposed to environmental contaminants," was not reported accurately. This generally occurred because personnel had varying interpretations of what constituted completion of measures. Background At the time of our review, EPA had 10 long-term strategic goals1 that identified environmental results the Agency is working to achieve. Under Goal 8, Sound Science, the Agency is charged to develop and apply the best available science for addressing current and future environmental hazards, as well as new approaches toward improving environmental problems. Under the Agency's current goal structure, there are several objectives under Goal 8. One objective is to improve the scientific basis for managing environmental exposures and risks, and developing new models and methodologies to integrate exposure and effects findings. Each year, as required by GPRA, the Agency prepares an annual plan that translates long-term goals into specific actions to be taken and resources to be used during the fiscal year. These annual plans identify what they intend to accomplish, measure how well they are doing, make appropriate decisions based on information they have gathered, and communicate information about their performance to Congress and to the public. At the end of a fiscal year, the Agency develops an Annual Report that describes the year's achievements and progress in protecting human health. Agency performance information is reported, per GPRA, in the form of annual performance goals and associated annual performance measures (APMs). For a fiscal year, annual performance goals represent the target level of performance expected. The Agency develops many annual performance goals and related measures for a given year. However only goals and measures that are identified as Congressional2 are reported in the annual report. 1 EPA's goal structure is currently being revised. The draft strategic architecture establishes five Agency goals. Under the new goal structure, Research is not a goal but rather an objective under each of EPA's five goals. 2 Beginning with FY 2004, annual performance goals that address the most significant planned accomplishments are called "External." External are of the level of importance corresponding to Congressional in previous years. 2 ------- In 1998, ORD initiated the Children's Health program to support research on environmental risks to children. The primary objective of the program is to conduct the research and provide the methods to reduce uncertainties in EPA risk assessments for children, leading to effective measures for risk reduction. Scope and Methodology ORD published a Strategy for Research on Environmental Risks to Children in 2000, which established directions for research in children's risk and documented the rationale for the chosen program direction. The Children's Health Research Program has been arrayed and tracked under a series of objectives, subobjectives, and annual milestones for reporting under GPRA. Congressional annual performance goals and APMs are expected to address the most significant planned accomplishments, focusing on results. Congressional (externally reported) annual performance goals and APMs are used in the Agency's Annual Performance Plan to support the Agency's budget request in the Congressional Justification. Therefore, our review focused on the performance goal and related measures that were identified by the ORD as Congressional. Our review specifically focused on the children's health Congressional measures3 reported under Goal 8, Objective 2 (Table 1). Table 1: Focus of Review Goal 8: Sound Science, Improved Understanding of Environmental Risk, and Greater Innovation to Address Environmental Problems Objective 2: Improve the scientific basis to identify, characterize, assess, and manage environmental exposures that pose the greatest health risks to the American public by developing models and methodologies to integrate information about exposures and effects from multiple pathways.4 Annual Performance Goal 60: Develop risk assessment guidance and regional assessments concerning risks to children exposed to environmental contaminants Annual Performance Measures: 1. Assess pesticide exposure to children in Washington, Minnesota, and Arizona 2. Report on the use of mechanistic data in developmental toxicity risk 3. Develop exposure factors handbook for children To assess the accuracy and presentation of EPA's GPRA measures related to children's health, we interviewed key officials within ORD, the Office of the Chief Financial Officer, and related grantees; and reviewed external publications as well as internal planning documents and grants. We performed our review in accordance with Government Auditing Standards, issued by the Comptroller General of the United States. We performed our field work from November 18, 2002, through March 28, 2003. 3 In FY 2000, ORD had 13 annual APMs, three of which were identified as Congressional. For FY 2001 and 2002, there were 13 and 10 APMs, respectively, none of which were identified as Congressional. 4 As reported in the Agency's FY 2000 Annual Report. 3 ------- Findings EPA's Reports Were Inaccurate for Two of the Three Measures Regarding Annual Performance Goal 60 Performance Measure 2, "Report on the use of mechanistic data in developmental toxicity risk," ORD indicated this measure was met by publication in 2000 of the book, Scientific Frontiers in Developmental Toxicology and Risk Assessment. Based on our review, we concluded that this measure was reported accurately. However, we determined that information ORD reported for Performance Measures 1 and 3 was not sufficiently accurate. Details follow. Annual Performance Goal 60 Performance Measure 1: Assess pesticide exposure to children in Washington, Minnesota, and Arizona EPA reported that "assessments of pesticide exposures to children in Washington, Minnesota, and Arizona, were published in FY 2000." However, we found the reporting on this APM during FY 2000s to be inaccurate. The Agency awarded three "Science To Achieve Results" grants to accomplish this measure. Each was awarded in September 1996 and focused on children's exposures in urban and suburban settings. EPA reported that these assessments were completed in FY 2000. However, we found that the project period for each grant was extended beyond FY 2000, and supplemental funding was provided for each grant after FY 2000. As illustrated in Table 2, one final report was issued in FY 2002 and the other two were issued in FY 2003. Table 2: Status of "Science To Achieve Results" Grants Grant Number Recipient Original Project Completion Date Amended Project Completion Date Final Report Date R825169-01-0 University of Arizona 9/30/1999 9/30/2001 9/04/2002 R825283-01-0 University of Minnesota 9/30/1999 9/30/2002 5/20/2003 R825171-01-0 University of Washington 9/30/1999 9/30/2002 7/01/2003 In reviewing grants we identified several journal articles and publications related to these research projects; however according to the grantees and the annual progress reports for 2000, the analysis related to these grants was not completed by the end of FY 2000. For example, one grantee progress report, covering the period of January 2000 - December 2000, indicated that more detailed analysis for all phases of the study were remaining. We acknowledge that, over the life of a grant, a series of products, including papers and 5 During FY 2001 and FY 2002, no annual performance goals and related APMs were reported in the annual report under Goal 8, Objective 2. 4 ------- journal articles, can be published prior to the completion of the grant. However issuance of such documents does not indicate that the project has been completed. For example, a grantee may publish papers throughout the entire project period related to different stages of its analysis. Annual Performance Goal 60 Performance Measure 3: Develop exposure factors handbook for children EPA reported within its FY 2000 Annual Report that, "The Exposure Factors Handbook was not completed due to the extension of the public comment period. The final handbook will be released in FY 2001." Subsequently, EPA reported within its 5-year performance data report (for FY 1999-2003) attached to the FY 2003 Congressional Justification that the handbook was released in FY 2001. We found these statements to be inaccurate. • The reporting of the completion of the handbook in FY 2001 was incorrect. The external review draft of the handbook was released on EPA's web site in FY 2000. The interim version of the handbook was placed on EPA's web site January 15, 2003. According to ORD officials, the Office of the Chief Financial Officer was not made aware of the delayed issuance of the handbook, which resulted in the inaccurate information being included in the Five Year Performance review attached to the FY 2003 Congressional Justification. According to an ORD official, the release of the final handbook is projected for 2004. We also found the explanation provided for the delay to be inaccurate. The annual report stated the handbook was delayed "due to the extension of the public comment period." However, we found that a public comment period was not held. Rather, a peer review was conducted at EPA's request. Peer reviews, although an important component of the review process, are not synonymous with a public comment period. Generally, a public comment period is open to all issues whereas the peer review is limited to technical issues.6 ORD recognized that there was no formal public comment period; however, they provided the American Chemical Council the opportunity to submit unsolicited comments in addition to the official peer review that was conducted. We recognize that there were several reasons for the delay of the handbook, including a decision to coordinate with guidance being developed by the EPA Risk Assessment Forum. Nonetheless, the reporting of the completion date and related reasons for delay were reported inaccurately. 6 The goal of a peer review is to obtain an independent third party review of the product from experts who have not substantially contributed to its development. 5 ------- ORD Has Initiated Efforts to Improve Measures Since the development and initial reporting of Annual Performance Goal 60, ORD has examined its approach to annual performance goals and related APM development. ORD convened a workgroup to recommend specific improvements for enhancing the quality of performance information in multi-year plans. The workgroup provided clear definitions for appropriate annual performance goals and APMs, as well as development guidance. In addition, ORD issued guidance in October 2002 on multi-year planning. This guidance explains that annual performance goals and APMs should be quantifiable and measurable. The current guidance stresses that every effort should be made to ensure that the wording of an APM permits a determination of when the annual performance goal or APM has been accomplished. In addition, the ORD guidance states that while the requirement for clearly articulated performance information applies to both internal and external annual performance goals/APMs, it is particularly important that externally-reported annual performance goals/APMs communicate several key pieces of information: (1) the importance of ORD actions to various stakeholders; (2) the impetus for ORD actions; and (3) how the research outputs enable clients to achieve outcomes. Recommendations We continue to recognize, as we had in previous OIG reports, that it may be difficult to measure the annual accomplishments of scientific research, since research typically takes several years to complete. However, since performance information that EPA reports is intended to assist Agency managers in effectively managing their programs, it is imperative that this information be reported accurately. Externally reported annual performance goals and APMs communicate the importance and relevance of research to Congress and other external stakeholders. In addition, an effective accountability process not only provides feedback on the success of specific programs, but also introduces a higher level of integrity into planning and budgeting by holding managers responsible for performance. Further, because annual performance goals and APMs are used as tools for planning and measurement of progress toward achieving Agency goals, these goals and measures should be written to clearly show the linkage to the Agency objective and/or goal. We recommend that the Assistant Administrator, Office of Research and Development: 1. Direct staff to consistently follow current guidance when developing annual performance goals and APMs related to children's health to ensure the wording clearly identifies what constitutes completion. 2. Put a system in place to verify the completion of APMs related to children's health before reporting in the annual report. 6 ------- Agency Response and OIG Evaluation ORD concurred with both our recommendations and our positions on Performance Measures 2 and 3, but did not concur with the finding related to Performance Measure 1. According to ORD, the evidence citing specific site visits, annual grant reports, and peer reviewed publications collectively constituted completion of this APM. A list documenting these publications and activities was provided. We reviewed pertinent grant files and spoke with ORD and grantee officials concerning the completion of this APM. The grantee officials advised us that the work was not completed in FY 2000. Specifically, a grantee official said that only the data collection was completed by 2000. Consequently, additional funding was requested after 2000 to interpret and analyze the data. Additionally, another grant's 2000 Annual Performance Report stated that, ". . . the round of sampling should be completed by September 2001." We acknowledge in our report that, over the life of a grant, a series of products, including papers and journal articles, can be published prior to the completion of the grant. The Performance Measure 1 read: "Assess pesticide exposure to children in Washington, Minnesota, and Arizona." Due to the wording of the APM, the issuance of such documents or the evidence of site visits would not constitute the completion of the assessment. However, if the intent for completion was the evidence of a site visit, or publication of a paper or a journal article, the performance measure should have been written to indicate such. The wording of the Performance Measure 1 did not clearly indicate what constituted completion. ORD convened a workgroup to recommend improvements for enhancing the quality of performance information in the multi-year plan. The workgroup reported that APMs should be written in complete sentences and every effort should be made to ensure that the wording of a goal or measure permits a determination of when the annual performance goal/APM has been accomplished. Additionally, ORD's multi-year guidance dated October 2002 clarifies that a reader should be able to determine what constitutes successful completion of the goal or measure without further explanation. It further adds that every effort should be made to ensure the wording of the goal or measure permits determination of when the annual performance goal/APM has been accomplished. ORD expressed concern regarding the clarity of the purpose sentence. Consequently, we modified the purpose sentence to more clearly articulate the focus of the review. ORD agreed with the report's emphasis on the importance of performance indicators in effective management and accountability, and commended the OIG staff for emphasizing this point. ORD concurred with the report's recommendations, for which they have already implemented and completed corrective actions. In 2002, ORD completed a draft multi-year plan for Human Health Research that established long-term goals, annual performance goals, and APMs for the Children's Health Research program. The text of ORD's memorandum responding to our report is in Appendix A; the three attachments to that memorandum are available upon request. 7 ------- Appendix A Auditee's Response i Q \ 13S, ? UNITED STATES ENVIRONMENTAL PROTECTION AGENCY ^ WASHINGTON, D.C. 20460 PRO"**" OFFICE OF RESEARCH AND DEVELOPMENT August 26, 2003 MEMORANDUM SUBJECT: Response to the Office of Inspector General (OIG) Draft Memorandum Report "Review of Selected Annual Performance Measures Reported for Goal 8: Sound Science, Objective 2" (No. 2002- 000650) FROM: Paul Gilman /s/ Paul Gilman Assistant Administrator (8101R) TO: Jeffrey K. Harris Director for Program Evaluation, Cross-Media Issues Office of Inspector General (2460T) I cannot concur with the Findings section or with its Results in Brief section of this draft report because of problems with factual accuracy. These problems persist, despite clear and significant evidence already provided by ORD which demonstrates that information and conclusions presented in the draft report should be revised before the report is issued in final form. In the Findings section, the report asserts that Annual Performance Goal 60, Performance Measure 1 was not completed during FY 2000, despite evidence already provided by ORD citing specific site visits, annual grant reports, and peer reviewed publications that collectively constitute completion of the Annual Performance Measure. A list of this evidence is provided in Attachment 3. Although the draft report acknowledges these publications, it concludes that "issuance of such documents does not indicate that the project has been completed. For example, a grantee may publish papers throughout the entire project period related to different stages of 8 ------- their analysis." This statement is factually incorrect. It indicates that the draft OIG report confuses completion of the publication of the scientific knowledge with completion of the period of performance of the assistance agreement. It is the former event which ensures achievement of the performance measure, not the latter event. Correcting this factual inaccuracy will alter the conclusion now presented in both the Findings section and the Results in Brief section. The revised conclusion should be that "EPA's reports did not communicate accurately about one of three performance measures that contribute to accomplishing Annual Performance Goal 60 in fiscal year 2000." We are also concerned that there appears to be a disconnect between the purpose and the findings of the report. The purpose of the report states that the OIG, "considered the question: To what extent do EPA's Congressional GPRA measures of children's health research progress accurately represent the Agency's efforts?" The first sentence of the purpose statement makes clear that OIG was only reviewing a subset of measures related to children's health research. In addition, the findings of the report make clear that OIG only examined the reporting on this subset of measures. I suggest that the purpose statement be modified, as follows, to more clearly represent the focus of the report: "We considered the question: To what extent does EPA accurately communicate the status of its Congressional GPRA measures of children's health research?" However, the draft report underlines the importance of performance indicators for effective management and accountability. I agree with this point and I commend your staff for emphasizing it. In addition, I concur with both of the report's recommendations, for which ORD has already implemented and completed corrective actions (Attachment 2). The draft report recognizes that ORD issued guidance in October 2002, that helps members of ORD, and its cross-agency Multi-Year Planning Teams, improve communication about, and enhance the quality of, its performance indicators. This guidance is directed to all of our Multi-Year Research Plans and thus is even broader in its extent than the OIG recommendations included in this draft report. Attachment 1 is our consolidated, page-specific comments which we offer for incorporation in the final report. As noted, we have also included our corrective action plan to the draft's recommendations. Since ORD has already completed the required actions, we believe that the final report should state this and be closed out upon issuance. Thank you for the opportunity to review and respond to this draft report. Should your staff have questions or require further information, please have them contact Arnold Bloom at (202) 564-6687. Attachments (3) 9 ------- Appendix B Distribution Assistant Administrator, Office of Research and Development (8101R) Comptroller (2731 A) Agency Followup Official (271 OA) Agency Followup Coordinator (2724A) Audit Followup Coordinator, Office of Research and Development (8102) Associate Administrator for Congressional and Intergovernmental Relations (1301 A) Associate Administrator for Communications, Education, and Media Relations (1101 A) Inspector General (2410) 10 ------- |