j A * 1® U.S. ENVIRONMENTAL PROTECTION AGENCY OFFICE OF INSPECTOR GENERAL Improving EPA research programs EPA Office of Research and Development Needs to Address Barrier to Gathering External Customer Feedback Report No. 19-P-0277 August 19, 2019 ------- Report Contributors: Patrick Gilbride Erin Barnes-Weaver Jenny Drzewiecki Allison Krenzien Fred Light Abbreviations EPA U.S. Environmental Protection Agency FY Fiscal Year OIG Office of Inspector General OMB Office of Management and Budget ORD Office of Research and Development SM Strategic Measure Cover Photo: Graphic depicting types of customer feedback. (EPA OIG image) Are you aware of fraud, waste or abuse in an EPA program? EPA Inspector General Hotline 1200 Pennsylvania Avenue, NW (2431T) Washington, DC 20460 (888) 546-8740 (202) 566-2599 (fax) OIG Hotline@epa.gov Learn more about our OIG Hotline. EPA Office of Inspector General 1200 Pennsylvania Avenue, NW (2410T) Washington, DC 20460 (202) 566-2391 www.epa.gov/oiq Subscribe to our Email Updates Follow us on Twitter @EPAoig Send us your Project Suggestions ------- .tfEDs 19-P-0277 * — U.S. Environmental Protection Agency August 19,2019 • u.o. ciiviiuMMiciiidi nuietu % Office of Inspector General At a Glance Why We Did This Project We conducted this audit to determine whether (1) the data collected and used to assess the U.S. Environmental Protection Agency (EPA) Office of Research and Development's (ORD's) Strategic Measure (SM) 21 — whether research products are meeting customer needs— provide a reasonable and accurate measure of customer satisfaction, and (2) the ORD faces any barriers when collecting data to support this measure. The ORD was assigned objective 3.3 (prioritize robust science) in the EPA Fiscal Years (FYs) 2018-2022 Strategic Plan, and SM-21 was the strategic measure for that objective. Per the ORD, a product is a deliverable that results from a specific research project or task and is meant for use by an ORD customer. The ORD developed a data collection instrument for FY 2018 baseline data. This report addresses the following: • Improving EPA research programs. Address inquiries to our public affairs office at (202) 566-2391 or OIG WEBCOMMENTS@epa.gov. List of OIG reports. EPA Office of Research and Development Needs to Address Barrier to Gathering External Customer Feedback What We Found The ORD is in the process of gathering internal customer satisfaction data for FY 2019. However, without sufficient data beyond the FY 2018 baseline, we cannot completely answer our first objective on whether the data collected and used to assess the ORD's SM-21 provides a reasonable and accurate measure of customer satisfaction. Surveying non-federal external customers will widen the ORD's reporting under the strategic measure on whether research products meet customer needs. With regard to our second objective, the ORD faces a key barrier of being significantly limited in surveying non-federal external customers when collecting data to support its strategic measure. Under the Paperwork Reduction Act, the ORD cannot survey more than nine non-federal external customers without approval from the Office of Management and Budget (OMB). The ORD is aware of this barrier and noted that, because of it, current SM-21 customer input is not representative of the ORD's full user base. To address this barrier, the ORD is in the process of developing, for the OMB's approval, an information collection request to obtain non-federal external customer feedback. The information collection request is in draft and the ORD has not yet submitted it to the OMB for review. According to the ORD, the process to obtain approval from the OMB for the information collection request is expected to take as many as 18 months. Completing actions to address this barrier to gathering non-federal external input will widen the customer base the ORD could survey to determine whether its research products meet customer needs. Recommendation and Planned Agency Corrective Actions We recommend that the Assistant Administrator for Research and Development submit to the OMB for review an information collection request to be allowed to survey more than nine non-federal external customers regarding research products meeting customer needs. Should the OMB not grant approval of the information collection request, we recommend that the ORD develop alternatives to assess non-federal external customer satisfaction to more fully inform reporting under the strategic measure. The ORD concurred with our recommendation and plans to complete corrective actions by July 30, 2020 (July 30, 2021, in the event the OMB rejects the information collection request). Our recommendation is resolved with corrective actions pending. ------- UNITED STATES ENVIRONMENTAL PROTECTION AGENCY WASHINGTON, D.C. 20460 OFFICE OF INSPECTOR GENERAL August 19, 2019 MEMORANDUM SUBJECT: EPA Office of Research and Development Needs to Address Barrier to Gathering External Customer Feedback Report No. 19-P-0277 FROM: Charles J. Sheehan, Deputy Inspector General TO: Jennifer Orme-Zavaleta, Principal Deputy Assistant Administrator for Science Office of Research and Development This is our report on the subject audit conducted by the Office of Inspector General (OIG) of the U.S. Environmental Protection Agency (EPA). The project number for this assignment was OA&E-FY19-0084. This report contains findings that describe the problems the OIG has identified and corrective actions the OIG recommends. This report represents the opinion of the OIG and does not necessarily represent the final EPA position. Final determinations on matters in this report will be made by EPA managers in accordance with established resolution procedures. In accordance with EPA Manual 2750, your office provided acceptable corrective actions and a milestone date in response to our recommendation. Our recommendation is resolved, and no final response to this report is required. However, if you submit a response, it will be posted on the OIG's website, along with our memorandum commenting on your response. Your response should be provided as an Adobe PDF file that complies with the accessibility requirements of Section 508 of the Rehabilitation Act of 1973, as amended. The final response should not contain data that you do not want to be released to the public; if your response contains such data, you should identify the data for redaction or removal along with corresponding justification. We will post this report to our website at www.epa.gov/oig. ------- EPA Office of Research and Development Needs to Address Barrier to Gathering External Customer Feedback 19-P-0277 Table of C Purpose 1 Background 1 Scope and Methodology 2 Results 2 Recommendation 3 Agency Response and OIG Evaluation 3 Status of Recommendations and Potential Monetary Benefits 4 Appendices A Agency Comments on Draft Report 5 B Distribution 7 ------- Purpose We conducted this audit to determine whether (1) the data collected and used to assess the U.S. Environmental Protection Agency (EPA) Office of Research and Development's (ORD's) Strategic Measure (SM) 21—whether research products are meeting customer needs—provide a reasonable and accurate measure of customer satisfaction, and (2) the ORD faces any barriers when collecting data to support this measure. Background The ORD was assigned the responsibility for the EPA Fiscal Years (FYs) 2018- 2022 Strategic Plan objective 3.3 (prioritize robust science), and SM-21 was listed as the measure for that strategic objective. The ORD formed a Strategic Measure Workgroup to develop approaches to assess the ORD's progress toward its goal of "[increasing the percentage of ORD research products that meet customer needs." The ORD defined "product" as a deliverable that results from a specific research project or task and is meant for use by an ORD customer. This can include one of many scientific deliverables such as journal articles, agency reports, software tools and databases. The ORD defines "meeting customer needs" through three criteria: quality, usability and timeliness, as shown in Table 1. Table 1: ORD criteria for meeting customer needs Criteria Description Quality The extent to which a research product meets scientific and technical standards, is rigorous and transparent, and advances the state of knowledge for a particular topic. Usability The degree to which a research product is suited for and easily applied to actions within EPA's mission. Timeliness Whether a product is delivered within a timeframe appropriate for customer use. Source: ORD's Strategic Measure: Developing and piloting ORD's Strategic Measure on the number of research products that meet customer needs (May 4, 2018). According to the ORD, if a given research product is high quality, highly usable and delivered in a timely fashion, that product is considered to meet customer needs. The ORD developed and pilot-tested a survey/data collection instrument and used it to gather FY 2018 baseline data on research products randomly selected from the ORD's Research Management System.1 The sample selection of research products to evaluate under SM-21 excluded all products designed exclusively for 1 The ORD's Research Management System houses the ORD's research portfolio information. It provides a comprehensive view of the ORD's ongoing and planned research, including research programs, projects and products. 19-P-0277 1 ------- non-federal partners (such as state governments or local utilities) and, instead, focused on internal EPA customers because the ORD was able to survey internal customers. The ORD cannot survey more than nine non-federal external customers absent approval from the Office of Management and Budget (OMB). Based on the FY 2018 data collection, the ORD's targeted customer satisfaction is 77 percent in FY 2019 and 80 percent in FY 2020. Scope and Methodology We conducted this performance audit from February 27, 2019, to June 18, 2019, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based upon our audit objectives. For our first objective, the ORD has collected 1 year of baseline data and is in the process of collecting additional data. Data collected includes responses from internal EPA customers only. As such, information including data from non-federal external customers and multiple years for comparison was not yet available to fully answer our first objective. We reviewed the ORD's written materials on SM-21, including the Strategic Measure Workgroup Report (May 2018), FY 2018 baseline data, the data collection methodology and instrument, and guidance on identifying survey recipients. We also interviewed the SM-21 workgroup lead and other staff and managers responsible for SM-21 in the ORD's Office of Program Accountability and Resource Management. Results The ORD is in the process of gathering internal customer satisfaction data for FY 2019. However, without sufficient data beyond the FY 2018 baseline, we cannot completely answer our first objective on whether the data collected and used to assess the ORD's SM-21 provides a reasonable and accurate measure of customer satisfaction. With regard to our second objective, the ORD faces a key barrier of being significantly limited in surveying non-federal customers when collecting data to support its strategic measure. Under the Paperwork Reduction Act, the ORD cannot survey more than nine non-federal external customers without approval from the OMB. The ORD is aware of this barrier and noted that, because of it, current SM-21 customer input is not representative of the ORD's full user base. To address this barrier, the ORD is in the process of developing, for the OMB's approval, an information collection request to obtain non-federal external customer feedback to inform the ORD's FY 2020 reporting under SM-21. The information collection request is in draft and the ORD has not yet submitted it to the OMB for review. According to the ORD, the process to obtain approval from 19-P-0277 2 ------- the OMB for the information collection request is expected to take as many as 18 months. Completing actions to address this barrier to gathering non-federal external input will widen the customer-base the ORD could survey to determine whether its research products meet customer needs. Recommendation We recommend that the Assistant Administrator for Research and Development: 1. Submit to the Office of Management and Budget for review an information collection request to be allowed to survey more than nine non-federal external customers regarding research products meeting customer needs. Should the Office of Management and Budget not grant approval of the information collection request, develop alternatives to assess non-federal external customer satisfaction to more fully inform reporting under the strategic measure. Agency Response and OIG Evaluation The ORD concurred with our recommendation and provided a corrective action date of July 30, 2020. In the event the OMB rejects the information collection request, the ORD will work to identify alternative data collection methods to capture non-federal customer satisfaction and will do so by July 30, 2021. Recommendation 1 is resolved with corrective actions pending. Appendix A provides the ORD's full response to the draft report. 19-P-0277 3 ------- Status of Recommendations and Potential Monetary Benefits RECOMMENDATIONS Potential Planned Monetary Rec. Page Completion Benefits No. No. Subject Status1 Action Official Date (In $000s) 1 3 Submit to the Office of Management and Budget for review an R Assistant Administrator for 7/30/20 information collection request to be allowed to survey more than Research and nine non-federal external customers regarding research products Development meeting customer needs. Should the Office of Management and Budget not grant approval of the information collection request, develop alternatives to assess non-federal external customer satisfaction to more fully inform reporting under the strategic measure. 1 C = Corrective action completed. R = Recommendation resolved with corrective action pending. U = Recommendation unresolved with resolution efforts in progress. 19-P-0277 4 ------- Appendix A Agency Comments on Draft Report UNITED STATES ENVIRONMENTAL PROTECTION AGENCY WASHINGTON. D C. 20460 July 22, 2019 OFFICE Of RESEARCH AND DEVELOPMENT MEMORANDUM SUBJECT: Response to Office of Inspector General (OIG) Draft Report No. OA&E-FY 19-0084 ~EPA Office of Research and Development Needs to Address Barrier on Gathering External Customer Feedback" dated June 18, 2019 The EPA s Olfice ot Research and Development (ORD) welcomes the opportunity to review and comment on the OIG s draft report titled "EPA Office of Research and Developments Needs to Address Barrier on Gathering External Customer Feedback" (Project No. OA&E-F Y19-0084) (Draft Report). ORD believes OIG staff have been diligent in researching the background of ORD's Strategic \leasure-21. as well as understanding ORD's methodology to gather customer feedback to measure the number of ORD research products that meet customer needs through three criteria: quality, usability and timeliness. OlG's observations are largely accurate and balanced. Accordingly, we do not take issue with the overall content of the Draft Report and do not have any additional comments. Recommendation 1: "Submit to the Office of Management and Budget for review an information collection request to be allowed to survey non-federal customers regarding research products meeting customer needs. Should the Office of Management and Budget not grant approval of the information collection request, develop alternatives to assess non-federal customer satisfaction to more fully inform reporting under the strategic measure." Response 1: ORD concurs with this recommendation and proposes the following corrective action and completion date. Corrective Action I: ORD will complete the information collection request (ICR) to survey external stakeholders of ORD's scientific research products and submit it to the Office of Management and Budget (OMB) for approval. In the event that OMB rejects the ICR, ORD will work to identify alternative data collection methods that will capture non-federal customer satisfaction for Fiscal Year 2021 data collection. Planned Completion Date: July 30, 2020 (July 30, 2021 in Ihe event that OMB rejects the ICR) FROM: Jennifer Orme-Zavaleta, Ph.D O* Principal Deputy Assistant AdministratOcfor Science Office of Research and DevclopmW J ±JeX> TO: Kevin Christensen Assistant Inspector General Office of Inspector General 19-P-0277 5 ------- If you have any questions regarding this response, please contact Deborah Ileckman. Acting Director, Office ot Program Accountability and Resource Management, at heckman.deborah a epa^ov. Attachment cc: Christopher S. Robbins Deborah Heckman Sean Paul Kelly van Bronkhorst Heather Cursio Maureen Mingciey Patrick Gilbride, OIG Erin Barnes-Weaver. OIG 19-P-0277 6 ------- Appendix B Distribution The Administrator Deputy Administrator Chief of Staff Deputy Chief of Staff Assistant Administrator for Research and Development and EPA Science Advisor Agency Follow-Up Official (the CFO) Agency Follow-Up Coordinator General Counsel Associate Administrator for Congressional and Intergovernmental Relations Associate Administrator for Public Affairs Director, Office of Continuous Improvement, Office of the Administrator Principal Deputy Assistant Administrator for Science, Office of Research and Development Associate Director for Science, Office of Research and Development Deputy Assistant Administrator for Research and Development, Office of Research and Development Deputy Assistant Administrator for Management, Office of Research and Development Audit Follow-Up Coordinator, Office of the Administrator Audit Follow-Up Coordinator, Office of Research and Development 19-P-0277 7 ------- |