j A * 1® U.S. ENVIRONMENTAL PROTECTION AGENCY OFFICE OF INSPECTOR GENERAL Improving EPA research programs Regional Research Programs Address Agency Needs but Could Benefit from Enhanced Project Tracking Report No. 19-P-0123 April 18, 2019 ------- Report Contributors: Patrick Gilbride Erin Barnes-Weaver Fred Light Allison Krenzien Raul Adrian Abbreviations EPA U.S. Environmental Protection Agency OIG Office of Inspector General ORD Office of Research and Development R2P2 Regional Research Partnership Program RARE Regional Applied Research Effort RESES Regional Sustainability and Environmental Sciences ROCS-Net Regional-ORD Community of Science Networking RSL Regional Science Liaison RSP Regional Science Program SHC Sustainable and Healthy Communities Cover Photo: Village Green station at Jane Addams Elementary School in Chicago, Illinois, which is part of Region 5's RARE Project 1685. (EPA photo) Are you aware of fraud, waste or abuse in an EPA program? EPA Inspector General Hotline 1200 Pennsylvania Avenue, NW (2431T) Washington, DC 20460 (888) 546-8740 (202) 566-2599 (fax) OIG Hotline@epa.gov Learn more about our OIG Hotline. EPA Office of Inspector General 1200 Pennsylvania Avenue, NW (2410T) Washington, DC 20460 (202) 566-2391 www.epa.gov/oiq Subscribe to our Email Updates Follow us on Twitter @EPAoig Send us your Project Suggestions ------- ^£DS7X • JL v I®/ U.S. Environmental Protection Agency Office of Inspector General At a Glance 19-P-0123 April 18, 2019 Why We Did This Project We conducted this audit to determine whether the Office of Research and Development's (ORD's) support of regional research programs helps accomplish the U.S. Environmental Protection Agency's (EPA's) mission. We also examined whether results of regional science initiatives impact the agency's decision- making. Our audit focused on two regional research programs: 1. Regional Applied Research Effort (RARE). 2. Regional Sustainability and Environmental Sciences (RESES). ORD has Regional Science Liaisons (RSLs) within each region who participate on Regional Science Councils that help identify RARE and RESES projects. In 2015, ORD's Office of Science Policy developed the Regional Science Program (RSP) Tracker database to monitor RARE projects. This report addresses the following: • Improving EPA research programs. Address inquiries to our public affairs office at (202) 566-2391 or OIG WEBCOMMENTS@epa.oiq. List of OIG reports. Regional Research Programs Address Agency Needs but Could Benefit from Enhanced Project Tracking What We Found For the eight RARE and two RESES projects we sampled, regional staff said that ORD's support helped accomplish the agency's mission by addressing the regions' high-priority, near-term research needs. Additionally, regional and ORD staff described positive impacts that these projects had on agency operations and/or decision benefits that their offices gained by participating in Updates to the RSP tracking system would improve how ORD documents and communicates the benefits of regional research programs. -making, as well as other regional research programs. While interviewees expressed overall satisfaction with regional research programs, we found that ORD could better document project progress, outputs and impacts. Specifically, the RSP Tracker neither included RESES projects, nor consistently included start and end dates, project outputs, or significant project events for all RARE projects. The incomplete data stemmed from turnover among RSL staff, as well as the 2015 adoption of the RSP Tracker, which featured additional data fields not present for older projects. According to ORD, in 2018, the office began steps to clarify RSL roles, including revisiting a 2015 memorandum of understanding between ORD and the regions, developing an RSL implementation plan, and updating RARE guidelines. ORD said that it is also updating the RSP Tracker to include RESES projects and RARE project dates. Recommendations and Planned Agency Corrective Actions We recommend that the Assistant Administrator for Research and Development complete data entry of all RESES projects into the RSP Tracker; verify and update information for RARE projects in the RSP Tracker; and update the RSP Tracker to improve project tracking by documenting timelines, significant outputs and how results were used in agency decision-making. We also recommend that ORD require RSLs to use the RSP Tracker, increase awareness of the RSP Tracker among regional staff and regularly hold events for RSLs to share best practices. ORD concurred with all recommendations. One recommendation is completed. All other recommendations are resolved with corrective actions pending. The agency plans to complete all corrective actions by October 1, 2020. Noteworthy Achievements Prior to our audit, ORD launched a program evaluation—which it plans to complete in fiscal year 2019—on RESES project accomplishments and impacts. We also noted several effective practices relating to Regional Science Councils. Lastly, in September 2018, RSLs met to discuss challenges, goals, best practices and successes in their regions. One product resulting from that meeting was the development of an implementation workgroup focused on using the identified lessons learned to improve internal processes. ------- UNITED STATES ENVIRONMENTAL PROTECTION AGENCY WASHINGTON, D.C. 20460 OFFICE OF INSPECTOR GENERAL April 18, 2019 MEMORANDUM SUBJECT: Regional Research Programs Address Agency Needs but Could Benefit from Enhanced Project Tracking Report No. 19-P-0123 FROM: Charles J. Sheehan, Acting Inspector General TO: Jennifer Orme-Zavaleta, Principal Deputy Assistant Administrator for Science Office of Research and Development This is our report on the subject audit conducted by the Office of Inspector General (OIG) of the U.S. Environmental Protection Agency (EPA). The project number for this assignment was OA&E-FY18-0247. This report contains findings that describe the problems the OIG has identified and corrective actions the OIG recommends. This report represents the opinion of the OIG and does not necessarily represent the final EPA position. Final determinations on matters in this report will be made by EPA managers in accordance with established resolution procedures. In accordance with EPA Manual 2750, your office provided acceptable corrective actions and milestone dates in response to the OIG recommendations. One recommendation is completed. All other recommendations are resolved, and no final response to this report is required. However, if you submit a response, it will be posted on the OIG's website, along with our memorandum commenting on your response. Your response should be provided as an Adobe PDF file that complies with the accessibility requirements of Section 508 of the Rehabilitation Act of 1973, as amended. The final response should not contain data that you do not want to be released to the public; if your response contains such data, you should identify the data for redaction or removal along with corresponding justification. We will post this report to our website at www.epa.gov/oig. ------- Regional Research Programs Address 19-P-0123 Agency Needs but Could Benefit from Enhanced Project Tracking Table of C Purpose 1 Background 1 Responsible Office 4 Noteworthy Achievements 5 Scope and Methodology 5 Results 6 Offices Described Benefits and Impacts from Participating in Regional Research Programs 7 ORD Needs to Better Document Project Progress and Impacts 9 Conclusion 12 Recommendations 12 Agency Response and OIG Evaluation 13 Status of Recommendations and Potential Monetary Benefits 14 Appendices A RARE and RESES Project Sample 15 B Agency Comments on Draft Report 16 C Distribution 20 ------- Purpose We conducted this audit to determine whether the Office of Research and Development's (ORD's) support of regional research programs helps accomplish the U.S. Environmental Protection Agency's (EPA's) mission. We also examined whether results of regional science initiatives impact the agency's decision- making. Background ORD's Regional Science Program Science at the EPA provides the foundation for credible decision-making to safeguard human health and the environment. ORD is the scientific research arm of the EPA, and its research underpins all science and technology used across the agency, including in the regions. As highlighted in a 2015 memorandum of understanding between ORD and the regions, ORD recognizes the importance of science-based decision-making in the regions and the value of the scientific support that it provides to the regions.1 ORD's Office of Science Policy manages the Regional Science Program (RSP), which addresses high-priority regional science needs through collaborative research activities and technical support. The RSP manages three regional research programs: 1. Regional Applied Research Effort (RARE). This program fosters collaboration between regions and ORD laboratories and centers, as well as builds a regional/ORD network for future scientific interaction. 2. Regional-ORD Community of Science Networking (ROCS-Net). This orientation and networking program provides opportunities for regional and state staff who have limited familiarity with ORD to visit an EPA research facility and discuss science priorities and collaborative research opportunities with ORD scientists. 3. Regional Research Partnership Program (R2P2). This program provides opportunities for regional technical staff to travel to an ORD laboratory, center or office to work directly with ORD scientists on projects that target specific regional priorities. A 2015 memorandum of understanding between ORD and the regions "signifies the shared commitment by ORD and the regions to a strong partnership that supports the Agency's mission through effective coordination, collaboration and communication." 1 EPA, Regional Science Program Memorandum of Understanding Between the Office of Research and Development and the Regions, December 23, 2015. 19-P-0123 1 ------- Within ORD, research is planned by national research programs, one of which is the Sustainable and Healthy Communities (SHC) research program. The SHC conducts research focused on remediating contaminated sites, sediments and ground water; managing sustainable materials; and revitalizing communities impacted by contamination or recovering from natural disasters or extreme weather events. The SHC administers the Regional Sustainability and Environmental Sciences (RESES) program, which provides resources for regions and ORD researchers to collaborate on user-engaged research that advances regional science needs to improve state and community environmental and public health outcomes. Our audit focused on RARE and RESES projects (Figure 1). Figure 1: Regional research programs Focus of OIG Audit i SHC Office of Science Policy Source: Office of Inspector General (OIG) analysis. Per the 2015 memorandum of understanding, RARE projects support near-term, highly targeted research to address priority regional science needs. Most projects are designed to be completed within 2 years. In addition to fostering collaboration between the regions and ORD, RARE projects provide opportunities for ORD scientists to apply their expertise to regional issues and explore new research challenges. The 2015 memorandum of understanding also notes that RESES projects emphasize combining practical field measures and measurements with effective, long-term community involvement. Although RESES projects are like RARE projects in that they address regional science priorities and foster interoffice collaboration, RESES projects must also align with the SHC mission and meet the criteria outlined in the annual project solicitation, such as emphasizing community involvement. Through RESES projects, ORD and the regions work together to assist states and communities in improving environmental and public health outcomes. Although overseen by different entities, solicitations for both RARE and RESES projects require that all proposals be aligned with the Fiscal Year 2018-2022 EPA Strategic Plan and the Administrator's priorities. Figure 2 summarizes information about RARE and RESES projects. 19-P-0123 2 ------- Figure 2: RARE and RESES project information RARE / Criteria: \ RESES Clearly V'" / identified \ Management & Oversight: Office of Science Policy Project Timeline: 1-2 years Tracking: Biannual reports through RSP Tracker Programmatic Emphasis: Targeted research to address high priority regional needs Desired Outcomes: Regional, state and/or local decisions based on sound science Project Selection: Regions with input from ORD Funding: $260,000/region/year Alignment: Strategic Plan and Administrator's priorities Solicitation: Candidate projects identified by Regional Science Liaisons Management & Oversight: SHC Project Timeline: 2-3 years Tracking: Quarterly reports Programmatic Emphasis: Set annually by SHC (includes stakeholder engagement, transferability and alignment with SHC) Desired Outcomes: Science to inform state and community decisions affecting human health and the environment Project Selection: SHC with input from EPA peer review panel Funding: Competitive process across all regions ($850,000 for fiscal year 2019) Source: OIG analysis with input from ORD. Program Selection, Funding and Tracking Within each region, ORD has Regional Science Liaisons (RSLs) who are full- time regional employees funded by the Office of Science Policy and who, among other duties, serve as the primary link between ORD's research programs and the regional offices. Per ORD, "By being in close communication with regional management and technical staff, the RSLs are uniquely situated to identify ORD research that can impact high-priority regional policy and regulatory actions with state-of-the-art science." RSLs often chair or participate on Regional Science Councils that include other regional staff and management. These councils help identify potential RARE and RESES projects,2 and each council has a slightly different methodology for this process. ORD's Office of Science Policy has 4.6 full-time equivalents in headquarters for the RSP. ORD allocates funding for RARE projects so that ORD and each of the EPA's 10 regional offices can pursue collaborative research efforts. Each region conducts its own solicitation process and, after ORD reviews the RARE proposals, selects projects that best address the region's highest priority needs. The Office of Science Policy oversees the funding process for chosen projects. All RARE projects should be designed to generate products in 1 or 2 years. Region 10's RSL presents at an RSL/ORD poster session. (EPA photo) All RESES funds are managed by ORD's SHC, which competitively selects proposals put forth by regions. The SHC has two rounds of review when selecting 2 Superfund and Technology Liaisons, who report to the Office of Science Policy, also help select RARE and RESES projects. Superfund and Technology Liaisons are the primary technical liaisons between ORD and the EPA's regional offices on issues related to hazardous wastes. These liaisons work to ensure that the regions have access to technical support that can help them make scientifically defensible decisions during site cleanups. 19-P-0123 3 ------- RESES projects: one by the SHC's Director and Deputy, and another by a panel with organizational representation across laboratories and regions. Proposals must be consistent with the direction of SHC programs and are ranked based on the six criteria that were identified in the solicitation. Multi-region proposals that involve two or more separate geographic locations are encouraged and may request funding up to $200,000 per project. Projects that involve one geographic location may request funding up to $150,000 per project. All RESES projects should be designed to generate products in 2 or 3 years. As of March 2019, ORD planned to allocate $2.6 million for the RARE program ($260,000 per region) and $850,000 for the RESES program in fiscal year 2019, contingent on ORD's actual budget allocation. Figure 3 shows RSP and RESES program funding over 5 fiscal years. Figure S 3,000,000 S 2,500,000 $i,«o,ooo S 500,000 $0 FY 2014 FY 2015 FY 2016 FY 20!? FY 2018 -#-«« — *• RESES —#- IBtWt ROCS-Net Source: OIG analysis of funding information provided by ORD. Note: The ROCS-Net program started in fiscal year 2017. In 2015, the Office of Science Policy developed the RSP Tracker database to monitor RARE projects. The RSP Tracker is a searchable internal EPA system that contains all RARE projects, including the project proposals, funding information, dates and final reports on project results. The SHC monitors RESES projects by requiring that the ORD project technical leads submit quarterly reports using a standard template, meeting with principal investigators and visiting project sites. Responsible Office ORD has primary responsibility for the issues discussed in this report. 3: RSP and RESES funding, fiscal years 2014-2018 19-P-0123 4 ------- Noteworthy Achievements Before we began our audit, the SHC launched a program evaluation of RESES project accomplishments and impacts on decision-making. The SHC plans to complete the evaluation—which comprises case studies with related interviews and a survey distributed to project technical leads—in fiscal year 2019. Also, we noted several effective practices relating to Regional Science Councils: • The Region 4 Regional Science Council's charter includes a concise, one-page table describing each regional research program, as well as ORD and regional processes and resources. • Region 5's Regional Science Council comprises mostly branch chiefs (or above) to help facilitate outreach within each division. • The Region 7 Regional Science Council was the first to include state and tribal partners. • Region 8's Regional Science Council has a robust intranet site and extensive resources, including a science needs form to facilitate connecting regional scientists to ORD staff doing similar work. Lastly, in September 2018, RSLs from across the agency met with the Office of Science Policy to discuss challenges, goals, best practices and successes in their regions. The meeting spurred the development of an implementation workgroup focusing on how to use the identified "lessons learned" to improve internal RSL processes. Scope and Methodology We conducted this audit from July 2018 to February 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform our work to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for the findings and conclusions in this report based on our audit objectives. We reviewed federal laws and agency guidance on regional research, as well as EPA and ORD materials, including strategic plans, websites, policies and procedures. To understand ORD's regional research program coordination and support, we interviewed ORD staff, as well as regional staff from Regions 4, 5, 7 and 8, including all RSLs and members of the Regional Science Councils from those regions. Of the 14 projects—12 RARE and two RESES—completed from fiscal 19-P-0123 5 ------- year 2014 through July 2018 in Regions 4, 5, 7 and 8, we judgmentally selected a sample of eight RARE and both RESES projects. For each of these 10 completed projects, we reviewed documentation available in the RSP Tracker or provided by ORD, and we interviewed project technical leads in ORD and the regional offices to discuss project selection and any impacts on agency decision-making. Appendix A lists the projects sampled. Results For the 10 projects we sampled, regional staff said that ORD's support of regional research programs helped accomplish the agency's mission by addressing the regions' high-priority, near-term research needs. Additionally, regional and ORD staff described impacts on agency operations and/or decision-making, as well as other benefits that their offices gained by participating in regional research programs. These results demonstrate that RARE and RESES projects are fulfilling the objectives outlined in the 2015 memorandum of understanding between ORD and the regions. While interviewees expressed overall satisfaction with ORD's regional research programs, we found that ORD could better document project progress, outputs and impacts on the agency's decision-making. Specifically, we found that the RSP Tracker did not include the RESES projects we sampled. In addition, the RSP Tracker did not include all start and end dates, consistent date formats, all outputs, or all significant project events for the eight RARE projects we sampled. These incomplete data stemmed from a combination of two factors: 1. RSL turnover. 2. The RSP Tracker was not developed until 2015 and introduced new data fields not present for older projects. In its response to the draft report, ORD indicated that, in 2018, it began several steps to clarify RSL roles, including revisiting its 2015 memorandum of understanding, developing an RSL implementation plan and updating RARE guidelines. According to ORD, it is also updating the RSP Tracker to include all dates for RARE projects funded in 2015 and beyond and to enter all RESES projects. These enhancements—as well as including project-specific timelines, interim project outputs/events and more information on how project results impact decision-making—would improve how ORD documents and communicates the benefits of regional research programs to the agency's mission. Algal toxin-tainted water addressed through RARE Project 1614. (EPA photo) 19-P-0123 6 ------- Offices Described Benefits and Impacts from Participating in Regional Research Programs Per the 2015 memorandum of understanding, the RSP "links ORD to the regions by building partnerships and networks, promoting collaborative research efforts, providing technical assistance, and delivering research results on high priority regional science issues." While the regions in our scope varied in how long their Regional Science Councils were in place and the processes they used to identify proposals for RARE and RESES funding (see sidebar), all agreed that the selected projects addressed high priorities. Interviewees also said that their processes are nimble and flexible to allow regions to address critical emerging issues. For example, after a drinking water crisis in Toledo, Ohio, Region 5 senior leaders worked with the Office of Science Policy to expedite funding for an already submitted RARE proposal, which, according to the Region 5 Deputy Regional Administrator, "includes much of the research needed to help water systems respond during toxic algal bloom events." Additionally, regional staff responsible for the projects in our sample indicated that both RARE and RESES projects have a very positive effect on the development of new ideas and approaches to environmental problems that affect the regions. Many said that these ideas would go unfunded were it not for these programs. Regional and ORD staff said that one great benefit of RARE and RESES projects was the collaborative exchange of ideas and information between the offices and their state and local partners, which often spurred new research and/or technologies—and facilitated community buy-in for these new efforts— to address regional needs. One RSL said, "ORD success stories are very likely RARE/RESES projects because those are the projects where the RSLs and STLs [Superfund and Technology Liaisons] have gone out and formed relationships with state stakeholders in the regions. They are successful because you have stakeholder input." Figure 4 describes impacts that our sample projects had on agency operations and decision-making. Various Council Processes Region 4: Annually solicits top priorities from regional leadership, and priorities accompany the call for proposals. Region 5: Branch chiefs or higher review RARE (but not RESES) proposals for alignment with priorities developed every 4 years. Region 7; Has a SharePoint site, meets quarterly with states and tribes on needs, and biannually develops a science needs list vetted with the Regional Administrator. Region 8: Meets monthly, holds an annual retreat each January and recently developed a list of science research priorities. 19-P-0123 7 ------- Figure 4: Impacts of sampled projects on agency operations and decision-making Region 4 Region 7 NE MO KS KY NC sc Region 5 Region 8 ND MT SD WY UT CO MN OH RARE Project 1685 helped develop performance standards for other sensor projects. RARE Project 1669 results were used as the basis for developing draft permit requirements. RARE Project 1963 produced data used for assessing damage from oil spills and choosing future sites for nutrient screening projects. RARE Project 1878 influenced decisions about sensors to be incorporated into current sensor pod designs for EPA's development of its Sensor Loan Program. RARE Project 1965 influenced the agency's decision to include a Superfund site on the National Priorities List and to work innovatively with states in the region. knowledge of air quality conditions for future placement of air monitors. RESES Project provided regional and municipal partners with RESES Project identified institutional context for remediation, restoration and community revitalization goals. Data can be used to inform similar projects in the future. RARE Project 1877 developed metrics to evaluate how different soils and climatic conditions may influence the design and performance of similar green infrastructure projects. RARE Project 1839 increased understanding of the links between ammonia concentrations with upwind emissions sources. Information will be used to determine whether to initiate an early warning system for ammonia concentrations. RARE Project 1614 provided critical information to water treatment plant operators in the Great Lakes area for minimizing tap water contamination during algal blooms (in response to a public health emergency in 2013). Source: OIG analysis of ORD and regional data. Additionally, we learned about other impacts through our interviews with project technical leads, such as the following examples: About RARE Project 1963 "This is a very exciting project and further development of this work has a lot of implications for state use to determine if waters are meeting nutrient goals." —Chief Scientist, Gulf of Mexico Program Office About RARE Project 1965 "The Chattanooga site was just listed on the National Priorities List. They used our methodology to determine that lead contamination was not a city- 19-P-0123 8 ------- wide problem, but rather, it was localized in a particular site." —Chief, Technical Support Section, Region 4 Superfund Program About RARE Project 1878 "It was foundational work in getting test data for sensors and viability for air monitoring purposes." —Chief, Region 7Monitoring and Environmental Sampling Branch About Region 4 RESES Project "We learned pros/cons (e.g., limitations on when a sensor can fail in certain conditions, resulting in lost data) of this particular sensor in these projects, and we gained expertise. As a result, we may move to different technologies in other deployments. We gained a huge knowledge base of the strengths and limitations." —Research Chemist, ORD's National Exposure Research Laboratory RARE projects in our sample appeared to have more immediate effects than RESES projects, which is consistent with the criteria and objectives that ORD has set for both programs. However, as we discuss in the section below, the RSP Tracker could be enhanced to allow EPA staff to more readily glean project progress and impacts, as well as to apply research results to new and related priority science needs. ORD Needs to Better Document Project Progress and Impacts In its October 2015 email introducing the RSP Tracker, ORD said the system communicates project-level information throughout the full life cycle of RARE projects—from the proposal through the final report and communications on project impact. Per the RARE Program Annual Process Guidelines document, which was issued in January 2018, and the 2015 memorandum of understanding, RSLs are responsible for documenting steps in the RSP Tracker to note that work products are completed, delivered in a timely fashion and effectively used in regional programs. However, for the RARE projects that we sampled, we found that the RSP Tracker entries and attachments did not always include key dates or depicted dates using different formats (Table 1). "For applied research you must be nimble. There are 80,000 chemicals and all need to be evaluated - RARE is a way to quickly get funds to emerging areas." —Region 5 Regional Science Council member 19-P-0123 9 ------- Table 1: Date information in the RSP Tracker for sampled RARE projects Region Project Proposal starta Proposal end a RSP start RSP end Final report 4 1963 -- -- 1/1/15 9/30/16 January 2018 1965 -- -- 10/1/14 9/30/16 Date Missing 5 1614 -- -- 12/1/14 7/1/16 Date Missing 1685 -- -- 7/1/15 6/1/18 Date Missing 7 1877 June 2014 June 2016 6/1/14 6/30/16 10/31/2017 1878 Spring 2014 Fiscal year 2016 6/1/15 12/31/17 Date Missing 8 1669 April 2014 April 2015 10/6/14 9/30/15 Date Missing 1839 July 2014 October 2015 10/3/16 b 10/4/16 b Date Missing Source: OIG analysis based on information and attachments in the RSP Tracker. a Prior to the issuance of the RARE Program Annual Process Guidelines in January 2018, there was no requirement for including start and end dates in proposals. b The "comments" section of the RSP Tracker was updated in May 2016 to clarify start and end dates of, respectively, October 2014 and November 2015. Absent complete or consistent dates, we could not always discern whether RARE project durations complied with the goal of a l-to-2-year project timeline. In addition, the Office of Science Policy uses the RSP Tracker to monitor projects, and RSLs rely upon the system to identify collaborative research. Incomplete data entry could make it difficult for stakeholders to use the system as a management tool. We also found that the RSP Tracker did not contain or document all project outputs or significant events. For example, for RARE Project 1685, we noted that the final report in the RSP Tracker listed eight outputs—six presentations and two fact sheets. However, only one fact sheet was posted to the system. Some regional project technical leads were unaware of the RSP Tracker and unsure whether they were responsible for entering project updates. The 2015 memorandum of understanding between ORD and the regions states that RSLs are required to provide feedback on behalf of their regions to ORD on the utility and impacts of science products on regional decision-making. Thq RARE Program Annual Process Guidelines document requires that RSLs maintain the RSP Tracker database. If RSLs do not post all outputs to the RSP Tracker, RARE projects could appear to languish when, in fact, the opposite may be true. ORD and regional staff provided us documentation (e.g., a sampling and analysis plan, a quality assurance plan, interim briefings and presentations) that were developed earlier than project end dates. Figure 5 highlights this inconsistency using one project as an example. 19-P-0123 10 ------- Figure 5: RARE Project 1963 status and outputs—the RSP Tracker versus interview data The RSP Tracker Start Date The RSP Tracker End Date = usingonly official documents = using interview information Final Report Project Initiated Fie Id work Laboratory Analysis Draft Final Report Final Report Regional Briefing Source: OIG analysis of RARE Project 1963 documentation and interviews. We found that the incomplete data stemmed from a combination of two factors: 1. RSL turnover. 2. The RSP Tracker was not developed until 2015 and introduced new data fields that had not been required for older projects. As of December 2018, based on our identification of the issue, the Office of Science Policy and RSLs were updating RARE project dates in the RSP Tracker. Additionally, ORD was revisiting its 2015 memorandum of understanding to increase RSL roles. Continued events like the monthly RSL calls conducted by ORD and the September 2018 RSL meeting could help identify additional best practices. The Office of Science Policy and the RSP also were in the process of developing an "Implementation Plan" that identifies best practices to further clarify RSL roles and responsibilities highlighted in the 2015 memorandum of understanding. Also, we found that RESES projects were not documented in the RSP Tracker. As of December 2018, ORD's SHC was working with the Office of Science Policy to incorporate RESES into the RSP Tracker. Our interviews with regional staff identified additional enhancements that would improve the utility of the RSP Tracker: • Region 4 officials thought that project staff and other interested parties would benefit from having a complete timeline of events recorded in the RSP Tracker. They said that it should be the responsibility of project staff to enter—or at least help enter—this information. • One RSL said that ORD should update its 2015 memorandum of understanding with the regions to require that RSLs use the RSP Tracker and increase awareness of the system among regional staff. The RSL said that noting the RSP Tracker requirements in the memorandum of 19-P-0123 11 ------- understanding could support RSL discussions with regional supervisors about the importance of data entry, as well as illustrate to regional supervisors the level of effort and time needed to keep the RSP Tracker current. • Some regional staff suggested a "pulse" or "tickler" (i.e., a reminder sent by the RSP Tracker) approximately a year after project completion to document research results and any actions or decisions taken based on projects. These enhancements could position the RSP Tracker as a one-stop resource for all regional research project information. They could also help RSLs communicate regional research programs and identify opportunities for collaboration with national colleagues on shared research priorities. Conclusion RARE and RESES projects contribute to regional research needs, but their impacts could be enhanced through additional documentation and information- sharing. Enhancing the RSP Tracker and maintaining a complete set of data about each RARE and RESES project in the RSP Tracker would assist RSLs in identifying similar research projects ongoing in other regions. These efforts would increase the effectiveness of programs that regional staff already greatly use to address priority science needs. Recommendations We recommend that the Assistant Administrator for Research and Development: 1. Complete data entry of all Regional Sustainability and Environmental Sciences projects into the Regional Science Program Tracker. 2. Verify and update information for Regional Applied Research Effort projects in the Regional Science Program Tracker. Air monitors to deploy for a Region 4 RESES project. (EPA photo) 19-P-0123 12 ------- 3. Update the Regional Science Program Tracker to improve Regional Applied Research Effort/Regional Sustainability and Environmental Sciences project tracking by including: a. A timeline with significant dates/milestones and events. b. Significant products/outputs that stem from a project, including interim products/outputs to show project progress prior to completion/final report. c. A feature to prompt staff to add impacts and/or evidence of use of project results in decision-making. 4. Update the Regional Applied Research Effort Program Annual Process Guidelines to require that Regional Science Liaisons use the Regional Science Program Tracker and increase awareness of the system among regional staff as a one-stop source of information on regional research projects. 5. Hold regular events where Regional Science Liaisons can share best practices on ways to increase regional communication on project opportunities and results. Agency Response and OIG Evaluation ORD concurred with all recommendations and provided corrective action dates. Recommendations 1 through 4 are resolved with corrective actions pending, with all planned for completion by October 1, 2020. Recommendation 5 was considered completed as of the date of ORD's response to the draft report (March 13, 2019). In its response, ORD committed to continue hosting biweekly meetings with RSLs in addition to annual face-to-face events. ORD indicated that it held its first annual RSL face-to-face meeting on September 27, 2018. The next meeting is planned for the fall of 2019. In addition, ORD provided specific suggestions for our consideration, and we applied edits as appropriate. Appendix B provides ORD's full response to the draft report. 19-P-0123 13 ------- Status of Recommendations and Potential Monetary Benefits RECOMMENDATIONS Rec. No. Page No. Subject Status1 Action Official Planned Completion Date Potential Monetary Benefits (in $000s) 1 12 Complete data entry of all Regional Sustainability and Environmental Sciences projects into the Regional Science Program Tracker. R Assistant Administrator for Research and Development 10/1/20 2 12 Verify and update information for Regional Applied Research Effort projects in the Regional Science Program Tracker. R Assistant Administrator for Research and Development 10/1/20 3 13 Update the Regional Science Program Tracker to improve Regional Applied Research Effort/Regional Sustainability and Environmental Sciences project tracking by including: a. A timeline with significant dates/milestones and events. b. Significant products/outputs that stem from a project, including interim products/outputs to show project progress prior to completion/final report. c. A feature to prompt staff to add impacts and/or evidence of use of project results in decision-making. R Assistant Administrator for Research and Development 10/1/20 4 13 Update the Regional Applied Research Effort Program Annual Process Guidelines to require that Regional Science Liaisons use the Regional Science Program Tracker and increase awareness of the system among regional staff as a one-stop source of information on regional research projects. R Assistant Administrator for Research and Development 10/1/20 5 13 Hold regular events where Regional Science Liaisons can share best practices on ways to increase regional communication on project opportunities and results. C Assistant Administrator for Research and Development 3/13/19 1 C = Corrective action completed. R = Recommendation resolved with corrective action pending. U = Recommendation unresolved with resolution efforts in progress. 19-P-0123 14 ------- Appendix A RARE and RESES Project Sample Project type and number Project title Fiscal years funded Area of focus Funding amount Region 4 RARE 1963 Water Quality and Aquatic Life Responses to Implementation of Best Management Practices in Gulf of Mexico Initiative Focus Watersheds: A Federal-State-Tribal Partnership 2015, 2016 Gulf of Mexico $172,000 RARE 1965 An Urban Contaminant Background Study 2015, 2016 Tampa, FL Orlando, FL Chattanooga, TN 172,000 RESES CitySpace Air Sensor Network: Evaluating Spatial Gradients of Urban Air Pollution with Low-Cost Air Sensor Technology 2016-2017 Shelby County, TN 145,000 Region 5 RARE 1614 Methods for Assessing the Water Quality Degradation through Water Treatment Plants during Algal Blooms 2015 Lake Erie 85,000 RARE 1685 Application of Lower Cost Air Monitoring Technologies for Local- scale Air Quality Investigations in an Environmental Justice Community 2015 Chicago, IL 155,000 RESES How the Relative Valuation of Ecosystem Goods and Services Empowers Communities to Impact the Outcomes of Remediation, Restoration, and Revitalization Projects 2016, 2017 Duluth, MN 100,000 Region 7 RARE 1877 Green Infrastructure Monitoring - Investigation of Soils for Green Infrastructure Implementation in Omaha, NE, Phase 2014 Omaha, NE 100,000 RARE 1878 Field Evaluation of Low Cost, Continuous Measurements of Air Pollutants 2013, 2014, 2015 Atlanta, GA Denver, CO * 125,000 Region 8 RARE 1669 Method to Evaluate the Capacity of Downgradient Ore Host Rock to Decrease Post-Restoration Contaminant Concentrations in ISR Wellfields and Protect Underground Sources of Drinking Water Downgradient for ISR Wellfields 2014, 2015 Black Hills, SD 200,000 RARE 1839 Ammonia Monitoring in Northeast Colorado 2016 Greeley, CO 59,000 Source: OIG analysis of RARE and RESES data. * While the project was funded for 3 years, it only received $125,000 in RARE funds in fiscal year 2015. 19-P-0123 15 ------- Appendix B Agency Comments on Draft Report UNITED STATES ENVIRONMENTAL PROTECTION AGENCY WASHINGTON, D.C, 20460 MAR 1 3 2019 OFFICE OF RESEARCH AND DEVELOPMENT MEMORANDUM SUBJECT: Response to Office of Inspector General (OIG) Draft Report No. OA&E-F Y18-0247 "Regional Research Programs Address Agency Needs and Could Benefit from Better Project Tracking" dated February 11, 2019 The EPA's Office of Research and Development (ORD) welcomes the opportunity to review and comment on the OIG's draft report titled, "Regional Research Programs Address Agency Needs and Could Benefit from Better Project Tracking" (Project No. OA&E-FY18-0247). The report evaluates ORD's Regional Science Programs (RSP), with a focus on the Regional Applied Research Effort (RARE) and the Regional Sustainability and Environmental Sciences (RESES) programs. Highlighted in the report is the value of RARE and RESES to the regions, the success of these programs in addressing the regions' high-priority, near-term research needs, and the direct impact that RARE and RESES have on Agency operations and decision-making. The "Noteworthy Achievements" portion of the report illustrates ORD's ongoing commitment to the success of these programs. ORD appreciates the OIG's recommendations for enhancing project tracking. Since the establishment of the RSP Tracker in 2015, we have been taking steps to improve the database, including adding RESES projects. We have also been working with our Regional Science Liaisons (RSL) to develop a document that captures best practices for implementing RSP program activities and tracking projects in the RSP Tracker. The report recommendations are of considerable value in helping us to further improve the tracking of research projects and impacts, as well as enhancing the visibility of the RARE and RESES programs. 19-P-0123 16 FROM: Jennifer Ortne-Zavaleta, Ph.D, Principal Deputy Assistant Adrr TO: Kevin Christensen, Assistant Inspector General Office of Inspector General ------- Below are ORD's responses to the OIG's recommendations. In the attachment, we provide additional detailed comments, including specific language suggestions to promote accuracy and clarity in the final report. Recommendation 1: Complete data entry of all Regional Sustainability and Environmental Sciences projects into the Regional Science Program Tracker. Response 1: ORD concurs with this recommendation and proposes the following corrective action and completion date. Corrective Action 1: ORD will continue to work with the support contractor to expand the RSP Tracker infrastructure to include RESES projects. ORD will work with the RSLs to complete data entry of RESES project records into the RSP tracker. Planned Completion Date: October 1, 2020 Recommendation 2: Verify and update information for Regional Applied Research Effort projects in the Regional Science Program Tracker. Response 2: ORD concurs with this recommendation and proposes the following corrective action and completion date. Corrective Action 2: As the OIG highlighted in the report on page 6, the RSP Tracker was launched in 2015 and includes new data fields that were not originally required for older projects. RSLs and OSP will verify and update information for all RARE projects in the RSP Tracker that were funded in 2015 and beyond. Planned Completion Date: October 1, 2020 Recommendation 3: Update the Regional Science Program Tracker to improve Regional Applied Research Effort/Regional Sustainability and Environmental Sciences project tracking by including: a. A timeline with significant dates/milestones and events. b. Significant products/outputs that stem from a project, including interim products/outputs to show project progress prior to completion/final report. c. A feature to prompt staff to add impacts and/or evidence of use of project results in decision-making. Response 3: ORD concurs with this recommendation and proposes the following corrective action and completion date. Corrective Action for Recommendations 3a and b: ORD will continue working with the support contractor to redesign and reconfigure RSP Tracker data fields to more intuitively display key milestones, status updates and interim and final project products. 19-P-0123 17 ------- Corrective Action for Recommendation 3c: ORD plans to update the RSP Tracker data fields to capture project impacts at project completion and post-completion. ORD will enable the system to send post-completion prompts to RARE and RESES project teams to add evidence of use and impacts of project results. Planned Completion Date: October 1, 2020 Recommendation 4: Update the Regional Applied Research Effort Program Annual Process Guidelines to require that Regional Science Liaisons use the Regional Science Program Tracker and increase awareness of the system among regional staff as a one-stop source of information on regional research projects. Response 4: ORD concurs with this recommendation and proposes the following corrective action and completion date. Corrective Action 4: Prior to sending out the latest solicitation in October 2018, ORD amended the RARE guidelines to clarify that RSLs are responsible for entering project data in the tracker and that ORD leads are responsible for providing that information to RSLs. Additionally, the draft RSL Implementation Plan identifies best practices for RSLs, including: ¦ Hosting events in regions related to RARE, RESES and other Regional Science Program opportunities ¦ Highlighting the RSP tracker database as a one-stop source of information on regional research projects ORD also plans to perform the following outreach on the RSP Tracker: ¦ Hosting webinars on the existing RSP tracker for ORD and the regions in coordination with the RSLs ¦ Developing a communication plan to roll out the fully revised RSP Tracker to ORD and the regions. Planned Completion Date: October 1, 2020 Recommendation 5: Hold regular events where Regional Science Liaisons can share best practices on ways to increase regional communication on project opportunities and results. Response 5: ORD concurs with this recommendation and proposes the following corrective action successfully addresses this recommendation. Corrective Action 5: In addition to current biweekly meetings with RSLs, ORD will continue to host annual RSL face-to-face events. The next RSL face-to-face meeting will be in Fall of 2019 and include sessions on building collaborations, identifying project opportunities, and improving communication of RARE and RESES results across the Agency. Completion Date: September 27, 2018 19-P-0123 18 ------- If you have any questions regarding this response, please contact Maureen Hingeley, Office of Research and Development, Office of Program Accountability and Resource Management at 202-564-1306. Attachment cc: Heather Cursio Kathleen Deener Andrew Geller Fred Hauchman Maureen Hingeley Maggie LaVay Sarah Mazur Michael Slimak Patrick Gilbride, OIG Erin Barnes-Weaver, OIG 19-P-0123 19 ------- Appendix C Distribution The Administrator Associate Deputy Administrator and Chief of Operations Chief of Staff Deputy Chief of Staff Assistant Administrator for Research and Development and EPA Science Advisor Agency Follow-Up Official (the CFO) Agency Follow-Up Coordinator General Counsel Associate Administrator for Congressional and Intergovernmental Relations Associate Administrator for Public Affairs Director, Office of Continuous Improvement, Office of the Administrator Principal Deputy Assistant Administrator for Science, Office of Research and Development Associate Director for Science, Office of Research and Development Deputy Assistant Administrator for Research and Development, Office of Research and Development Director, Office of Science Policy, Office of Research and Development Director, Office of the Science Advisor, Office of Research and Development Audit Follow-Up Coordinator, Office of the Administrator Audit Follow-Up Coordinator, Office of Research and Development Audit Follow-Up Coordinators, Regions 1-10 19-P-0123 20 ------- |