j A *
1®
U.S. ENVIRONMENTAL PROTECTION AGENCY
OFFICE OF INSPECTOR GENERAL
Improving EPA research programs
Regional Research Programs
Address Agency Needs but
Could Benefit from
Enhanced Project Tracking
Report No. 19-P-0123
April 18, 2019

-------
Report Contributors:
Patrick Gilbride
Erin Barnes-Weaver
Fred Light
Allison Krenzien
Raul Adrian
Abbreviations
EPA	U.S. Environmental Protection Agency
OIG	Office of Inspector General
ORD	Office of Research and Development
R2P2	Regional Research Partnership Program
RARE	Regional Applied Research Effort
RESES	Regional Sustainability and Environmental Sciences
ROCS-Net Regional-ORD Community of Science Networking
RSL	Regional Science Liaison
RSP	Regional Science Program
SHC	Sustainable and Healthy Communities
Cover Photo: Village Green station at Jane Addams Elementary School in Chicago, Illinois,
which is part of Region 5's RARE Project 1685. (EPA photo)
Are you aware of fraud, waste or abuse in an
EPA program?
EPA Inspector General Hotline
1200 Pennsylvania Avenue, NW (2431T)
Washington, DC 20460
(888) 546-8740
(202) 566-2599 (fax)
OIG Hotline@epa.gov
Learn more about our OIG Hotline.
EPA Office of Inspector General
1200 Pennsylvania Avenue, NW (2410T)
Washington, DC 20460
(202) 566-2391
www.epa.gov/oiq
Subscribe to our Email Updates
Follow us on Twitter @EPAoig
Send us your Project Suggestions

-------
^£DS7X
• JL v
I®/
U.S. Environmental Protection Agency
Office of Inspector General
At a Glance
19-P-0123
April 18, 2019
Why We Did This Project
We conducted this audit to
determine whether the Office
of Research and
Development's (ORD's)
support of regional research
programs helps accomplish the
U.S. Environmental Protection
Agency's (EPA's) mission. We
also examined whether results
of regional science initiatives
impact the agency's decision-
making.
Our audit focused on two
regional research programs:
1.	Regional Applied Research
Effort (RARE).
2.	Regional Sustainability and
Environmental Sciences
(RESES).
ORD has Regional Science
Liaisons (RSLs) within each
region who participate on
Regional Science Councils that
help identify RARE and RESES
projects. In 2015, ORD's Office
of Science Policy developed
the Regional Science Program
(RSP) Tracker database to
monitor RARE projects.
This report addresses the
following:
• Improving EPA research
programs.
Address inquiries to our public
affairs office at (202) 566-2391 or
OIG WEBCOMMENTS@epa.oiq.
List of OIG reports.
Regional Research Programs Address Agency Needs
but Could Benefit from Enhanced Project Tracking
What We Found
For the eight RARE and two RESES projects we
sampled, regional staff said that ORD's support
helped accomplish the agency's mission by
addressing the regions' high-priority, near-term
research needs. Additionally, regional and ORD
staff described positive impacts that these
projects had on agency operations and/or decision
benefits that their offices gained by participating in
Updates to the RSP tracking
system would improve how
ORD documents and
communicates the benefits
of regional research
programs.
-making, as well as other
regional research programs.
While interviewees expressed overall satisfaction with regional research
programs, we found that ORD could better document project progress, outputs
and impacts. Specifically, the RSP Tracker neither included RESES projects, nor
consistently included start and end dates, project outputs, or significant project
events for all RARE projects. The incomplete data stemmed from turnover among
RSL staff, as well as the 2015 adoption of the RSP Tracker, which featured
additional data fields not present for older projects. According to ORD, in 2018,
the office began steps to clarify RSL roles, including revisiting a 2015
memorandum of understanding between ORD and the regions, developing an
RSL implementation plan, and updating RARE guidelines. ORD said that it is also
updating the RSP Tracker to include RESES projects and RARE project dates.
Recommendations and Planned Agency Corrective Actions
We recommend that the Assistant Administrator for Research and Development
complete data entry of all RESES projects into the RSP Tracker; verify and
update information for RARE projects in the RSP Tracker; and update the RSP
Tracker to improve project tracking by documenting timelines, significant outputs
and how results were used in agency decision-making. We also recommend that
ORD require RSLs to use the RSP Tracker, increase awareness of the RSP
Tracker among regional staff and regularly hold events for RSLs to share best
practices. ORD concurred with all recommendations. One recommendation is
completed. All other recommendations are resolved with corrective actions
pending. The agency plans to complete all corrective actions by October 1, 2020.
Noteworthy Achievements
Prior to our audit, ORD launched a program evaluation—which it plans to
complete in fiscal year 2019—on RESES project accomplishments and impacts.
We also noted several effective practices relating to Regional Science Councils.
Lastly, in September 2018, RSLs met to discuss challenges, goals, best practices
and successes in their regions. One product resulting from that meeting was the
development of an implementation workgroup focused on using the identified
lessons learned to improve internal processes.

-------
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
OFFICE OF
INSPECTOR GENERAL
April 18, 2019
MEMORANDUM
SUBJECT: Regional Research Programs Address Agency Needs but Could Benefit
from Enhanced Project Tracking
Report No. 19-P-0123
FROM: Charles J. Sheehan, Acting Inspector General
TO:
Jennifer Orme-Zavaleta, Principal Deputy Assistant Administrator for Science
Office of Research and Development
This is our report on the subject audit conducted by the Office of Inspector General (OIG) of the
U.S. Environmental Protection Agency (EPA). The project number for this assignment was
OA&E-FY18-0247. This report contains findings that describe the problems the OIG has identified and
corrective actions the OIG recommends. This report represents the opinion of the OIG and does not
necessarily represent the final EPA position. Final determinations on matters in this report will be made
by EPA managers in accordance with established resolution procedures.
In accordance with EPA Manual 2750, your office provided acceptable corrective actions and milestone
dates in response to the OIG recommendations. One recommendation is completed. All other
recommendations are resolved, and no final response to this report is required. However, if you submit a
response, it will be posted on the OIG's website, along with our memorandum commenting on your
response. Your response should be provided as an Adobe PDF file that complies with the accessibility
requirements of Section 508 of the Rehabilitation Act of 1973, as amended. The final response should
not contain data that you do not want to be released to the public; if your response contains such data,
you should identify the data for redaction or removal along with corresponding justification.
We will post this report to our website at www.epa.gov/oig.

-------
Regional Research Programs Address	19-P-0123
Agency Needs but Could Benefit
from Enhanced Project Tracking
	 Table of C	
Purpose		1
Background		1
Responsible Office		4
Noteworthy Achievements		5
Scope and Methodology		5
Results		6
Offices Described Benefits and Impacts from Participating
in Regional Research Programs		7
ORD Needs to Better Document Project Progress and Impacts		9
Conclusion		12
Recommendations		12
Agency Response and OIG Evaluation		13
Status of Recommendations and Potential Monetary Benefits		14
Appendices
A RARE and RESES Project Sample		15
B Agency Comments on Draft Report		16
C Distribution		20

-------
Purpose
We conducted this audit to determine whether the Office of Research and
Development's (ORD's) support of regional research programs helps accomplish
the U.S. Environmental Protection Agency's (EPA's) mission. We also examined
whether results of regional science initiatives impact the agency's decision-
making.
Background
ORD's Regional Science Program
Science at the EPA provides the foundation for credible decision-making to
safeguard human health and the environment. ORD is the scientific research arm
of the EPA, and its research underpins all science and
technology used across the agency, including in the regions.
As highlighted in a 2015 memorandum of understanding
between ORD and the regions, ORD recognizes the
importance of science-based decision-making in the regions
and the value of the scientific support that it provides to
the regions.1
ORD's Office of Science Policy manages the Regional
Science Program (RSP), which addresses high-priority
regional science needs through collaborative research activities and technical
support. The RSP manages three regional research programs:
1.	Regional Applied Research Effort (RARE). This program fosters
collaboration between regions and ORD laboratories and centers, as well
as builds a regional/ORD network for future scientific interaction.
2.	Regional-ORD Community of Science Networking (ROCS-Net). This
orientation and networking program provides opportunities for regional
and state staff who have limited familiarity with ORD to visit an EPA
research facility and discuss science priorities and collaborative research
opportunities with ORD scientists.
3.	Regional Research Partnership Program (R2P2). This program provides
opportunities for regional technical staff to travel to an ORD laboratory,
center or office to work directly with ORD scientists on projects that target
specific regional priorities.
A 2015 memorandum of
understanding between ORD and
the regions "signifies the shared
commitment by ORD and the
regions to a strong partnership
that supports the Agency's
mission through effective
coordination, collaboration and
communication."
1 EPA, Regional Science Program Memorandum of Understanding Between the Office of Research and
Development and the Regions, December 23, 2015.
19-P-0123
1

-------
Within ORD, research is planned by national research programs, one of which is
the Sustainable and Healthy Communities (SHC) research program. The SHC
conducts research focused on remediating contaminated sites, sediments and
ground water; managing sustainable materials; and revitalizing communities
impacted by contamination or recovering from natural disasters or extreme
weather events. The SHC administers the Regional Sustainability and
Environmental Sciences (RESES) program, which provides resources for regions
and ORD researchers to collaborate on user-engaged research that advances
regional science needs to improve state and community environmental and public
health outcomes.
Our audit focused on RARE and RESES projects (Figure 1).
Figure 1: Regional research programs
Focus of OIG Audit i
SHC
Office of Science Policy
Source: Office of Inspector General (OIG) analysis.
Per the 2015 memorandum of understanding, RARE projects support near-term,
highly targeted research to address priority regional science needs. Most projects
are designed to be completed within 2 years. In addition to fostering collaboration
between the regions and ORD, RARE projects provide opportunities for ORD
scientists to apply their expertise to regional issues and explore new research
challenges.
The 2015 memorandum of understanding also notes that RESES projects emphasize
combining practical field measures and measurements with effective, long-term
community involvement. Although RESES projects are like RARE projects in that
they address regional science priorities and foster interoffice collaboration, RESES
projects must also align with the SHC mission and meet the criteria outlined in the
annual project solicitation, such as emphasizing community involvement. Through
RESES projects, ORD and the regions work together to assist states and
communities in improving environmental and public health outcomes.
Although overseen by different entities, solicitations for both RARE and RESES
projects require that all proposals be aligned with the Fiscal Year 2018-2022 EPA
Strategic Plan and the Administrator's priorities. Figure 2 summarizes
information about RARE and RESES projects.
19-P-0123
2

-------
Figure 2: RARE and RESES project information
RARE
/ Criteria: \
RESES


Clearly V'"
/ identified \


Management & Oversight: Office of
Science Policy
Project Timeline: 1-2 years
Tracking: Biannual reports through
RSP Tracker
Programmatic Emphasis: Targeted
research to address high priority
regional needs
Desired Outcomes: Regional, state
and/or local decisions based on sound
science
Project Selection: Regions with
input from ORD
Funding: $260,000/region/year
Alignment:
Strategic Plan
and
Administrator's
priorities
Solicitation:
Candidate
projects
identified by
Regional
Science
Liaisons
Management & Oversight: SHC
Project Timeline: 2-3 years
Tracking: Quarterly reports
Programmatic Emphasis: Set annually by
SHC (includes stakeholder engagement,
transferability and alignment with SHC)
Desired Outcomes: Science to inform
state and community decisions affecting
human health and the environment
Project Selection: SHC with input from
EPA peer review panel
Funding: Competitive process across all
regions ($850,000 for fiscal year 2019)
Source: OIG analysis with input from ORD.
Program Selection, Funding and Tracking
Within each region, ORD has Regional Science Liaisons (RSLs) who are full-
time regional employees funded by the Office of Science Policy and who, among
other duties, serve as the primary link between ORD's research programs and the
regional offices. Per ORD, "By being in close communication with regional
management and technical staff, the RSLs are uniquely situated to identify ORD
research that can impact high-priority regional policy and regulatory actions with
state-of-the-art science." RSLs often chair or participate
on Regional Science Councils that include other regional
staff and management. These councils help identify
potential RARE and RESES projects,2 and each council
has a slightly different methodology for this process.

ORD's Office of Science Policy has 4.6 full-time
equivalents in headquarters for the RSP. ORD allocates
funding for RARE projects so that ORD and each of the
EPA's 10 regional offices can pursue collaborative
research efforts. Each region conducts its own
solicitation process and, after ORD reviews the RARE
proposals, selects projects that best address the region's highest priority needs.
The Office of Science Policy oversees the funding process for chosen projects.
All RARE projects should be designed to generate products in 1 or 2 years.
Region 10's RSL presents at an RSL/ORD
poster session. (EPA photo)
All RESES funds are managed by ORD's SHC, which competitively selects
proposals put forth by regions. The SHC has two rounds of review when selecting
2 Superfund and Technology Liaisons, who report to the Office of Science Policy, also help select RARE and
RESES projects. Superfund and Technology Liaisons are the primary technical liaisons between ORD and the
EPA's regional offices on issues related to hazardous wastes. These liaisons work to ensure that the regions have
access to technical support that can help them make scientifically defensible decisions during site cleanups.
19-P-0123
3

-------
RESES projects: one by the SHC's Director and Deputy, and another by a panel
with organizational representation across laboratories and regions. Proposals must
be consistent with the direction of SHC programs and are ranked based on the six
criteria that were identified in the solicitation. Multi-region proposals that involve
two or more separate geographic locations are encouraged and may request
funding up to $200,000 per project. Projects that involve one geographic location
may request funding up to $150,000 per project. All RESES projects should be
designed to generate products in 2 or 3 years.
As of March 2019, ORD planned to allocate $2.6 million for the RARE program
($260,000 per region) and $850,000 for the RESES program in fiscal year 2019,
contingent on ORD's actual budget allocation. Figure 3 shows RSP and RESES
program funding over 5 fiscal years.
Figure
S 3,000,000
S 2,500,000
$i,«o,ooo
S 500,000
$0
FY 2014	FY 2015	FY 2016	FY 20!?	FY 2018
-#-«« — *• RESES —#- IBtWt	ROCS-Net
Source: OIG analysis of funding information provided by ORD.
Note: The ROCS-Net program started in fiscal year 2017.
In 2015, the Office of Science Policy developed the RSP Tracker database to
monitor RARE projects. The RSP Tracker is a searchable internal EPA system
that contains all RARE projects, including the project proposals, funding
information, dates and final reports on project results.
The SHC monitors RESES projects by requiring that the ORD project technical
leads submit quarterly reports using a standard template, meeting with principal
investigators and visiting project sites.
Responsible Office
ORD has primary responsibility for the issues discussed in this report.
3: RSP and RESES funding, fiscal years 2014-2018

19-P-0123
4

-------
Noteworthy Achievements
Before we began our audit, the SHC launched a program evaluation of RESES
project accomplishments and impacts on decision-making. The SHC plans to
complete the evaluation—which comprises case studies with related interviews
and a survey distributed to project technical leads—in fiscal year 2019.
Also, we noted several effective practices relating to Regional Science Councils:
•	The Region 4 Regional Science Council's charter includes a concise,
one-page table describing each regional research program, as well as ORD
and regional processes and resources.
•	Region 5's Regional Science Council comprises mostly branch chiefs
(or above) to help facilitate outreach within each division.
•	The Region 7 Regional Science Council was the first to include state and
tribal partners.
•	Region 8's Regional Science Council has a robust intranet site and
extensive resources, including a science needs form to facilitate
connecting regional scientists to ORD staff doing similar work.
Lastly, in September 2018, RSLs from across the agency met with the Office of
Science Policy to discuss challenges, goals, best practices and successes in their
regions. The meeting spurred the development of an implementation workgroup
focusing on how to use the identified "lessons learned" to improve internal RSL
processes.
Scope and Methodology
We conducted this audit from July 2018 to February 2019 in accordance with
generally accepted government auditing standards. Those standards require that we
plan and perform our work to obtain sufficient, appropriate evidence to provide a
reasonable basis for our findings and conclusions based on our audit objectives.
We believe that the evidence obtained provides a reasonable basis for the findings
and conclusions in this report based on our audit objectives.
We reviewed federal laws and agency guidance on regional research, as well as
EPA and ORD materials, including strategic plans, websites, policies and
procedures.
To understand ORD's regional research program coordination and support, we
interviewed ORD staff, as well as regional staff from Regions 4, 5, 7 and 8,
including all RSLs and members of the Regional Science Councils from those
regions. Of the 14 projects—12 RARE and two RESES—completed from fiscal
19-P-0123
5

-------
year 2014 through July 2018 in Regions 4, 5, 7 and 8, we judgmentally selected a
sample of eight RARE and both RESES projects. For each of these 10 completed
projects, we reviewed documentation available in the RSP Tracker or provided by
ORD, and we interviewed project technical leads in ORD and the regional offices
to discuss project selection and any impacts on agency decision-making.
Appendix A lists the projects sampled.
Results
For the 10 projects we sampled, regional staff said that ORD's support of regional
research programs helped accomplish the agency's mission by addressing the
regions' high-priority, near-term research needs. Additionally, regional and ORD
staff described impacts on agency operations and/or decision-making, as well as
other benefits that their offices gained by participating in regional research
programs. These results demonstrate that RARE and RESES projects are fulfilling
the objectives outlined in the 2015 memorandum of understanding between ORD
and the regions.
While interviewees expressed overall satisfaction with
ORD's regional research programs, we found that
ORD could better document project progress, outputs
and impacts on the agency's decision-making.
Specifically, we found that the RSP Tracker did not
include the RESES projects we sampled. In addition,
the RSP Tracker did not include all start and end
dates, consistent date formats, all outputs, or all
significant project events for the eight RARE projects
we sampled. These incomplete data stemmed from a
combination of two factors:
1.	RSL turnover.
2.	The RSP Tracker was not developed until 2015 and introduced new data
fields not present for older projects.
In its response to the draft report, ORD indicated that, in 2018, it began several
steps to clarify RSL roles, including revisiting its 2015 memorandum of
understanding, developing an RSL implementation plan and updating RARE
guidelines. According to ORD, it is also updating the RSP Tracker to include all
dates for RARE projects funded in 2015 and beyond and to enter all RESES
projects. These enhancements—as well as including project-specific timelines,
interim project outputs/events and more information on how project results impact
decision-making—would improve how ORD documents and communicates the
benefits of regional research programs to the agency's mission.
Algal toxin-tainted water
addressed through RARE
Project 1614. (EPA photo)
19-P-0123
6

-------
Offices Described Benefits and Impacts from Participating in
Regional Research Programs
Per the 2015 memorandum of understanding, the RSP "links
ORD to the regions by building partnerships and networks,
promoting collaborative research efforts, providing technical
assistance, and delivering research results on high priority
regional science issues." While the regions in our scope varied
in how long their Regional Science Councils were in place and
the processes they used to identify proposals for RARE and
RESES funding (see sidebar), all agreed that the selected
projects addressed high priorities.
Interviewees also said that their processes are nimble and
flexible to allow regions to address critical emerging issues. For
example, after a drinking water crisis in Toledo, Ohio, Region 5
senior leaders worked with the Office of Science Policy to
expedite funding for an already submitted RARE proposal,
which, according to the Region 5 Deputy Regional
Administrator, "includes much of the research needed to help
water systems respond during toxic algal bloom events."
Additionally, regional staff responsible for the projects in our sample indicated
that both RARE and RESES projects have a very positive effect on the
development of new ideas and approaches to environmental problems that affect
the regions. Many said that these ideas would go unfunded were it not for these
programs. Regional and ORD staff said that one great benefit of RARE and
RESES projects was the collaborative exchange of ideas and information between
the offices and their state and local partners, which often spurred new research
and/or technologies—and facilitated community buy-in for these new efforts—
to address regional needs. One RSL said, "ORD success stories are very likely
RARE/RESES projects because those are the projects where the RSLs and STLs
[Superfund and Technology Liaisons] have gone out and formed relationships
with state stakeholders in the regions. They are successful because you have
stakeholder input."
Figure 4 describes impacts that our sample projects had on agency operations and
decision-making.
Various Council Processes
Region 4: Annually solicits top
priorities from regional leadership,
and priorities accompany the call
for proposals.
Region 5: Branch chiefs or higher
review RARE (but not RESES)
proposals for alignment with
priorities developed every 4 years.
Region 7; Has a SharePoint site,
meets quarterly with states and
tribes on needs, and biannually
develops a science needs list
vetted with the Regional
Administrator.
Region 8: Meets monthly, holds
an annual retreat each January
and recently developed a list of
science research priorities.
19-P-0123
7

-------
Figure 4: Impacts of sampled projects on agency operations and decision-making
Region 4
Region 7
NE
MO
KS
KY
NC
sc
Region 5
Region 8
ND
MT
SD
WY
UT
CO
MN
OH
RARE Project
1685 helped
develop
performance
standards for
other sensor
projects.
RARE Project
1669 results
were used as
the basis for
developing
draft permit
requirements.
RARE Project
1963 produced
data used for
assessing
damage from oil
spills and
choosing future
sites for nutrient
screening
projects.
RARE Project 1878
influenced decisions
about sensors to be
incorporated into
current sensor pod
designs for EPA's
development of its
Sensor Loan
Program.
RARE Project 1965 influenced the agency's
decision to include a Superfund site on the
National Priorities List and to work
innovatively with states in the region.
knowledge of
air quality
conditions for
future
placement of
air monitors.
RESES Project
provided
regional and
municipal
partners with
RESES Project identified institutional context for remediation,
restoration and community revitalization goals. Data can be
used to inform similar projects in the future.
RARE Project 1877
developed metrics to
evaluate how different
soils and climatic
conditions may
influence the design
and performance of
similar green
infrastructure projects.
RARE Project 1839 increased understanding of
the links between ammonia concentrations with
upwind emissions sources. Information will be
used to determine whether to initiate an early
warning system for ammonia concentrations.
RARE Project 1614
provided critical
information to water
treatment plant
operators in the Great
Lakes area for
minimizing tap water
contamination during
algal blooms (in
response to a public
health emergency in
2013).
Source: OIG analysis of ORD and regional data.
Additionally, we learned about other impacts through our interviews with project
technical leads, such as the following examples:
About RARE Project 1963
"This is a very exciting project and further development of this work has a
lot of implications for state use to determine if waters are meeting nutrient
goals." —Chief Scientist, Gulf of Mexico Program Office
About RARE Project 1965
"The Chattanooga site was just listed on the National Priorities List. They
used our methodology to determine that lead contamination was not a city-
19-P-0123
8

-------
wide problem, but rather, it was localized in a particular site." —Chief,
Technical Support Section, Region 4 Superfund Program
About RARE Project 1878
"It was foundational work in getting test data for sensors and viability for
air monitoring purposes." —Chief, Region 7Monitoring and
Environmental Sampling Branch
About Region 4 RESES Project
"We learned pros/cons (e.g., limitations on when a sensor can fail in
certain conditions, resulting in lost data) of this particular sensor in these
projects, and we gained expertise. As a result, we may move to different
technologies in other deployments. We gained a huge knowledge base of
the strengths and limitations." —Research Chemist, ORD's National
Exposure Research Laboratory
RARE projects in our sample appeared to have
more immediate effects than RESES projects,
which is consistent with the criteria and objectives
that ORD has set for both programs. However, as
we discuss in the section below, the RSP Tracker
could be enhanced to allow EPA staff to more
readily glean project progress and impacts, as well
as to apply research results to new and related
priority science needs.
ORD Needs to Better Document Project Progress and Impacts
In its October 2015 email introducing the RSP Tracker, ORD said the system
communicates project-level information throughout the full life cycle of RARE
projects—from the proposal through the final report and communications on
project impact. Per the RARE Program Annual Process Guidelines document,
which was issued in January 2018, and the 2015 memorandum of understanding,
RSLs are responsible for documenting steps in the RSP Tracker to note that work
products are completed, delivered in a timely fashion and effectively used in
regional programs. However, for the RARE projects that we sampled, we found
that the RSP Tracker entries and attachments did not always include key dates or
depicted dates using different formats (Table 1).
"For applied research you
must be nimble. There are
80,000 chemicals and all
need to be evaluated -
RARE is a way to quickly
get funds to emerging
areas."
—Region 5 Regional
Science Council member
19-P-0123
9

-------
Table 1: Date information in the RSP Tracker for sampled RARE projects
Region
Project
Proposal
starta
Proposal
end a
RSP
start
RSP
end
Final report
4
1963
--
--
1/1/15
9/30/16
January 2018

1965
--
--
10/1/14
9/30/16
Date Missing
5
1614
--
--
12/1/14
7/1/16
Date Missing

1685
--
--
7/1/15
6/1/18
Date Missing
7
1877
June 2014
June 2016
6/1/14
6/30/16
10/31/2017

1878
Spring 2014
Fiscal year
2016
6/1/15
12/31/17
Date Missing
8
1669
April 2014
April 2015
10/6/14
9/30/15
Date Missing

1839
July 2014
October 2015
10/3/16 b
10/4/16 b
Date Missing
Source: OIG analysis based on information and attachments in the RSP Tracker.
a Prior to the issuance of the RARE Program Annual Process Guidelines in January 2018,
there was no requirement for including start and end dates in proposals.
b The "comments" section of the RSP Tracker was updated in May 2016 to clarify start and end
dates of, respectively, October 2014 and November 2015.
Absent complete or consistent dates, we could not always discern whether RARE
project durations complied with the goal of a l-to-2-year project timeline. In
addition, the Office of Science Policy uses the RSP Tracker to monitor projects,
and RSLs rely upon the system to identify collaborative research. Incomplete data
entry could make it difficult for stakeholders to use the system as a management
tool.
We also found that the RSP Tracker did not contain or document all project
outputs or significant events. For example, for RARE Project 1685, we noted that
the final report in the RSP Tracker listed eight outputs—six presentations and two
fact sheets. However, only one fact sheet was posted to the system. Some regional
project technical leads were unaware of the RSP Tracker and unsure whether they
were responsible for entering project updates. The 2015 memorandum of
understanding between ORD and the regions states that RSLs are required to
provide feedback on behalf of their regions to ORD on the utility and impacts of
science products on regional decision-making. Thq RARE Program Annual
Process Guidelines document requires that RSLs maintain the RSP Tracker
database. If RSLs do not post all outputs to the RSP Tracker, RARE projects
could appear to languish when, in fact, the opposite may be true. ORD and
regional staff provided us documentation (e.g., a sampling and analysis plan, a
quality assurance plan, interim briefings and presentations) that were developed
earlier than project end dates. Figure 5 highlights this inconsistency using one
project as an example.
19-P-0123
10

-------
Figure 5: RARE Project 1963 status and outputs—the RSP Tracker versus interview data
The RSP Tracker
Start Date
The RSP Tracker
End Date
= usingonly official documents
= using interview information
Final Report
Project
Initiated
Fie Id work
Laboratory
Analysis
Draft Final
Report
Final
Report
Regional
Briefing
Source: OIG analysis of RARE Project 1963 documentation and interviews.
We found that the incomplete data stemmed from a combination of two factors:
1.	RSL turnover.
2.	The RSP Tracker was not developed until 2015 and introduced new data
fields that had not been required for older projects.
As of December 2018, based on our identification of the issue, the Office of
Science Policy and RSLs were updating RARE project dates in the RSP Tracker.
Additionally, ORD was revisiting its 2015 memorandum of understanding to
increase RSL roles. Continued events like the monthly RSL calls conducted by
ORD and the September 2018 RSL meeting could help identify additional best
practices. The Office of Science Policy and the RSP also were in the process of
developing an "Implementation Plan" that identifies best practices to further
clarify RSL roles and responsibilities highlighted in the 2015 memorandum of
understanding.
Also, we found that RESES projects were not documented in the RSP Tracker. As
of December 2018, ORD's SHC was working with the Office of Science Policy to
incorporate RESES into the RSP Tracker.
Our interviews with regional staff identified additional enhancements that would
improve the utility of the RSP Tracker:
•	Region 4 officials thought that project staff and other interested parties
would benefit from having a complete timeline of events recorded in the
RSP Tracker. They said that it should be the responsibility of project staff
to enter—or at least help enter—this information.
•	One RSL said that ORD should update its 2015 memorandum of
understanding with the regions to require that RSLs use the RSP Tracker
and increase awareness of the system among regional staff. The RSL said
that noting the RSP Tracker requirements in the memorandum of
19-P-0123
11

-------
understanding could support RSL
discussions with regional supervisors
about the importance of data entry, as
well as illustrate to regional supervisors
the level of effort and time needed to keep
the RSP Tracker current.
• Some regional staff suggested a "pulse"
or "tickler" (i.e., a reminder sent by the
RSP Tracker) approximately a year after
project completion to document research
results and any actions or decisions taken
based on projects.
These enhancements could position the RSP
Tracker as a one-stop resource for all regional
research project information. They could also
help RSLs communicate regional research
programs and identify opportunities for
collaboration with national colleagues on shared
research priorities.
Conclusion
RARE and RESES projects contribute to regional research needs, but their
impacts could be enhanced through additional documentation and information-
sharing. Enhancing the RSP Tracker and maintaining a complete set of data about
each RARE and RESES project in the RSP Tracker would assist RSLs in
identifying similar research projects ongoing in other regions. These efforts would
increase the effectiveness of programs that regional staff already greatly use to
address priority science needs.
Recommendations
We recommend that the Assistant Administrator for Research and Development:
1.	Complete data entry of all Regional Sustainability and Environmental
Sciences projects into the Regional Science Program Tracker.
2.	Verify and update information for Regional Applied Research Effort
projects in the Regional Science Program Tracker.
Air monitors to deploy for a
Region 4 RESES project.
(EPA photo)
19-P-0123
12

-------
3.	Update the Regional Science Program Tracker to improve Regional
Applied Research Effort/Regional Sustainability and Environmental
Sciences project tracking by including:
a.	A timeline with significant dates/milestones and events.
b.	Significant products/outputs that stem from a project, including
interim products/outputs to show project progress prior to
completion/final report.
c.	A feature to prompt staff to add impacts and/or evidence of use of
project results in decision-making.
4.	Update the Regional Applied Research Effort Program Annual Process
Guidelines to require that Regional Science Liaisons use the Regional
Science Program Tracker and increase awareness of the system among
regional staff as a one-stop source of information on regional research
projects.
5.	Hold regular events where Regional Science Liaisons can share best
practices on ways to increase regional communication on project
opportunities and results.
Agency Response and OIG Evaluation
ORD concurred with all recommendations and provided corrective action dates.
Recommendations 1 through 4 are resolved with corrective actions pending, with
all planned for completion by October 1, 2020. Recommendation 5 was
considered completed as of the date of ORD's response to the draft report
(March 13, 2019). In its response, ORD committed to continue hosting biweekly
meetings with RSLs in addition to annual face-to-face events. ORD indicated that
it held its first annual RSL face-to-face meeting on September 27, 2018. The next
meeting is planned for the fall of 2019.
In addition, ORD provided specific suggestions for our consideration, and we
applied edits as appropriate. Appendix B provides ORD's full response to the
draft report.
19-P-0123
13

-------
Status of Recommendations and
Potential Monetary Benefits
RECOMMENDATIONS
Rec.
No.
Page
No.
Subject
Status1
Action Official
Planned
Completion
Date
Potential
Monetary
Benefits
(in $000s)
1
12
Complete data entry of all Regional Sustainability and
Environmental Sciences projects into the Regional Science
Program Tracker.
R
Assistant Administrator for
Research and Development
10/1/20

2
12
Verify and update information for Regional Applied Research
Effort projects in the Regional Science Program Tracker.
R
Assistant Administrator for
Research and Development
10/1/20

3
13
Update the Regional Science Program Tracker to improve
Regional Applied Research Effort/Regional Sustainability and
Environmental Sciences project tracking by including:
a.	A timeline with significant dates/milestones and events.
b.	Significant products/outputs that stem from a project,
including interim products/outputs to show project
progress prior to completion/final report.
c.	A feature to prompt staff to add impacts and/or
evidence of use of project results in decision-making.
R
Assistant Administrator for
Research and Development
10/1/20

4
13
Update the Regional Applied Research Effort Program Annual
Process Guidelines to require that Regional Science Liaisons
use the Regional Science Program Tracker and increase
awareness of the system among regional staff as a one-stop
source of information on regional research projects.
R
Assistant Administrator for
Research and Development
10/1/20

5
13
Hold regular events where Regional Science Liaisons can
share best practices on ways to increase regional
communication on project opportunities and results.
C
Assistant Administrator for
Research and Development
3/13/19

1 C = Corrective action completed.
R = Recommendation resolved with corrective action pending.
U = Recommendation unresolved with resolution efforts in progress.
19-P-0123
14

-------
Appendix A
RARE and RESES Project Sample
Project type
and number
Project title
Fiscal years
funded
Area of focus
Funding
amount
Region 4
RARE 1963
Water Quality and Aquatic Life
Responses to Implementation of
Best Management Practices in Gulf
of Mexico Initiative Focus
Watersheds: A Federal-State-Tribal
Partnership
2015, 2016
Gulf of Mexico
$172,000
RARE 1965
An Urban Contaminant Background
Study
2015, 2016
Tampa, FL
Orlando, FL
Chattanooga, TN
172,000
RESES
CitySpace Air Sensor Network:
Evaluating Spatial Gradients of
Urban Air Pollution with Low-Cost Air
Sensor Technology
2016-2017
Shelby County, TN
145,000
Region 5
RARE 1614
Methods for Assessing the Water
Quality Degradation through Water
Treatment Plants during Algal
Blooms
2015
Lake Erie
85,000
RARE 1685
Application of Lower Cost Air
Monitoring Technologies for Local-
scale Air Quality Investigations in an
Environmental Justice Community
2015
Chicago, IL
155,000
RESES
How the Relative Valuation of
Ecosystem Goods and Services
Empowers Communities to Impact
the Outcomes of Remediation,
Restoration, and Revitalization
Projects
2016, 2017
Duluth, MN
100,000
Region 7
RARE 1877
Green Infrastructure Monitoring -
Investigation of Soils for Green
Infrastructure Implementation in
Omaha, NE, Phase
2014
Omaha, NE
100,000
RARE 1878
Field Evaluation of Low Cost,
Continuous Measurements of Air
Pollutants
2013, 2014,
2015
Atlanta, GA
Denver, CO
* 125,000
Region 8
RARE 1669
Method to Evaluate the Capacity of
Downgradient Ore Host Rock to
Decrease Post-Restoration
Contaminant Concentrations in ISR
Wellfields and Protect Underground
Sources of Drinking Water
Downgradient for ISR Wellfields
2014, 2015
Black Hills, SD
200,000
RARE 1839
Ammonia Monitoring in Northeast
Colorado
2016
Greeley, CO
59,000
Source: OIG analysis of RARE and RESES data.
* While the project was funded for 3 years, it only received $125,000 in RARE funds in fiscal year 2015.
19-P-0123
15

-------
Appendix B
Agency Comments on Draft Report
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C, 20460
MAR 1 3 2019
OFFICE OF
RESEARCH AND DEVELOPMENT
MEMORANDUM
SUBJECT: Response to Office of Inspector General (OIG) Draft Report No. OA&E-F Y18-0247
"Regional Research Programs Address Agency Needs and Could Benefit from
Better Project Tracking" dated February 11, 2019
The EPA's Office of Research and Development (ORD) welcomes the opportunity to review and
comment on the OIG's draft report titled, "Regional Research Programs Address Agency Needs
and Could Benefit from Better Project Tracking" (Project No. OA&E-FY18-0247). The report
evaluates ORD's Regional Science Programs (RSP), with a focus on the Regional Applied
Research Effort (RARE) and the Regional Sustainability and Environmental Sciences (RESES)
programs. Highlighted in the report is the value of RARE and RESES to the regions, the success
of these programs in addressing the regions' high-priority, near-term research needs, and the
direct impact that RARE and RESES have on Agency operations and decision-making. The
"Noteworthy Achievements" portion of the report illustrates ORD's ongoing commitment to the
success of these programs.
ORD appreciates the OIG's recommendations for enhancing project tracking. Since the
establishment of the RSP Tracker in 2015, we have been taking steps to improve the database,
including adding RESES projects. We have also been working with our Regional Science
Liaisons (RSL) to develop a document that captures best practices for implementing RSP
program activities and tracking projects in the RSP Tracker. The report recommendations are of
considerable value in helping us to further improve the tracking of research projects and impacts,
as well as enhancing the visibility of the RARE and RESES programs.
19-P-0123	16
FROM: Jennifer Ortne-Zavaleta, Ph.D,
Principal Deputy Assistant Adrr

TO:
Kevin Christensen, Assistant Inspector General
Office of Inspector General

-------
Below are ORD's responses to the OIG's recommendations. In the attachment, we provide
additional detailed comments, including specific language suggestions to promote accuracy and
clarity in the final report.
Recommendation 1: Complete data entry of all Regional Sustainability and Environmental
Sciences projects into the Regional Science Program Tracker.
Response 1: ORD concurs with this recommendation and proposes the following
corrective action and completion date.
Corrective Action 1: ORD will continue to work with the support contractor to expand
the RSP Tracker infrastructure to include RESES projects. ORD will work with the RSLs
to complete data entry of RESES project records into the RSP tracker.
Planned Completion Date: October 1, 2020
Recommendation 2: Verify and update information for Regional Applied Research Effort
projects in the Regional Science Program Tracker.
Response 2: ORD concurs with this recommendation and proposes the following
corrective action and completion date.
Corrective Action 2: As the OIG highlighted in the report on page 6, the RSP Tracker
was launched in 2015 and includes new data fields that were not originally required for
older projects. RSLs and OSP will verify and update information for all RARE projects
in the RSP Tracker that were funded in 2015 and beyond.
Planned Completion Date: October 1, 2020
Recommendation 3: Update the Regional Science Program Tracker to improve Regional
Applied Research Effort/Regional Sustainability and Environmental Sciences project
tracking by including:
a.	A timeline with significant dates/milestones and events.
b.	Significant products/outputs that stem from a project, including interim
products/outputs to show project progress prior to completion/final report.
c.	A feature to prompt staff to add impacts and/or evidence of use of project results
in decision-making.
Response 3: ORD concurs with this recommendation and proposes the following
corrective action and completion date.
Corrective Action for Recommendations 3a and b: ORD will continue working with
the support contractor to redesign and reconfigure RSP Tracker data fields to more
intuitively display key milestones, status updates and interim and final project products.
19-P-0123
17

-------
Corrective Action for Recommendation 3c: ORD plans to update the RSP Tracker data
fields to capture project impacts at project completion and post-completion. ORD will
enable the system to send post-completion prompts to RARE and RESES project teams
to add evidence of use and impacts of project results.
Planned Completion Date: October 1, 2020
Recommendation 4: Update the Regional Applied Research Effort Program Annual Process
Guidelines to require that Regional Science Liaisons use the Regional Science Program
Tracker and increase awareness of the system among regional staff as a one-stop source of
information on regional research projects.
Response 4: ORD concurs with this recommendation and proposes the following
corrective action and completion date.
Corrective Action 4: Prior to sending out the latest solicitation in October 2018, ORD
amended the RARE guidelines to clarify that RSLs are responsible for entering project
data in the tracker and that ORD leads are responsible for providing that information to
RSLs. Additionally, the draft RSL Implementation Plan identifies best practices for
RSLs, including:
¦	Hosting events in regions related to RARE, RESES and other Regional Science
Program opportunities
¦	Highlighting the RSP tracker database as a one-stop source of information on
regional research projects
ORD also plans to perform the following outreach on the RSP Tracker:
¦	Hosting webinars on the existing RSP tracker for ORD and the regions in
coordination with the RSLs
¦	Developing a communication plan to roll out the fully revised RSP Tracker to
ORD and the regions.
Planned Completion Date: October 1, 2020
Recommendation 5: Hold regular events where Regional Science Liaisons can share best
practices on ways to increase regional communication on project opportunities and results.
Response 5: ORD concurs with this recommendation and proposes the following
corrective action successfully addresses this recommendation.
Corrective Action 5: In addition to current biweekly meetings with RSLs, ORD will
continue to host annual RSL face-to-face events. The next RSL face-to-face meeting will
be in Fall of 2019 and include sessions on building collaborations, identifying project
opportunities, and improving communication of RARE and RESES results across the
Agency.
Completion Date: September 27, 2018
19-P-0123
18

-------
If you have any questions regarding this response, please contact Maureen Hingeley, Office of
Research and Development, Office of Program Accountability and Resource Management at
202-564-1306.
Attachment
cc: Heather Cursio
Kathleen Deener
Andrew Geller
Fred Hauchman
Maureen Hingeley
Maggie LaVay
Sarah Mazur
Michael Slimak
Patrick Gilbride, OIG
Erin Barnes-Weaver, OIG
19-P-0123
19

-------
Appendix C
Distribution
The Administrator
Associate Deputy Administrator and Chief of Operations
Chief of Staff
Deputy Chief of Staff
Assistant Administrator for Research and Development and EPA Science Advisor
Agency Follow-Up Official (the CFO)
Agency Follow-Up Coordinator
General Counsel
Associate Administrator for Congressional and Intergovernmental Relations
Associate Administrator for Public Affairs
Director, Office of Continuous Improvement, Office of the Administrator
Principal Deputy Assistant Administrator for Science, Office of Research and Development
Associate Director for Science, Office of Research and Development
Deputy Assistant Administrator for Research and Development, Office of Research and
Development
Director, Office of Science Policy, Office of Research and Development
Director, Office of the Science Advisor, Office of Research and Development
Audit Follow-Up Coordinator, Office of the Administrator
Audit Follow-Up Coordinator, Office of Research and Development
Audit Follow-Up Coordinators, Regions 1-10
19-P-0123
20

-------