f O
U.S. ENVIRONMENTAL PROTECTION AGENCY
OFFICE OF INSPECTOR GENERAL
Improving EPA research programs
EPA Needs a Comprehensive
Vision and Strategy for
Citizen Science that Aligns
with Its Strategic Objectives
on Public Participation
Report No. 18-P-0240
September 5, 2018
-------
Report Contributors:
Patrick Gilbride
Erin Barnes-Weaver
Raul Adrian
Monica Brym
Abbreviations
EPA
U.S. Environmental Protection Agency
GAO
Government Accountability Office
GSA
General Services Administration
NACEPT
National Advisory Council for Environmental Policy and Technology
OEI
Office of Environmental Information
OMB
Office of Management and Budget
ORD
Office of Research and Development
OSTP
Office of Science and Technology Policy
QAPP
Quality Assurance Project Plan
QMP
Quality Management Plan
Cover Photos: Photos involving various citizen science projects. From left: citizen scientists
working with air monitoring equipment; a prototype open source sensor system
used for water quality monitoring; EPA staff and local partners evaluating the
effectiveness of vegetation and noise barriers near a school. (EPA photos)
Are you aware of fraud, waste or abuse in an
EPA program?
EPA Inspector General Hotline
1200 Pennsylvania Avenue, NW (2431T)
Washington, DC 20460
(888) 546-8740
(202) 566-2599 (fax)
OIG Hotline@epa.gov
Learn more about our OIG Hotline.
EPA Office of Inspector General
1200 Pennsylvania Avenue, NW (2410T)
Washington, DC 20460
(202) 566-2391
www.epa.gov/oiq
Subscribe to our Email Updates
Follow us on Twitter @EPAoig
Send us your Project Suggestions
-------
x-^tD ST/ij.
* U.S. Environmental Protection Agency 18-P-0240
I" mm \ Office of Inspector General Septembers, 2018
At a Glance
Why We Did This Project
We conducted this audit to
determine whether the
U.S. Environmental Protection
Agency (EPA) has developed
controls to manage the use of
citizen science results to meet
the agency's mission.
Citizen science is a form of
open collaboration in which
individuals or organizations
participate voluntarily in the
scientific process in various
ways, including collecting and
analyzing data. Citizen science
provides a way for members of
the public to participate and
support EPA programs.
On September 30, 2015, the
White House Office of Science
and Technology Policy
recommended actions to build
citizen science capacity and
directed agencies to take
specific steps to advance
application of citizen science.
Within the EPA, a team in the
Office of Research and
Development facilitates citizen
science projects implemented
throughout the agency's
program and regional offices.
This report addresses the
following:
• Improving EPA research
programs.
Send all inquiries to our public
affairs office at (202) 566-2391
or visit www.epa.gov/oiq.
EPA Needs a Comprehensive Vision and
Strategy for Citizen Science that Aiigns with its
Strategic Objectives on Public Participation
What We Found
Although citizen science is carried out throughout the
EPA, the agency has not developed controls
necessary to manage citizen science agencywide,
including a clear vision and objectives for using
results. Absent this, the EPA cannot undertake a
systematic effort to analyze the risks and
opportunities that citizen science presents.
Without uniform
guidance and direction,
the EPA will be unable
to fully use citizen
science data that could
contribute to the
agency's mission.
EPA staff identified barriers to effectively using citizen science results—including
lack of a comprehensive vision and support/resources from senior management,
and lack of understanding and buy-in for citizen science—that exist because EPA
leadership has not developed a strategy for citizen science. Citizen science is
evolving as advancements in technology provide greater access to the public,
and as public involvement grows it will place pressure on the EPA to understand
and determine how to use the data collected and provided to the agency.
Recommendations and Planned Agency Corrective Actions
We recommend that the EPA Deputy Administrator establish a strategic vision
and objectives for citizen science, and direct completion of an assessment to
identify the data management requirements for using citizen science data and an
action plan. Further, we recommend that the Office of Research and
Development finalize a draft handbook for citizen science and build the capacity
for managing the use of citizen science. The agency concurred with our
recommendations and indicated that it plans to convene an agencywide
workgroup to oversee implementation of our recommendations, and it plans to
complete corrective actions by December 31, 2020.
Noteworthy Achievements
Prior to the start of our audit, the EPA had begun to develop a checklist of
administrative and legal considerations for citizen science projects. The EPA had
also drafted an outreach tool that highlights a representative distribution of citizen
science projects at the EPA. Additionally, the EPA has taken the lead among
federal agencies in addressing a barrier related to the Paperwork Reduction Act
by obtaining approval for an Information Collection Request (generic clearance
for citizen science) for projects contributing to EPA research.
Listing of OIG reports.
-------
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
THE INSPECTOR GENERAL
September 5, 2018
MEMORANDUM
SUBJECT: EPA Needs a Comprehensive Vision and Strategy for Citizen Science that
Aligns with Its Strategic Objectives on Public Participation
Report No. 18-P-0240
FROM: Arthur A. Elkins Jr.
TO:
Henry Darwin, Acting Deputy Administrator
Jennifer Orme-Zavaleta, Principal Deputy Assistant Administrator for Science
Office of Research and Development
This is our report on the subject audit conducted by the Office of Inspector General (OIG) of the
U.S. Environmental Protection Agency (EPA). The project number for this audit was OPE-FY18-0002.
This report contains findings that describe the problems the OIG has identified and corrective actions the
OIG recommends. This report represents the opinion of the OIG and does not necessarily represent the
final EPA position.
The Office of the Administrator and the Office of the Science Advisor within the Office of Research and
Development have the primary responsibility for the issues discussed in this report.
In accordance with EPA Manual 2750, your office provided acceptable corrective actions and milestone
dates in response to OIG recommendations. All recommendations are resolved and no final response to
this report is required. However, if you submit a response, it will be posted on the OIG's website, along
with our memorandum commenting on your response. Your response should be provided as an Adobe
PDF file that complies with the accessibility requirements of Section 508 of the Rehabilitation Act of
1973, as amended. The final response should not contain data that you do not want to be released to the
public; if your response contains such data, you should identify the data for redaction or removal along
with corresponding justification.
We will post this report to our website at www.epa.gov/oig.
-------
EPA Needs a Comprehensive Vision and
Strategy for Citizen Science that Aligns with
Its Strategic Objectives on Public Participation
18-P-0240
Table of C
Chapters
1 Introduction 1
Purpose 1
Background 1
Responsible Offices 5
Noteworthy Achievements 5
Scope and Methodology 5
2 EPA Does Not Manage Citizen Science from an Agencywide
Perspective and Should Complete Actions to Address Barriers 8
EPA Uses Citizen Science Primarily for Community Engagement,
Research, Monitoring and Environmental Education 8
EPA Has Not Developed Controls for Managing Use of
Citizen Science, but Does Have Controls for Quality and
Use of Environmental Data 11
Barriers Exist to Using Citizen Science Results in EPA's Work 14
EPA Needs to Complete Actions to Mitigate Barriers 16
Conclusion 17
Recommendations 18
Agency Response and OIG Evaluation 18
Status of Recommendations and Potential Monetary Benefits 19
Appendices
A Information on Sampled Citizen Science Projects 20
B Details on Barriers to Using Citizen Science Results 23
C Agency Response to Draft Report 27
D Distribution 32
-------
Chapter 1
Introduction
Purpose
We conducted this audit to determine whether the U.S. Environmental Protection
Agency (EPA) has developed controls to manage the use of citizen science results
to meet the agency's mission.
Background
What is citizen science?
Citizen science is a form of open collaboration in which individuals or organizations
participate voluntarily in the scientific process in various ways, including conducting
scientific experiments, collecting and analyzing data, and solving problems.
Details on various citizen science projects are in boxes throughout this report
and in Appendix A.
Federal and Agency Guidance on Citizen Science
The EPA defines citizen science on the EPA's "What is Citizen Science" website
as "the involvement of the public in scientific research—whether community-
driven research or global investigations.
Citizen science mobilizes the public to
participate in the scientific process to address
problems."
On September 30, 2015, the White House
Office of Science and Technology Policy
(OSTP)1 noted the benefits of citizen science,
outlined principles agencies should apply,
recommended agency actions to build
capacity, and directed agencies to take two
specific steps to advance these methods:
1. Identify an agency coordinator for citizen science.
2. Catalog agency-specific citizen science projects on a governmentwide
online database and website—developed by the General Services
Administration (GSA)—to make these projects easier for the public to
1 Another White House memo from OSTP and the Office of Management and Budget from July 9, 2015,
encouraged agencies to use approaches such as citizen science to foster innovation in scientific discovery.
"Environmental protection
belongs to all of us, and
participating in environmental
science is one way that
members of the public can have
an impact. Citizen science
broadens environmental
protection by enabling people to
work together with government
and other institutions toward
shared goals."
- Blog post by former EPA
acting Deputy Administrator
18-P-0240
1
-------
discover, help improve collaboration within and across agencies, and
reveal opportunities for new projects.2
The OSTP memo encouraged—but did not require—federal agencies to take
actions in the areas of policy, resources and staffing, technologies and scientific
instmmentation, diversity of projects, and rigorous research and evaluation to
improve current practice. For example, the memo encouraged agencies to:
• Develop clear policies, procedures and guidance to encourage and aid agency
coordinators in developing and carrying out effective citizen science projects.
• Devote resources to evaluating the effectiveness of citizen science projects
in achieving agency objectives.
• Develop methods for validating data and results from projects.
On January 6, 2017, Congress enacted the Crowdsourcing and Citizen Science
Act as part of the American Innovation and Competitiveness Act, which gave
federal science agencies—including the EPA—the authority to use citizen science
to conduct projects designed to advance agency missions.3 Congress noted that
citizen science projects have unique benefits, including accelerating scientific
research, increasing cost effectiveness to maximize
the return on taxpayer dollars, addressing societal
needs, and connecting members of the public
directly to federal science agency missions and to
each other.
Within the EPA, citizen science is not one
centralized program. Rather, it includes projects
implemented separately by the EPA's program and
regional offices. The EPA's citizen science efforts
are facilitated by the Office of Research and
Development's (ORD's) Office of the Science
Advisor, which provides cross-agency leadership
on science policy development and
implementation. Within the Office of the Science
Advisor, a Chief Innovation Officer and a team of
three staff (along with fellows and student
contractors) support agency citizen science efforts,
among other areas. According to the Director of
the Office of the Science Advisor, in fiscal year
2017, this team had a budget of just over
$1.4 million for investments in innovation
projects, including citizen science.
2 See the GSA Federal Crowdsourcing and Citizen Science Toolkit.
3 The act addresses both citizen science and crowdsourcing, the latter of which is defined as a method to obtain
needed sendees, ideas or content by soliciting voluntary contributions from a group of individuals or organizations,
especially from an online community. While related/complementary, our review focused on citizen science.
Measuring Air Pollution
Mitigation Strategies at Schools
The EPA (Region 9 and ORD) and state
and local stakeholders engage teachers
and students in studying the effectiveness
of roadside vegetation barriers. The project
seeks to reduce exposure to vehicle
emissions at an Oakland, California,
elementary school that is adjacent to a
busy road. This is an active project that
started in the fall 2017 with $38,000.
EPA staff and local partners evaluate
the effectiveness of vegetation and
noise barriers near the Brookfield
Elementary School in Oakland,
California. (EPA photo)
18-P-0240
2
-------
The ORD's Chief Innovation Officer serves as the agency's coordinator for
citizen science and holds a monthly meeting with the EPA's citizen science
community of practice group. The Chief Innovation Officer also co-chairs—
with an Innovation Specialist from the U.S. Geological Survey—the federal
community of practice on citizen science. In addition, two EPA regional offices—
Regions 1 and 2—have appointed citizen science coordinators to facilitiate
communication and coordination of citizen science in their regions.4
The EPA has both public and internal websites on citizen science that provide
links to citizen science resources at the agency. These EPA sites also link to
resources on the federal citizen science website, managed by
the GSA, including the Federal Crowdsourcing and Citizen
Science Toolkit, and governmentwide databases on citizen
science projects.5
In 2015, the EPA tasked the National Advisory Council for
Environmental Policy and Technology (NACEPT)—an EPA
advisory board—to assess the agency's approach to citizen
science. NACEPT issued a report in December 2016,
Environmental Protection Belongs to the Public: A Vision for
Citizen Science at EPA, that made several recommendations,
including articulating and implementing a vision for citizen
science at the EPA, dedicating funding, and building technical capacity. The 2016
NACEPT report included the following spectrum depicting the variety of uses of
citizen science data:6
Figure 1: Spectrum of citizen science uses
"Citizen science is much more
than collecting data. It provides
a way to engage all parts of
society in gaining a deeper
understanding of human
environments, build an
informed population that can
advocate successfully for
environmental protection, and
more effectively protect human
health and the environment."
- NACEPT
Community
Engagement
Education
Condition
Indicator
Research
Management
Regulatory
Decisions
Regulatory
Standard
Setting
Enforcement
Source: OIG-generated image based on 2016 NACEPT report.
NACEPT issued its second report in April 2018, Information to Action—
Strengthening EPA Citizen Science Partnerships for Environmental Protection.
The EPA's fiscal years 2018-2022 Strategic Plan—dated February 12, 2018—
includes broad strategic objectives on public participation and robust science, and
mentions citizen science in Objective 3.5 ("Improve Efficiency and
4 The Region 1 citizen science coordinator position is a temporary detail.
5 The Federal Crowdsourcing and Citizen Science Toolkit helps federal employees use crowdsourcing and citizen
science in their work. It provides five basic process steps for planning, designing and carrying out a crowdsourcing
or citizen science project.
6 NACEPT describes "community engagement" as including awareness, partnership, development, stakeholder
engagement and public outreach. NACEPT describes "education" as including STEAM (science, technology,
engineering, art and math) literacy. ORD's Chief Innovation Officer added that he views education as one-
directional where community engagement goes beyond that in terms of participant involvement.
18-P-0240
3
-------
Effectiveness") by noting that the "EPA will develop a comprehensive data
management strategy that addresses the collection, management, and use of data
generated both internally and from external partners including ... citizen science."
Federal and Agency Guidance on Internal Controls
Office of Management and Budget (OMB) Circular A-123, Management's
Responsibility for Enterprise Risk Management and Internal Control (July 15,
2016), states that federal leaders and managers are responsible for establishing
goals and objectives around operating environments and managing both expected
and unexpected or unanticipated events. Additionally, managers are responsible for
implementing management practices that identify, assess, respond to and report on
risks. Risk management practices must be forward-looking and designed to help
leaders make better decisions, alleviate threats, and identify previously unknown
opportunities to improve the efficiency and effectiveness of government operations.
The circular requires agencies to implement enterprise risk management
coordinated with the process for strategic planning and internal controls. It also
requires that federal managers establish and maintain internal controls—based on
the U.S. Government Accountability Office's (GAO's) Standards for Internal
Control in the Federal Government—to achieve specific objectives, and that risk
management be considered when designing and assessing internal controls.
The GAO's Standards for Internal Control in the Federal Government defines
internal control as a process that provides reasonable assurance that the objectives
of an entity will be achieved. Internal control comprises the plans, methods,
policies and procedures used to fulfill the mission, strategic plan, goals and
objectives. A key factor in improving accountability is to implement an effective
internal control system that helps an entity adapt to shifting environments,
evolving demands, changing risks and new priorities. Management is responsible
for an effective internal control system by setting objectives, implementing
controls and evaluating the internal control system.
The GAO identified five components that must be effectively designed, implemented
and operating in an integrated manner for an effective internal control system:
1. Control Environment. The foundation for an internal control system. It
provides the structure to help an entity achieve its objectives.
2. Risk Assessment. Assesses the risks facing the entity as it seeks to achieve
its objectives. This assessment provides the basis for developing
appropriate risk responses.
3. Control Activities. The actions management establishes through policies
and procedures to achieve objectives and respond to risks.
4. Information and Communication. The quality information management
and personnel communicate and use to support the internal control system.
18-P-0240
4
-------
5. Monitoring. Activities management establishes and operates to assess the
quality of performance over time and promptly resolve the findings of
audits and other reviews.
EPA Order 1000.24 CHG 2, Management's Responsibility for Internal Control
(July 18, 2008), implements OMB Circular A-123 and outlines agency senior
managers' roles and responsibilities for developing, implementing and assessing
internal controls. The order requires all EPA organizations to establish and
maintain internal controls to achieve objectives, evaluate controls on an ongoing
basis, and take prompt action to correct any vulnerabilities identified.
Responsible Offices
Within the EPA, the Deputy Administrator (within the Office of the Administrator)
and the ORD's Office of the Science Advisor have the primary responsibility for
the issues discussed in this report.
Noteworthy Achievements
Prior to the start of our audit, the ORD's innovation team had begun to develop a
checklist of administrative and legal considerations for citizen science and
crowdsourcing projects at the EPA. Additionally, the team drafted an outreach
and communication tool that highlights citizen science projects at the EPA,
selected to provide a representative distribution across the agency. The team
intends to link the document to the citizen science intranet site to help EPA staff
understand the breadth of projects underway at the agency.
The EPA has taken the lead among federal agencies in addressing a barrier related
to the Paperwork Reduction Act by obtaining OMB approval for an Information
Collection Request (generic clearance for citizen science and crowdsourcing).
This clearance expedites the OMB approval process for the collection of
information. It is limited to collecting information for projects contributing to
EPA research, not regulation and enforcement activities.7
Scope and Methodology
We conducted this performance audit from October 2017 to July 2018 in
accordance with generally accepted government auditing standards. Those
standards require that we plan and perform our work to obtain sufficient,
appropriate evidence to provide a reasonable basis for our findings and
conclusions based on our audit objective. We believe that the evidence obtained
provides a reasonable basis for the findings and conclusions in this report based
on our audit objective.
7 This generic Information Collection Request was approved by the OMB for a 3-year period (April 2016-
April 2019). As of December 2017, the EPA has submitted, and the OMB has approved, six information collection
titles under this generic Information Collection Request, and three additional requests are pending approval.
18-P-0240
5
-------
We reviewed federal laws and guidance on citizen science as well as EPA
materials, including strategic plans; websites; and policies, procedures and
regulations on quality assurance. We reviewed the NACEPT reports and other
relevant external reviews. We also reviewed journal articles and attended
webinars and other presentations to obtain various perspectives. We interviewed
key ORD staff and managers responsible for innovation and citizen science, as
well as each national research program director and laboratory/center director on
the extent to which they used citizen science.
To obtain agencywide perspectives on citizen science, we:
• Queried Deputy Assistant Administrators in six headquarters program
offices8 on the ways they use citizen science and factors that impact use.
• Interviewed staff and managers in the offices of the Administrator, Air and
Radiation, Water, Environmental Information, and Enforcement and
Compliance Assurance; and within Regions 2 and 9, given their history
and experience with citizen science. We also interviewed a staff person
within the Office of General Counsel on federal criteria.
• Issued a survey to 253 targeted agency contacts, including the EPA's
citizen science community of practice and regional science liaisons. We
received 113 responses—a response rate of 44.66 percent. In developing
the survey, we received input from the ORD's innovation team and from
federal community of practice members.9 We also worked with the ORD's
innovation team to judgmentally target our survey to those within the
agency who would most likely have knowledge of and use citizen science.
We cite survey results in Chapter 2.
• Developed a draft inventory of 83 likely EPA citizen science projects,10
spanning from 2000 onward, from which we selected 11 projects to review
for evidence of controls to manage results. Sampled projects included one
each from seven program offices and, for geographic diversity, one each
from EPA Regions 2, 4, 7 and 9.11 For each project, we sought
information on the EPA's intended uses of results and any evidence of
controls in place to manage use. While we reviewed documentation for
evidence of controls (e.g., Quality Assurance Project Plans and peer
reviews), we did not review the scientific or technical quality of provided
8 The offices of Air and Radiation, Chemical Safety and Pollution Prevention, Enforcement and Compliance
Assurance, International and Tribal Affairs, Land and Emergency Management, and Water.
9 We received input from the Intelligence Advanced Research Projects Activity, National Oceanic and Atmospheric
Administration, National Science Foundation, and National Aeronautics and Space Administration.
10 This is not a complete universe of the EPA's citizen science efforts but, rather, reflects projects we identified
through the GSA Toolkit, our survey responses and other sources. We provided our draft inventory to the ORD.
11 Our review of sampled citizen science projects found some attributed to different offices than we thought initially.
Our sample of 11 projects ended up with more from regional offices (e.g., addition of projects in Regions 6 and 10)
than headquarters program offices.
18-P-0240
6
-------
materials. We also interviewed some project leads and, in one case, a
community organizer of a citizen science project. We highlight projects
throughout this report and in Appendix A.
• Separate from our sample, we conducted a site visit to Region 1—a region
active in citizen science—to meet with EPA staff and external officials on
four additional projects that include citizen science:
o EPA Preparing for Extreme Weather (Mattapoisett, Massachusetts),
o Mystic River Watershed Association Report Card Baseline
Monitoring Program and Annual Report Card,
o Casco Bay Estuary Partnership and Friends of Casco Bay.
o Cyanobacteria12 Monitoring Collaborative.
We also interviewed citizen science leads in other federal agencies to benchmark
best practices, including measures to address barriers to using citizen science.
Cyanobacteria Monitoring Collaborative
The Cyanobacteria Monitoring Collaborative—a project for
EPA Region 1 and ORD—is an active project started in
May 2015. It has three components to allow for participation
from the public (BloomWatch), trained citizen scientists
(CyanoScope) and environmental professionals
(Cyanomonitoring), as follows:
• BloomWatch is a crowdsourcing, citizen science smart
phone app tool the public can use to identify and report
potential cyanobacteria blooms, along with photo
documentation.
• The CyanoScope project uses citizen scientists to collect
and analyze samples with microscopy kits and then send
the images to a central database that records details on
location of the sample, date and time of collection.
Experts then update the database with the
identification of species in captured images. The initial
effort proposed developing the microscopy kits, training
volunteers, and piloting the program as an enhancement
to existing monitoring programs in New England and the
Great Lakes regions.
A CyanoScope field kit used by
citizen scientists. (EPA OIG photo)
• The Cyanomonitoring component engages trained citizen scientists and environmental
professionals in cyanobacteria monitoring using a hand-held field fluorometerto test water
samples.
12 Cyanobacteria are photosynthetic bacteria. Cyanobacteria and their associated toxins are of concern from a
human and ecological perspective. Increased nutrient loads to aquatic systems can lead to recurring algal blooms*
and wildlife and pet deaths have occurred from ingestion of these algal toxins.
18-P-0240
7
-------
Chapter 2
EPA Does Not Manage Citizen Science
from an Agencywide Perspective and Should
Complete Actions to Address Barriers
Although citizen science is carried out throughout EPA program and regional
offices, the EPA has not developed controls necessary to manage it from an
agencywide perspective. The agency has not established a clear vision and
objectives for using citizen science results from an agencywide perspective.
Consequently, the EPA cannot undertake a systematic effort to analyze the risks
and opportunities that citizen science presents. Both Congress and the White
House's OSTP encourage citizen science. EPA staff identified barriers to
effectively using citizen science results in the agency's work. Key barriers include
lack of a comprehensive vision and support/resources from senior management,
and lack of understanding and buy-in for citizen science. These barriers exist
because EPA leadership has not set a specific strategy for citizen science. Citizen
science is evolving and will take on a greater role as advancements in technology
provide greater access to the public. As the public's involvement grows, it will
place pressure on the EPA to acknowledge and address the environmental data
collected and provided to the agency.
EPA Uses Citizen Science Primarily for Community Engagement,
Research, Monitoring and Environmental Education
OSTP directed agencies to catalog agency-specific citizen science projects in a
governmentwide online database, which the EPA did. However, the EPA's
voluntary data entry into the federal database did not reflect all of the agency's
citizen science activities; rather, it only reflects those projects EPA staff sought to
highlight.13
Absent a complete inventory capturing the EPA's uses of citizen science, we
queried program and regional offices—via a survey instrument, interviews and a
project sample—on the extent to which they use citizen science results. Over
70 percent of survey respondents indicated that their programs use citizen science
to some extent, primarily for community engagement (78.5 percent), research
(68 percent), monitoring environmental conditions (63 percent), and
environmental education (61.5 percent).14 A further breakdown is in Figure 2.
13 The citizen science coordinator within the National Aeronautics and Space Administration described a similar
approach to how their agency catalogued its projects. We understand that GSA is currently in the process of working
with federal agencies' citizen science coordinators to update the catalog with more projects.
14 In response to a question asking whether their program uses citizen science, over 70 percent of respondents
(79 out of 112 individuals) answered "yes." In response to another question asking respondents to describe ways
citizen science is used in their respective programs, 65 respondents provided answers.
18-P-0240
8
-------
Figure 2: Survey responses on uses of citizen science
¦ Community Engagement
¦ Research
¦ Monitor Environmental Conditions
¦ Environmental Education
¦ Other
¦ Regulatory Decisions
¦ Enforcement
Source: OIG analysis of survey results. Percentages are out of a total of 65 responses to a
question asking respondents to describe ways their program uses citizen science.
All six Deputy Assistant Administrators that we queried indicated that their
program offices use citizen science for community engagement, environmental
education or research. For example:
• The Office of Enforcement and Compliance Assurance Deputy Assistant
Administrator said that the office uses citizen science data as a starting
point to assess the need for further enforcement monitoring/investigation.
• The Office of Water Deputy Assistant Administrator said that states use
citizen science data as part of water quality assessments, and some data is
shared via the agency's water quality exchange system.15
• The Office of Air and Radiation Deputy Assistant Administrator said,
"Citizen science offers a way to further evaluate new air quality sensor
technologies; to collect information in areas where EPA does not have
regulatory measurements; and to greatly help educate the public about
their local air quality."
Within the ORD: two of six National Research Program directors said they
embrace citizen science efforts, two said they use it minimally mostly to guide the
direction of their research, and two do not use it given the nature of their work.
15 The Office of Water said that it is building tools to integrate water data from a myriad of sources including citizen
science. The office is championing the Interoperable Watersheds Network project, which is focused on defining a
common set of data formats and standards, and testing and validating both the standards and new methods of sharing
data. When completed, the network is expected to expand data sharing and use, thereby streamlining surface water
quality assessment, restoration and protection activities at all levels of government.
18-P-0240
9
-------
Additionally, the ORD's National Center for
Environmental Assessment is monitoring citizen
science as an "emerging area" for its possible use
to monitor and/or observe ecology, and the ORD's
National Exposure Research Laboratory is
embracing the opportunity for using citizen science
in exposure science facilitated by advancements in
sensors and new technologies.
Interviews with program and regional office staff
indicated that whether the agency "uses" citizen
science varied depending on the definition of "use"
akin to the spectrum of uses noted in the 2016
NACEPT report described in Chapter 1. Our sample
of citizen science projects noted various uses that we
describe below and in project summaries in
Appendix A. We found—similar to what the Deputy
Assistant Administrators described—that our
sampled projects fell mostly on the side of
N ACEPT's spectrum that includes community
engagement, education and research. In general, we
did not find any examples where the EPA has used
citizen science data by itself to inform decision-making, although we did hear of
examples of states acting in response to citizen science information.
We identified anecdotal benefits to the EPA and its state partners from using
citizen science:
• In Region 1, we learned that many states rely on citizen science organizations
to collect data used in meeting the states' water monitoring and reporting
responsibilities. Region 1 staff said that limited state resources prevent the
states from conducting effective, long-term monitoring of water systems on
their own. Region 1 staff said states use citizens to supplement the data
collection needed, and universities/non-profits to analyze data.
• The Virginia Department of Environmental Quality described its return on
investment from volunteer water monitoring and noted that it receives
over $750,000 worth of data collected for its $200,000 investment in
citizen science (i.e., through state grants and staff support), for an over
275 percent return on investment of agency resources.
In addition, two projects that we sampled continue even though the initial EPA
funding supporting each project has lapsed:
• Gardenroots. Community members living near the Iron King Mine and
Humboldt Smelter Superfund sites in Dewey-Humboldt, Arizona, worked
Ironbound Citizen Science
Air Monitoring Collaboration
The ORD and EPA Region 2 partnered to
help the Ironbound community (in New
Jersey) to collect data on environmental
conditions. The EPA designed sensor units
and trained community members. This
resulted in an Air Sensor Toolbox, an online
resource with information on low-cost
technologies for measuring air quality. This
project was completed in 2015 through
funding of $100,000.
Citizen scientists
working with air
monitoring
equipment.
(EPA photo)
18-P-0240
10
-------
with a researcher to investigate the uptake of arsenic in commonly grown
vegetables, evaluate arsenic exposure and potential risk, and report results
in an effective and meaningful way. As a result of the study, community
members could make educated choices about the type of garden
vegetables they would eat and leveraged the results to encourage
government officials to act and be more stringent in their cleanup efforts.
An ORD grant of $15,000 helped fund this study, and though the study
ended in 2012 the program continued by expanding to three additional
counties with over 100 citizen scientists trained.
• IDAH20 Master Water Stewards. This EPA grant project, which launched
in 2010 and was funded through 2014, provided $77,000 for the training
of 50 volunteer water quality monitors to adopt a stream that they would
monitor regularly. Monitoring includes habitat, biological, chemical and
physical assessments. The project, which involved Region 10, is still
active and has now increased to 150 volunteer monitors. The final report
said: "Budget cuts have forced the elimination of Idaho governmental
efforts to conduct statewide water quality monitoring in streams, lakes and
rivers, with the exception of known problem areas. ... The IDAH20
program has expanded its network of volunteer monitoring to help fill the
gap in gathering water quality data."
Additionally, NACEPT noted that:
Citizen science can play a role in complementing EPA's ongoing
policy, regulatory and enforcement work through careful design
and open partnerships between external groups and EPA.
Ultimately, citizen science can improve the Agency's enforcement
processes by helping to identify issues proactively.
Below we describe control measures to implement—and barriers to address—to
move the EPA toward these opportunities.
EPA Has Not Developed Controls for Managing Use of Citizen Science,
but Does Have Controls for Quality and Use of Environmental Data
The EPA does not have the foundation
that would establish a control
environment for managing the agency's
use of citizen science, such as policies
and procedures or clear objectives for
how to use the tool in meeting the
agency's mission. An OSTP 2015 memo
(see sidebar) encourages agencies to
build citizen science capacity by acting
in such areas as policy.
"Develop clear agency-specific policies,
procedures, and guidance to encourage
and aid agency coordinators in
developing and carrying out effective
citizen science. ... The policies,
procedures, and guidance should
address common legal and process
steps, including data collection and
management. ..."
- OSTP memo
18-P-0240
11
-------
Objectives in the EPA's strategic plan signal the agency's intention to increase
public participation platforms, which could include citizen science. However, the
agency does not currently have a clear vision and objectives for using citizen
science to meet those strategic objectives. Survey respondents and interviewees
consistently noted that the EPA needs a vision, policies, procedures and/or
guidance to provide the structure for using citizen science in the agency's work
(see below and Appendix B for additional details). Moreover, EPA staff said a
high-level vision could clarify how the agency works with state/local partners on
citizen science as part of the cooperative federalism noted in the agency's strategic
plan. One regional staff member added: "EPA should make it clear to states and
local governments through consistent program policies, program guidance and
training, that citizen science efforts are supported by EPA leadership."
The EPA has controls that cover the quality and use of environmental data,
including citizen science data. For example, contract/grant awardees on citizen
science projects are required to adhere to specific quality standards16 identified in
required data quality and data management plans. These requirements directly
align with existing agency policies. However, these controls are not linked to any
objectives or risk analysis on the use of citizen science results.
Without a clear vision and objectives, the
EPA is unable to systematically assess the
opportunities and risks citizen science
presents for the agency. Per the GAO, a
prerequisite to an internal control risk
assessment is the identification of goals and
objectives. Once goals and objectives are
identified, an assessment of risks associated
with achieving those goals and objectives can be performed.17 The EPA, in
accordance with GAO standards, defines a risk assessment as the identification and
analysis of relevant risk associated with achieving the agency's mission.18
Absent a systematic assessment, the EPA has made some efforts to identify
opportunities and risks in using citizen science. For example, in 2015, the EPA
charged NACEPT with reviewing the agency's use of citizen science,19 and
NACEPT noted some barriers. In addition, the EPA's Chief Innovation Officer
has identified risks, such as the risk of receiving poor quality data and frustrating
16 ANSI/ASQC E4-1994 identifies specifications and guidelines for quality systems for environmental data
collection and enviromnental technology programs.
17 Under the risk assessment component, the GAO notes that management should (1) define objectives clearly to
enable the identification of risks and define risk tolerances; and (2) identify, analyze and respond to risks related to
achieving the defined objectives.
18 EPA Order 1000.24, CHG 2, Management's Responsibility for Internal Control (July 18, 2008), states that
program managers should identify internal and external risks that may prevent the organization from efficiently and
effectively meeting its objectives.
19 In its response to our draft report, the EPA said it is the only federal agency that has taken this step (i.e., of
seeking an advisory board review of the agency's use of citizen science).
"By prioritizing citizen science, the
Administrator has the opportunity
to increase the efficiency and
effectiveness of EPA programs
and empower stronger
partnerships and collaborations
for environmental protection."
- NACEPT
18-P-0240
12
-------
collection efforts by citizens due to limited communication on data quality
standards and requirements. The identification of these risks led, in part, to the
development of a Draft Handbook for Citizen Science Quality Assurance and
Documentation and establishment of a citizen science website to share guidance
and information on relevant policies and available resources.20 Development of
this handbook is a collaborative effort between the ORD, Office of Environmental
Information (OEI) and Region 1.
In the absence of EPA-wide direction on citizen science, the ORD's innovation
team has taken steps to build capacity and maintain the agency's expertise on
citizen science by, for example, communicating information about the potential
for using citizen science, as well as quality standards and requirements for data
quality. Additional information and communication efforts include the EPA's
citizen science websites, as well as the EPA's citizen science community of
practice meetings mentioned in Chapter 1. In addition, the ORD's innovation
team provided short training sessions on citizen science to five EPA regions so
far, and the ORD conducted three laboratory competitions that funded selected
citizen science projects during 2015-2017. However, the ORD's innovation team
noted that cuts to budget and staff could affect wider program implementation.
In our review of a sample of citizen science
projects, we asked about five control
measures that are required per agency
policies for any EPA-funded project where
environmental data is collected:
1. Quality Assurance Project Plan (QAPP)21 or equivalent documentation.
2. Quality Management Plan (QMP)22 or equivalent documentation.
3. Peer review.
4. Training on data collection requirements and/or data quality.
5. Data assessment.23
We found that projects contained these control measures relative to the project's
intended use and purpose for the data. The projects we sampled varied in the level
of citizen involvement in the collection and generation of data and the intended use
for the data. For some projects, the citizen component was limited to providing
feedback on the usability of monitoring equipment or the management of the
resulting data, while in other cases citizens collected and/or generated screening-
improving internal management
controls will involve multiple
offices, and EPA will need to move
from a project-level orientation to
an enterprise approach."
-ORD
211 EPA websites include those on citizen science generally as well as on collecting data and additional resources.
21 A QAPP defines and documents how environmental data collection activities are planned, implemented and
assessed during a project. EPA policy requires that all work performed by or on behalf of the EPA involving the
collection of enviromnental data be implemented in accordance with an agency-approved QAPP.
22 A QMP is a formal document or manual that describes the quality system in terms of the organizational structure,
functional responsibilities of management and staff, lines of authority, and required interfaces for those planning,
implementing and assessing all activities conducted.
23 Data assessment refers to any checks of the citizen science data performed by the EPA organization sponsoring
the project (i.e., program office or region).
18-P-0240
13
-------
level data.24 This type of project variability aligns with NACEPT's spectrum of
uses diagram noted in Chapter 1. We found examples where some measures were
implemented although not required. For example, the grantee of one project
developed a QAPP for its own use even though the data was not intended to be
used by the EPA (IDAH20 Master Water Stewards). In projects where the EPA
intended to use citizen-generated data, we saw evidence of the five control
measures (e.g., Smoke Sense). Although a QAPP is not always required,25 it can
provide validity to the data. The EPA's citizen science website and draft guidance
encourage groups to prepare a QAPP, even when not required, to aid in explaining
how the data was obtained and the level of quality assurance applied.
Barriers Exist to Using Citizen Science Results in EPA's Work
An agencywide internal control system should address barriers as part of risk
analysis to effectively use the results of citizen science in the EPA's work. Our
survey identified several barriers to using citizen science, as discussed in the
following blue box:
Survey Results on Barriers to Using Citizen Science
Two questions asked participants to identify barriers to their programs' use of citizen science. Question 8 asked
respondents to identify, in an open-ended response, the most important barrier to use. Question 9 asked them
to rank the extent to which eight specific barriers impact use; for this question, we provided a list of barriers
identified by the ORD's Chief Innovation Officer. The top six responses were similar for both questions.
Question 8
Question 9
1.
Data quality concerns
Resource limitations
2.
Resource limitations
Limited knowledge/experience with citizen science
3.
Limited knowledge/experience with citizen science
Data quality/data management concerns
4.
Lack of "buy-in"
Lack of guidance
5.
Technology issues
Lack of "buy-in"
6.
Lack of guidance
Technology issues
7.
Data management issues
Bureaucracy
8.
Quality assurance training needs
Legal/ethical issues
9.
Legal requirements
10.
Complexity of some EPA work
11.
Lack of coordination across the EPA
Source: OIG analysis.
24 Screening-level data refers to the use of citizen science data to provide context in identifying areas for future
research or possible noncompliance.
25 The implementation of the EPA Quality System is based on a graded approach where the quality systems vary for
different organizations and programs according to the specific objectives and needs of the organization. The need for
a QAPP may vary based on the purpose or intended use of the data (research program versus regulatory compliance
program). For example, a QAPP may not be required for data to be used in a screening level (though it could aid in
explaining how data was obtained), and may not be required when used for community educational purposes only.
18-P-0240
14
-------
In addition, we found a
relationship between
identified factors for success
(Question 10) and barriers to
use. For example,
respondents identified
"senior leadership and
management support" (i.e.,
"buy-in") as a factor
contributing to citizen
science success. This relates
to the identified barrier "lack
of'buy-in,' strategy, or
direction from
leadership/management." In
another example, "resources"
was identified as a barrier as well as a contributing factor to success.
Our interviews with EPA staff across several offices and regions, as well as with
NACEPT members, identified several barriers consistent with our survey results.
We grouped the five most common barriers under the following categories:
1. Strategic Communication and Support/Resources. "Top-down"
communication of the vision and direction for citizen science that ensures
alignment with the administration's priorities is provided, along with
support, coordination and resources for its use.
2. Understanding/Acceptance. A receptive culture for using the tool to
collect data.
3. Data Quality. The data meets quality standards for its intended use,
including educating citizen scientists on quality requirements, validating
that standards were met, and addressing misperceptions that may exist.
4. Data Management. Issues related to data storage and ownership, as well as
challenges associated with large amounts of real-time data, are resolved.
5. Technology. Issues are addressed, such as needed evaluations of available
sensor technologies and guidance for securing the data collected for use.
The first two barriers relate to our finding on the absence of a strategic vision and
objectives for agencywide use of citizen science. Per NACEPT members we
interviewed, the non-integrated approach to using citizen science at the EPA will
continue until agency leadership initiates a systematic approach that provides a
vision and strategy for a coordinated effort across the agency. Others we
interviewed within the EPA shared this sentiment. Appendix B provides
additional information and survey data on each category of barriers.
Image of Smoke Sense
mobile app from EPA's
Smoke Sense website.
Smoke Sense
This app—developed by the ORD and the EPA's
Office of Air and Radiation—collects data on health
effects of wildfire smoke on participants and what
they are willing to do to lower smoke exposure.
The EPA will use this data to determine the extent
to which exposure to wildfire smoke affects health
and productivity, and develop health-risk
communication strategies. This is an active 5-year
project launched in 2017 with $120,600.
18-P-0240
15
-------
EPA Needs to Complete Actions to Mitigate Barriers
Completing a risk analysis would help the agency realize the challenges and
opportunities citizen science presents. Absent an agencywide risk assessment for
using citizen science, the EPA has efforts underway that could mitigate some
barriers. For example:
~ Draft Handbook for Citizen Science Quality Assurance and Documentation.
This handbook will address some of the concerns for communicating data
quality standards and requirements externally to citizen scientists. The
handbook was developed by the EPA26 and was reviewed by tribes and state
officials and board members of the Environmental Research Institute of the
States, and it is currently undergoing a review with targeted subject matter
user groups familiar with citizen science. The EPA plans to pilot the
handbook with community groups later this year. The EPA has already
received some feedback from community groups on the handbook's
usefulness and need for providing guidance.
• Draft Communication Tools. The ORD has developed two draft
communication tools on citizen science for the EPA's internal use:
o The first highlights a representative distribution of collaborations
the agency has undertaken with various organizations to implement
citizen science projects (in air, water, etc.).
o The other lists potential administrative and legal requirements to
consider when developing citizen science projects.
We also found that Regions 1 and 2 have each
established a citizen science coordinator to
facilitate efforts and establish regional networks.
Region 1 has found it effective to have individuals
to help coordinate and link efforts in the region
with available resources at the ORD and within
other program offices. Others have indicated that
having regional citizen science coordinators in every region would be useful. For
example, staff in the Office of Water and Region 9 both said that regional
coordinators could work with their counterparts to facilitate communications and
access to available agency resources for citizen science.
"EPA needs to be more
corporate, systematic and
speak with one voice."
- ORD's Chief Innovation
Officer and agency citizen
science coordinator
26 According to the EPA, the OEI is leading the collaborative effort to develop the handbook with Region 1, along
with support from the ORD.
18-P-0240
16
-------
Citizen science is evolving as advancements in technology bring the public
greater access to low-cost tools and methods. Some in the EPA and NACEPT
predict that citizen science will test the legitimacy of the EPA's work as the
public places a greater demand on the EPA to acknowledge and address the
environmental data collected by citizen scientists. The EPA will face this growing
body of environmental data in the
future and has the potential to do so
now, in part by addressing barriers
(identified above), to manage its
future use of citizen science results.
The ORD's response to the 2017
NACEPT report states that the
engagement of the public in the work
of environmental protection is a
priority for the EPA. An informed and
engaged public can contribute to
effective policy making and citizen
science can help the EPA more
effectively connect with the people it serves. The EPA faces potential difficulties
stemming from the growing body of citizen science data, such as competing data
issues, quality assurance concerns, and conflicts with state/local enforcement
activities. With additional agencywide guidance and direction to enhance the
quality and management of citizen science results, the EPA will be better
equipped to use citizen science data in a manner that contributes to the EPA's
work.
Conclusion
Citizen science is a potentially significant method for obtaining environmental
data. Although the method itself is not new, it is predicted to take on a greater role
in the future, especially given rapid advancements in low-costs sensors and other
technologies for collecting and analyzing data. The EPA can be on the forefront
of this effort and faces potential problems if it does not address this growing body
of data in a systematic way. A clear vision or strategy for agencywide use of
citizen science will acknowledge the tool's growing use and help address the
agency's broad strategic objective on public participation. Further, clear guidance
and communication will enhance the quality and agencywide management of
results and help mitigate potential problems resulting from increased generation
of citizen science data.
"NACEPT members strongly believe that
a laissez faire approach by EPA will be
insufficient. EPA must advance a positive,
proactive agenda—to work in partnership
with communities and state, territorial and
tribal governments in ways that
strengthen citizen science infrastructure
and standardize citizen science methods.
... One of the great benefits that citizen
science offers EPA is the opportunity to
leverage expertise, networks and
resources of other parties."
-NACEPT (2018)
18-P-0240
17
-------
Recommendations
We recommend that the EPA Deputy Administrator:
1. Establish a strategic vision and objectives for managing the use of citizen
science that identifies:
a. Linkage to the agency's strategic goals,
b. Roles and responsibilities for implementation, and
c. Resources to maintain and build upon existing agency expertise.
2. Through appropriate EPA offices, direct completion of an assessment to
identify the data management requirements for using citizen science data
and an action plan for addressing those requirements, including those on
sharing and using data, data format/standards, and data testing/validation.
We recommended that the Assistant Administrator for Research and Development:
3. Finalize, in coordination with the Office of Environmental Information
and Region 1, the Draft Quality Assurance Handbook for Citizen Science,
and communicate to agency staff and citizen science groups the
availability and content of this handbook.
4. Build capacity for managing the use of citizen science, and expand
awareness of citizen science resources, by:
a. Finalizing the checklist on administrative and legal factors for
agency staff to consider when developing citizen science projects,
as well as identifying and developing any procedures needed to
ensure compliance with steps in the checklist;
b. Conducting training and/or marketing on the EPA's citizen science
intranet site for program and regional staff in developing projects;
and
c. Finalizing and distributing materials highlighting project successes
and how the EPA has used results of its investment in citizen science.
Agency Response and OIG Evaluation
The EPA concurred with our recommendations and provided acceptable
corrective actions and completion dates. In its response, the EPA said the Office
of the Administrator and ORD will collaborate with the EPA's programs and
regions on the implementation of our recommendations. To oversee
implementation, the EPA said it plans to convene an agencywide workgroup and
it expects the workgroup to complete final products by December 31, 2020.
Appendix C provides the agency's full response.
18-P-0240
18
-------
Status of Recommendations and
Potential Monetary Benefits
RECOMMENDATIONS
Rec.
No.
Page
No.
Subject
Status1
Action Official
Planned
Completion
Date
Potential
Monetary
Benefits
(in $000s)
1
18
Establish a strategic vision and objectives for managing the use
of citizen science that identifies:
a. Linkage to the agency's strategic goals,
b. Roles and responsibilities for implementation, and
c. Resources to maintain and build upon existing agency
expertise.
R
Deputy Administrator
12/31/20
2
18
Through appropriate EPA offices, direct completion of an
assessment to identify the data management requirements for
using citizen science data and an action plan for addressing
those requirements, including those on sharing and using data,
data format/standards, and data testing/validation.
R
Deputy Administrator
12/31/20
3
18
Finalize, in coordination with the Office of Environmental
Information and Region 1, the Draft Quality Assurance
Handbook for Citizen Science, and communicate to agency staff
and citizen science groups the availability and content of this
handbook.
R
Assistant Administrator
for Research and
Development
12/31/20
4
18
Build capacity for managing the use of citizen science, and
expand awareness of citizen science resources, by:
a. Finalizing the checklist on administrative and legal factors
for agency staff to consider when developing citizen
science projects, as well as identifying and developing
any procedures needed to ensure compliance with steps
in the checklist;
R
Assistant Administrator
for Research and
Development
12/31/20
b. Conducting training and/or marketing on the EPA's
citizen science intranet site for program and regional staff
in developing projects; and
c. Finalizing and distributing materials highlighting project
successes and how the EPA has used results of its
investment in citizen science.
1 C = Corrective action completed.
R = Recommendation resolved with corrective action pending.
U = Recommendation unresolved with resolution efforts in progress.
18-P-0240
19
-------
Appendix A
Information on Sampled Citizen Science Projects
Project Name and Summary: Smart City Air Challenge
• EPA conducted this contest for communities to
develop plans for deploying hundreds of air quality
sensors and making the data public in fall 2016.
EPA awarded two $40,000 prizes to Baltimore,
Maryland ("Greater Baltimore Open Air"), and
Lafayette, Louisiana ("Lafayette Engagement and
Research Network," or LEaRN), for their winning
submissions. After a year, the EPA will evaluate the
projects based on implementation and collaboration
and award up to $10,000 for each project.
Offices Involved: OEI, Office of Air and Radiation
Dates/Status: Active (October 2016 to Summer 2018)
Funding: $100,000
Sensor technology used by the LEaRN
program. (Photo courtesy of LEaRN)
Project Name and Summary: Local Environmental Observer Network
• The Local Environmental Observer Network is an organization of tribal individuals in Alaska and
Canada who share information about environmental events where they live, post observations on
public maps, and coordinate with technical experts to identify appropriate actions. The EPA
provided support via development of an app to facilitate recording of these visual observations.
Offices Involved: ORD, EPA Region 10
Dates/Status: App was launched in 2012 and the project is still active
Funding: Unable to identify
Project Name and Summary: Gardenroots
• In partnership with community members near a Superfund site in Arizona, Gardenroots aims to:
evaluate environmental quality and the potential exposure to contaminants of concern near active
or legacy resource extraction and hazardous waste sites, successfully communicate the study
results to all participating individuals and families, and disseminate the results broadly to
appropriately influence community prevention practices and environmental decision making.
Offices Involved: EPA Region 9, ORD
Dates/Status: Completed (2008-2012)
Funding: $15,000
Project Name and Summary: Region 10 Making a Visible Difference in
North/Northeast Portland: Engaging Communities Using Citizen Science
Assess and Address Children's Health from Transit and Air Pollution
• The objective of this project was to combine data from various citizen
science tools with local knowledge to understand mobile air pollution
concerns at bus stops used by students in north/northeast Portland,
Oregon, and to design a workshop model to integrate data sources
promoting problem solving of air pollution issues.
Offices Involved: ORD, EPA Region 10
Dates/Status: Completed (2015-2016)
Funding: $100,000
A hand-held air monitor
that measures
particulate matter in the
air. (EPA photo)
18-P-0240
20
-------
Project Name and Summary: Amigos Bravos
• Non-profit group Amigos Bravos received two separate grants under the Urban Waters program
to perform water quality testing by community volunteers and students in various New Mexico
water bodies. One of the projects was completed and the other remains active.
Offices Involved: Office of Water, EPA Region 6
Dates/Status: Completed (July 2014 to March 2018)
Funding: $115,326
Project Name and Summary: Ironbound Citizen Science Air Monitoring Collaboration
• The EPA partnered on a toolbox to enable communities like Ironbound (in Newark, New Jersey)
to collect their own environmental data and increase their ability to understand local
environmental conditions. The EPA designed and fabricated the sensor units and trained
community members in their use. This project resulted in the development of an Air Sensor
Toolbox—an online resource that provides information and guidance on new and low-cost
compact technologies used for measuring air quality.
Offices Involved: ORD, EPA Region 2
Dates/Status: Completed (2013-2015)
Funding: $100,000
Project Name and Summary: Low-Cost Water Sensors for
Real-Time Continuous Water Quality Monitoring
• This project funds activities for the equipment hardware and
calibration standards and maintenance for low-cost water
quality sensors. The EPA collaborated with the Georgia
Adopt-A-Stream program to host a workshop for watershed
monitoring groups in open source water quality sensor design,
programming and continuous monitoring. Participating groups
build and deploy sensors with the intention to provide
feedback to the EPA on the feasibility of incorporating this
technology and monitoring into citizen-based monitoring
programs. The EPA plans to conduct an evaluation of the
low-cost sensor and develop a low-cost quality sensor toolbox
for water quality managers and citizen scientists.
Offices Involved: EPA Region 4, ORD, Office of Water
Dates/Status: Active (August 2017 to December 2018)
Funding: $27,000
Project Name and Summary: Kansas City Transportation and Local-Scale Air Quality Study
(KC-TRAQS)
• EPA launched the Kansas City Transportation and Local-Scale Air Quality Study (KC-TRAQS) to
learn more about local community air quality in three neighborhoods that have multiple air
pollution sources from highways, railways and industry. A citizen science project is part of the
study and will involve area residents and students in air measurement activities by using
AirMappers. These lunchbox-size monitors developed by the EPA enable residents and students
to collect local air quality data by carrying or attaching the devices to a bicycle while walking or
biking around the study area.
Offices Involved: EPA Region 7, ORD
Dates/Status: Active (Fall 2017 to Fall 2018)
Funding: $150,000
A prototype open source sensor
system. (EPA photo)
18-P-0240
21
-------
Project Name and Summary: Building Capacity to Measure Air Pollution Mitigation Strategies at
Schools (California)
• The EPA, in collaboration with the Bay Area Air Quality Management District and the California
Air Resources Board, will engage teachers and students in studying the effectiveness of roadside
vegetation barriers. The project seeks to reduce exposure to vehicle emissions at an Oakland
elementary school that is adjacent to a busy road.
Offices Involved: EPA Region 9, ORD
Dates/Status: Active (started in the Fall 2017)
Funding: $38,000
Project Name and Summary: (Region 1 Project) EPA Preparing for Extreme Weather
(Mattapoisett, Massachusetts)
• This project investigated how sea level rise and disruptive extreme weather events may affect the
drinking water supply and critical infrastructure in the town of Mattapoisett. Modeling data was
collected and analyzed by the EPA. The community was engaged and contributed to this project
in the form of field investigation, historical storm surge, record retrieval, and impact to citizens in
the project area.
Offices Involved: ORD, EPA Region 1
Dates/Status: Completed (January 2015 to September 2016)
Funding: $50,000
Project Name and Summary: (Region 1 Project) Mystic River Watershed Association Baseline
Monitoring Program and Annual Report Card (Massachusetts)
• The Mystic River Watershed is a collection of rivers, streams, lakes and ponds that drain an area
of approximately 76 square miles and 21 municipalities north of Boston, Massachusetts. In
collaboration with the EPA, the Mystic River Watershed Association assembles and publishes
data in an annual water quality report card on the Mystic River watershed. Association volunteers
collect monthly baseline water quality samples, which are analyzed by the Massachusetts Water
Resources Authority or a private lab for bacteria. The EPA uses the authority's baseline bacteria
data, as well as additional bacteria data collected by the authority at other watershed sites, to
determine a set of 14 report card grades for the Mystic River and its tributaries each year.
Office Involved: EPA Region 1
Dates/Status: Active
Funding: None at this time
Project Name and Summary: (Region 1 Project) Casco Bay Estuary Partnership and Friends of
Casco Bay (Maine)
• A collaboration of agencies, organizations and individuals are working on behalf of the Casco
Bay. For 25 years, the Casco Bay Estuary Partnership has funded the Friends of Casco Bay
volunteer water quality monitoring program. Results from the Friends of Casco Bay data are used
for the State of the Bay reports and to inform the Casco Bay Plan. The Casco Bay Estuary—
a waterbody where rivers and the sea converge—encompasses 14 coastal communities,
including two of Maine's largest cities, Portland and South Portland.
Offices Involved: EPA Region 1, and Office of Water's National Estuary Program
Dates/Status: Active (1992 to present)
Funding: The Casco Bay Estuary Partnership receives about $600,000 annually and the partnership
provides Friends of Casco Bay with sub-awards.
18-P-0240
22
-------
Appendix B
Details on Barriers to Using Citizen Science Results
The following categories describe the five barriers cited most often in our interviews and survey.
Strategic Communication and Support Resources
The ORD's Chief Innovation Officer identified the lack of a strategic agencywide approach as a
barrier to using citizen science—primarily as a barrier to developing opportunities for using the
tool rather than as a barrier preventing its use. Survey respondents also identified barriers having
to do with strategic communication, guidance and lack of resources. In an open-ended question
on barriers:
• 15 percent of 73 respondents identified the lack of clear, top-level messaging and support
(i.e., on the quality and use of citizen science data, prioritization of efforts, addressing
data management issues, and ensuring alignment with EPA administration priorities).
• 10 percent of 73 respondents identified the lack of clear EPA guidance for using citizen
science data (i.e., protocols for when and how to use/not use the data and established
research frameworks that demonstrate the added value of data).
Many survey respondents cited "resource limitations" as a barrier (33 percent of 73 respondents).
Survey respondents also ranked resources as a factor greatly affecting their offices' use of citizen
science (Figure B-l).
Figure B-1: Extent to which resources affect citizen science use
o
1-Not at All 2 3 4 5 - Great
Extent
Source: OIG analysis of survey responses.
18-P-0240
23
-------
One Acting ORD lab director noted, "citizen science has a very strong future if we can get our
policies [established] ... so everyone knows what the ground rules are."
Understanding Acceptance
NACEPT members we met with identified as a barrier the general acceptance of citizen science
as a data collection tool. They noted the need for building a "cultural norm" that recognizes what
citizen science can do for the EPA, integrating it into the EPA's "toolbox" for data collection,
and valuing its utility for applications outside of regulatory and enforcement uses. The ORD's
Chief Innovation Officer also identified as a barrier promoting a cultural understanding and
"buy-in" for citizen science across the EPA. In addition, in response to a question on barriers,
survey respondents pointed to the lack of knowledge and/or acceptance of the EPA's potential
uses of citizen science data (18 percent of 73 respondents). We also asked survey respondents to
rank the extent to which experience with or understanding of citizen science (e.g., lack of trust in
citizen science data quality) affected their programs' use of citizen science, shown in Figure B-2.
Figure B-2: Extent to which experience or understanding affects
citizen science use
c
QJ
"D
C
o
Q.
tn
QJ
cd
c
QJ
U
L_
QJ
O-
25
20
15
10
Don't 1 - Not at
Know All
5 - Great
Extent
Source: OIG analysis of survey responses.
Data Quality
Strong data quality procedures promote confidence in the reliability of EPA reporting. The EPA
offices we surveyed or met with identified uncertainty over data quality as a barrier to using
citizen science. Similarly, data quality was the most common barrier identified by survey
respondents in an open-ended question (44 percent of 73 respondents). The ORD's Chief
Innovation Officer also identified various challenges associated with data quality, including
insufficient awareness of data requirements. NACEPT members also identified the need for
ensuring that data quality standards are communicated by the EPA and understood by those
18-P-0240
24
-------
gathering the data. Both the 2016 NACEPT report and our interviews with NACEPT members
emphasized that building partnerships with leaders in the citizen science community will help
"champion" the EPA's citizen science effort by providing training to those gathering the data and
helping communities meet the EPA's data requirements and needs. Community groups we met
with in EPA Region 1 also stressed the importance of complying with data quality standards on
citizen science projects.
Data Management
Staff in several EPA offices raised concerns with the perceived inability of the agency's
information technology systems to handle large data streams as well as addressing ownership and
storage issues for citizen science. The ORD's Innovation Team said that, currently, the OEI does
not have the capacity or resources to handle challenges posed by "big data."27 OEI staff said that
the problems stem from the volume, variety and velocity of the data; whether data formats are
inter-operable; and whether the frequency of measurements are comparable for analysis purposes,
among other challenges. Survey respondents also identified as a barrier data management issues
related to the storage, ownership and access to "big data." Figure B-3 captures survey responses
on the extent to which data and information management issues (e.g., cyberinfrastructure, data
platforms, data ownership) affect their programs' use of citizen science.
Figure B-3: Extent to which data and information management issues
affect use of citizen science
25
l.llll
0
Don't 1 - Not at 2 3 4 5 - Great
Know All Extent
Source: OIG analysis of survey responses.
27 "Big data" is an accumulation of data that is too large and complex for processing by traditional database
management tools. Challenges include analysis, capture, data curation, search, sharing, storage, transfer,
visualization querying and information privacy.
18-P-0240
25
-------
While the EPA Strategic Plan for fiscal years 2018-2022 includes a statement on the proposed
development of a "comprehensive data management strategy that addresses the collection,
management, and use of data generated both internally and from external partners including ...
citizen science," we obtained some inconsistent messages regarding this strategy:
• The EPA's Office of the Chief Financial Officer informed us that the OEI is the lead for
developing this comprehensive data management strategy and is in the early stages of
developing a framework; however, the Office of the Chief Financial Officer said citizen
science data would not be a part of this comprehensive strategy.
• We were informed that the OEI's Chief Data Scientist was not aware of this
comprehensive data management strategy and has not been asked to provide input,
although the OEI does have some efforts underway that could support this initiative.
• Citizen science leads in the ORD and Region 1 were not aware of this strategy and said
they were not consulted about any potential citizen science elements in the planned
comprehensive data management strategy.
Technology
Another barrier is making technology accessible and understood. Per NACEPT members, some
citizen scientists lack the technical understanding and capability for selecting the correct
instrument for data collection that will validate data. In addition, the availability in the market of
less expensive sensors and rapid advancements in technology present the EPA with the challenge
of "keeping up." The ORD's Chief Innovation Officer agreed that issues need to be addressed
related to information and sensor technology for collecting and securing the data for appropriate
use. Survey respondents (14 percent of 73 respondents) also identified sensor technology/tool
issues as a barrier (ranging from monitoring available technology, reliability for producing
quality data, and accountability for the equipment; to the appropriateness of sensor
measurements for use and how the measurement is represented in terms of environmental and
health risks).
18-P-0240
26
-------
Appendix C
Agency Response to Draft Report
UNITED STATES EtWtRONMGNTAL PROTECTION AGENCY
/¦ i U), 301S
MEMORANDUM
SUBJECT: Respooae to Office of ioipeetar General Dai Report No. OPE-FY11-0002 "EPA
Needs a Vision and States? ft* Citizen Science that Aligns wife. Its Stialegis
Objectives m Public Pi
lie U.S. Environmental Protection Agency welcomes tbe opportunity to review and
€CMfw»€llf Of* um UiIlCC 01 usSpQCiOi \lmmfml 1 Wul rcpOfl Utlw trA A v 4$1011 H® au»l«gy
for Citizen Science that Altps with Its Strategic Objectives on Public Participation''' (Pwject No,
OPE-FYI8-0002). The pomess used to collect infonroatkm for fills report was very thorough,
inciudttti* exten«i\ e Mien iews. surveys aid more iiMfeBili review ©ft subset of EPA Hfoiects, 'Tie
irai report illustrates the diversity of citizen sctoicc-wisted wxfc at the EPA ami show* how
citizen science touches all EPA progwms and wgiotw. Additionally, Ac site visit to Region 1
clearly provided a face-ui-tacc opportunity to understand tow citizen science workson the poind
and the nature of EPA partnerships with stales and other orgamTatioits,
The draft *f»ft reflects thtf §»wl> ami wanees of cilia® science work 4t tin* KPA .-wd the
OKI's observations provide important m««bn We appreciate tie fflO'i confirmation lit! tie
1 p \ Hjis token valuable sitp® in build capacity, invest is awl maintain agency expertise in cMm
science and concur with the recommendations. Howew, ft is impanel to tckaowlrfy tie strides
that the EPA has already taken toward developing • via*,, strategy and minimis tor awaagtog
cM/eii mJciut The AmI report identifies sevcril examples of EPA building capacity mi making
ftrogmw. iffwart develepiiig a virion, stategy md controls for Hie mm cf citijcn ww». but Qmm
exampU v \h n* hi^hlijjhtfd and »oi included as steps toward iaipieaMiiafioa of the
fCC®nioierfslipii> |« c\!».* EPA is recognized as a tender actow f ibcbk. that
action to iaipow current practice. In adtfilion, tie EPA tasked the National Adviaocy Couticll for
IO:
FROM:. Aaiww R. Wteeier
Aciint' Admrnistrator
ivcviii tnnstCQBcn, ass
Office of Audit aid Evaluation, Office of Inspector General
Jennifer Orme-Zv
Office of Research and
Science
18-P-0240
27
-------
Eavimaiimlil Policy and Technology to assess lbs ap»cy's approach to citizen science in 201 S;
the H'A is ifee util> federal agency that has taken this step. An important corax-m to note is tfte
titieof the dfoit report, "U'A Needs * Vmm ami Stiategy for Citizen Scwnu: that Aligns will lis
Strategic Oijeelives on Public Participation.'' dews not acknowledge the KPA's ellorfe toward a
citizen tcleoee vision ami stiate$Y m (tare- I betefbw* As EPA sujjjicsis the CJIO consider inserting
the word ^Comprehensive" io front of "Vision and Mrnu^tY* u> acknowledge recent projjfess.
Following the OKI's finding thai ciian science should be mow formally
dgeiicfwlcte, ihc Office of the Administrator and the Office of Research and Devdfopnient will
orfJafeoKtitf with the EPA's programs ad regions on the implementation ©f these
Kscommaidations. Implementation effort* will need to be closely coosdinaied with the EPA's
piograms, regions and state and tribal partners. as cit:/en science data and sntorimtMafi will
ultimately info® future work, acwss all environmental protection pp«i. To overset the
01 133© fc* A WHI1 COfIVC£M» & wH
workgroup tP6pioi:irf.fii|||. to the ^cteooe and Fficlfflolojjy f%3telic»^ expect tJUfttidiB wofk^rot^i
products b\ Dcu-mber 31. 201 *), and final rnxluus b) IVccir.hcr 51, 202©,
Esin^^^iisfd^ below uic A & to the OIG s ^ci»ific KXMMSi8ifijiiJ!iMl§©ci5t
krcommtflrfiliofl I: "Eitabliih a iliralfgtc vision and objective* for mamagiog the use itf
citizen science that idotifits: a. Linkage to the sgtacy's strategic b>. Roles and
responsibilities for implementation, and «, Resources t» ¦alnlaln and build npwa
Is Jim agency concurs with tiiis nxommetKlaiton aac piopoiCi lac leua'^Kig
Corrective Action 1? the l-PA will estabh«;h an a^en.-N wide work group to establish a
more formal stratepc vision and siltjectlves for managtag the use of citizen science,
including policies, procedure and clear obtcctives for how to collect, manage and use
cituen science to support the agency's mission. ORB will assist the agency as lie lead
to link the strstegic viilon and ob|«livei to the ^g»mcv*t strategic online u^Jes a»4
maff to ffmntatn and build iipwi existing 9gtm\ enprniise The agency*s anlegic vision
md ol^ecttves will h: guiici If the findings and memummiatims pwvi
-------
12: I he agency coocurs with this recotwivndJtton mid piopses lie follow®!
COtTCCtlVC llCtlllll 3fw yiffipIlSliJfl
Corrective Actios 2: lit I PA will complete an assessnwiu and icfioti p:.-w m identify
aiid address daw fnar.iigcroeni requiscaMOts for ®iti?cn svicncc. Devt^prat irivcs fine? piocr ixitrniii
»Ilal»iai«: and will fdy on eoisiillaliciii will experts »n data manager!>eni and citizen
scioct.
¦31,2020
IBWltilfc
tyl^.£%#9§i%vK X *|*j* t$4h& jS csp ci pj» Mi*Mt tit ft m j(t§kA/% ifa jf "jFyrfwiysE \*/*f a>|4 . t . p*
^ Wl w-" J* *km? ^ »2r <3 ifit »Vp jfsfc- »r J&MW&WMWw9^fJWir*k- **&>? 'w'-M w<*3N6r'w^iFw Cr«6£^ w sS^Stln-
Respmise 3: The agcucf CQficiirs Willi iiis newaiiiieBittioii Btid proposes the
SCttOn 8fld C0IIt|Ji©tMlil ifclfes
C«rtf«ttve Action 3; OR P ami the Office of Euviwiwiettlil Infboaalion will jointly
finrffyr the Dnfi Quality Assuror** Mmmi:bmk jot Citixtn Sctmm becrnw OU h»ts the
wspeisiMlitf for the •genet's OtuJity System — including ifufffwf of xmtiots&t Quality
M |..^|.^.|.- , 1- hj-Y -I, —A -fe I-Y] tf^-..."X. j. -Lr ¦•¦• <¦", ? |hc f T :i'. 'f.'UJ. ' x:'l. 1': 1' r ^ 1JT :il a?'. .i'V .51. v*-j- 1"- f ^ ¦' -"t. ¦»
A&ijyr»sC0 ^ «iiu UKi/ ikks ©@ itspoiiiiDiiit^ ior ouiiuiiig ©0 a£€iK>y s c«ip*4wiiy
f« om*gii| 'tie toe of ciiiiei #§I«ce, Alter isiiiig tie lepon, tie EM will iaplenieat
an cstornal a«l umtmi mtstmeh md e«a»i*iicat»ns plan m help EPA. xiaies aiiJ. mh*i%
and citiisi *i«tec p«ip» to strengthen quality mmmxx, pacicci.
; December 31,2020
RecoawndidM 4: BuiW fapscity f«»r Managing fii« use of dtken science, and
i as
idtolifyiiig and doelopiug *o> pro<«dure> needed to ensure compliance wi«h steps in the
far program and rigiotml stall in dtvcloptog projccis; and c. Fiaaliiing and distrifoating
4: I he agency toasws- with, this rectti.wci^lioii and purposes the fellowmg
CwwHw 4: ORD will consult will lie Office ui General Counsel aacf other
rdbvsnt EPA prog*®!! «ii rep©i* to finiliie ike ctocklisl ©n Mnitiisttsllw atrf It^l
18-P-0240
29
-------
.fecials for agency stiff to eoside? whett developing citizen science pwfc^is, ORB will
conduct training and iniileetiiig for program and itgponal staff, fIa*lj», ORD will have an
Planned C"k>tioii Pate: jDcwmN'* 'b
If you have any questions tegmimg, tills tesponse, please contact lay lenfwtlo. Chief
huwvatkHt Officer* Office of tfie Science Advisor, at Beajfof^Jay^epflugov.
cc: Science and fmbuoh^ Policy Councft
18-P-0240
30
-------
ills!
1II
j«l
ki
Hi
f
I
-------
Appendix D
Distribution
The Administrator
Deputy Administrator
Special Advisor, Office of the Administrator
Chief of Staff
Assistant Administrator for Research and Development and EPA Science Advisor
Agency Follow-Up Official (the CFO)
Agency Follow-Up Coordinator
General Counsel
Associate Administrator for Congressional and Intergovernmental Relations
Associate Administrator for Public Affairs
Director, Office of Continuous Improvement, Office of the Administrator
Principal Deputy Assistant Administrator for Science, Office of Research and Development
Associate Director for Science, Office of Research and Development
Deputy Assistant Administrator for Research and Development, Office of Research and
Development
Director, Office of the Science Advisor, Office of Research and Development
Audit Follow-Up Coordinator, Office of the Administrator
Audit Follow-Up Coordinator, Office of Research and Development
Audit Follow-Up Coordinators, Regions 1-10
18-P-0240
32
------- |