Handbook for Citizen Science Quality Assurance and Documentation - Version 1
Handbook for Citizen Science
Quality Assurance and Documentation
November 2018
Disclaimer: EPA is distributing this information solely as a public service. The inclusion of companies and
their products in this document does not constitute or imply endorsement or recommendation by the
EPA. EPA retains the sole discretion as to what extent it will use data or information produced or
resulting from use of this document.
This document does not define, or otherwise limit, the purpose to which citizen scientists may seek to
apply their data or information.
For more information, please contact us at:
https://www.epa.gov/citizen-science/forms/contact-us-about-citizen-science
https://www.epa.gov/qualitv/forms/contact-us-about-managing-quality-environmental-information
Page 1 of 32

-------
Handbook for Citizen Science Quality Assurance and Documentation - Version 1
Sponsors
Mike Flynn
Steven Fine
Deb Szaro
Jay Benforado
Contributors
Linda D. Adams
Daniel Bator
Emily Bender
Kristen Benedict
Nora Conlon
Holly Ferguson
Rachael Graham
Michelle Henderson
Bryan Hogan
Vincia Holloman
Matthew Liebman
Linda Mauel
Jenia McBrian
Ethan McMahon
Alison Parker
Liz Pucci
Patricia Sheridan
John Smaldone
Joanne Virgille
John Warren
Former EPA Acting Deputy Administrator (retired)
Former Principal Deputy Assistant Administrator, EPA Office of Environmental Information
Deputy Regional Administrator, EPA Region 1
Chief Innovation Officer, EPA Office of Research and Development Innovation Team
EPA Office of Research and Development, National Health and Environmental Effects
Research Laboratory
Association of Schools and Programs of Public Health (ASPPH) Environmental
Health Fellow, hosted by the EPA Office of Research and Development
Office of the Regional Administrator, EPA Region 1
Office of Air and Radiation, EPA Office of Air Quality Planning and Standards
Office of Environmental Measurement and Evaluation, EPA Region 1
Office of Research and Development, EPA National Health and Environmental Effects
Research Laboratory
Division of Environmental Science and Assessment, EPA Region 2
Office of Research and Development, EPA National Exposure Research Laboratory
Office of Environmental Measurement and Evaluation, EPA Region 1
Office of Environmental Information, EPA Office of Enterprise Information Programs
Office of Ecosystem Protection, EPA Region 1
Division of Environmental Science and Assessment, EPA Region 2
Office of Air and Radiation, EPA Office of Air Quality Planning and Standards
Office of Environmental Information, EPA Office of Digital Services and Technology
Architecture
Oak Ridge Institute for Science and Education (ORISE) Fellow, hosted by the EPA Office of
Research and Development
Office of Administration and Resource Management, EPA Region 1
Division of Environmental Science and Assessment, EPA Region 2
Office of Environmental Measurement and Evaluation, EPA Region 1
Office of Environmental Information, EPA Office of Customer Advocacy, Policy, and
Portfolio Management
EPA Office of Environmental Information (retired)
Page 2 of 32

-------
Handbook for Citizen Science Quality Assurance and Documentation - Version 1
Note to Reader	4
Introduction	5
How to Use This Document	6
Instructions for Citizen Science Quality Assurance and Documentation Templates 	14
Corresponding Template #1: Title and Preparer Page 	15
Corresponding Template #2: Table of Contents	15
Corresponding Template #3: Problem Definition, Background and Project Description	15
A.	Problem Definition 	15
B.	Background 	15
C.	Project Description	15
Corresponding Template #4: Data Quality Objectives and Data Quality Indicators 	16
A.	Data Quality Objectives 	16
B.	Data Quality Indicators	17
Corresponding Template #5: Project Schedule 	18
Corresponding Template #6: Training and Specialized Experience	18
Corresponding Template #7: Documents and Records	18
Corresponding Template #8: Existing Data and Data from Other Sources 	18
Corresponding Template #9: Sampling Design and Data Collection Methods	19
A.	Sampling Design	19
B.	Data Collection Methods 	21
Corresponding Template #10: Sample Handling and Custody	21
Corresponding Template #11: Equipment List, Instrument Maintenance, Testing, Inspection and Calibration .... 22
Corresponding Template #12: Analytical Methods	22
Corresponding Template #13: Field and Analytical Laboratory Quality Control Summary	22
Corresponding Template #14: Data Management	22
Corresponding Template #15: Reporting, Oversight and Assessments 	23
Corresponding Template #16: Data Review and Usability	23
Corresponding Template #17: Project Organization Chart	25
Corresponding Template #18: Project Organization 	25
Corresponding Template #19: Project Distribution List 	25
Definitions 	26
References	31
Page 3 of 32

-------
Handbook for Citizen Science Quality Assurance and Documentation - Version 1
Note to Reader
Dear Citizen Scientist,
Thank you for picking up this document. You are probably curious about the natural world or have concerns
about threats to your health (such as air pollution) or to the environment (such as algal blooms in a nearby lake).
You are part of a growing contingent of citizen scientists collecting data and reporting it to the public or to
government agencies. With new technologies, it is easier than ever to collect, analyze and report environmental
data. New sensors and smart phones make scientific research or environmental monitoring fun and interesting.
Sometimes, however, the quality and utility of the data you generate may be questioned. To be more confident
in your results, you should first ask a few questions such as: Are your sampling locations representative of your
study area? Do you think another volunteer can replicate how you collected the data?
In applying the concepts presented in this handbook, you are taking an important step in planning a successful
project. A Quality Assurance Project Plan, or QAPP, is a tool that scientists in government agencies, academia,
research organizations and others have used for decades. Although a QAPP is an imposing acronym, don't panic!
Most of the information in this handbook is common sense. Some of the quality assurance activities included
here you may already be doing and not all the information in this document may be applicable to your specific
project. We hope, however, that in documenting the quality assurance activities for your environmental
projects, you will be more confident in the results as you present them at a public meeting, in an annual report,
or in a meeting with government agencies.
Good luck!
Page 4 of 32

-------
Handbook for Citizen Science Quality Assurance and Documentation - Version 1
Introduction
With the advent of new technologies for environmental monitoring and tools for sharing information, citizens
are more and more engaged in collecting environmental data, and many environmental agencies are using these
data. A major challenge, however, is that data users, such as federal, state, tribal and local agencies, are
sometimes skeptical about the quality of the data collected by citizen science organizations. One of the keys to
breaking down this barrier is a Quality Assurance Project Plan (QAPP).
What is a Quality Assurance Project Plan? A QAPP is a document that explains how organizations ensure, using
quality assurance and quality control activities, that the data they collect can be used for its intended purpose.
By writing and applying a QAPP, an organization builds data quality procedures into the project from the
beginning and will be more confident that the data will meet the specific needs of the project. Importantly, the
individuals interested in the project, or the agencies that make decisions based on the data and information
from the project, will have a better understanding of the quality of the underlying data.
Audience: Citizen science has been defined as follows: "when the public participates voluntarily in the scientific
process, addressing real-world problems in ways that may include formulating research questions, conducting
scientific experiments, collecting and analyzing data, and interpreting results" (NACEPT, 2016). This Handbook is
targeted to organizations that are starting or growing a citizen science project, and where transparency in the
scientific methods for collecting the data are central to the outcome of the project. Examples of citizen science
include: organizations that collect water quality data to report to a state agency; programs that collect air sensor
data to post online; or groups that document the presence of invasive species. There is detailed information in
this Handbook and the companion Templates that more established organizations will find useful.
Purpose: The Handbook has two companion documents, Templates for Citizen Science Quality Assurance and
Documentation" and "Compendium of Examples." These documents provide tools and procedures to help
citizen science organizations properly document the quality of data. In doing so, this Handbook is meant to
convey common expectations for quality assurance and documentation; and best management practices to level
the playing field for organizations that train and use volunteers in the collection of environmental data.
Background: The Handbook and the companion documents are derived from an earlier EPA document the
Citizen Science Quality Assurance Project Plan originally developed for water projects by EPA Region 2 (EPA,
2013). That document's scope has been expanded here to broaden its use in other media, such as air. This
Handbook also contains elements from the Uniform Federal Policy for Quality Assurance Project Plans (IDQTF,
2012), and from a national consensus standard (ASQ/ANSI, 2014). And, we include strategies employed by citizen
scientist programs to increase the credibility of their monitoring data (EPA, 1996, Freitag et al., 2016, and
Williams et al., 2015).
Applicability: The Handbook and companion documents are applicable to the collection and use of
environmental data for three broad categories of citizen science projects: increasing public understanding;
scientific studies and research; and legal and policy action.
Page 5 of 32

-------
Handbook for Citizen Science Quality Assurance and Documentation - Version 1
This document is not intended for projects funded by EPA. If funded by EPA, Quality Assurance (QA)
documentation is required by federal regulations (for example, 2 CFR 1500.11 for grants and 48 CFR 46.202
for contracts). QA Documentation must be completed and approved prior to conducting data collection
activities. EPA's Quality Directives (provided in references) must be consulted for QA documentation.
Data used in regulatory and policy decision making and standard setting often must be collected using
approved methods, which may include acceptance testing to demonstrate equivalence to these methods.
Applicable Part(s) of Title 40, Protection of Environment, of the Code of Federal Regulations (CFR) should be
consulted to ensure the study meets sampling, siting, quality assurance and all other requirements.
How to Use This Document
This Handbook should be used with the two companion documents - the Templates and the Compendium of
Examples. This Handbook explains the purpose of each of the templates. The Templates provide instructions,
tables and questions that should be filled in or responded to, and the Compendium provides specific examples of
quality assurance documentation. Together, these documents will help organizations complete a QAPP and
provide information for data users to evaluate the quality of data collected by citizen scientists. The Templates
are recommended but not required. Federal, state, local, tribal, or other organizations may also be contacted for
more assistance or guidance. The following are steps to consider when writing a QAPP.
1. Frame the Project's Purpose
Citizen science and crowdsourcing are terms describing diverse activities involving a range of organizations, uses
and outcomes. Projects can span different environmental media (e.g. water, air, or biota) and different levels of
participant engagement and responsibility. Some citizen science projects are designed for educational or
community engagement purposes only, whereas others are designed to scientifically evaluate environmental
exposure, to perform legally-defensible measurements, or to affect policy. Table 1 lists data uses by EPA and
other organizations, organized by broad categories and specific project purposes (NACEPT, 2016).
We recommend that your organization design a QAPP that matches the intended purpose of the project, as listed
in Table 1. Determining where a project fits within the spectrum of project purposes can help characterize the
necessary level of quality assurance documentation.
Page 6 of 32

-------
Handbook for Citizen Science Quality Assurance and Documentation - Version 1
Table 1. Categories of data use associated with project purposes.
Categories of Data Use
Intended Project Purpose
Increasing public understanding
Community engagement
Education

Environmental condition indicators (screening,
Scientific studies and research
exposure)

Studies and research
Legal and policy action
Regulatory decisions
2. Define the Level of Quality Assurance and Documentation
Once you have identified the project's intended purpose, you should consider the activities you should perform
to meet the needs and expectations of the data user. These activities are the basis of Quality Assurance (EPA,
2002) and may include: designing a plan to sample at representative locations; identifying the volume of air
needed to meet appropriate levels of detection; or creating Standard Operating Procedures (SOPs) to ensure
that volunteers record data uniformly. Quality assurance planning ensures that participants agree to roles and
responsibilities, and that data can be used to answer the questions posed by the project, with a defined level of
confidence to meet the intended use of the data.
A citizen science organization should consider the project purpose and use of data as it selects the appropriate
level of quality assurance and documentation. With this graded approach, data collected for legal and policy
action would require more extensive quality assurance and documentation than data collected for increasing
public understanding. Below, we provide additional information on these broad categories so that your
organization can better determine the activities required to meet the appropriate level of quality assurance and
documentation.
Increasing Public Understanding (light shade)
Projects in this category include studies on basic phenomena or issues, and often have as a primary or
supporting objective to engage communities in environmental monitoring. Common objectives may center
around educating citizens about their environments, scientific processes, and STEM (science, technology,
engineering and mathematics) activities. These projects may result in more qualitative, or descriptive outcomes,
such as presence or absence of specific species. In these projects, sampling locations may be more dependent on
availability of volunteers, rather than based on a rigorous sampling scheme.
Scientific Studies and Research (medium shade)
Projects in this category are aimed at providing data useful for research, feasibility studies, or to identify baseline
conditions or trends in exposure from water or air pollutants. Many projects in this category determine the
effectiveness of environmental decisions, such as evaluating the number of fish traveling upstream after a dam
Page 7 of 32

-------
Handbook for Citizen Science Quality Assurance and Documentation - Version 1
has been removed. Many organizations in this category conduct screening studies for ecological or public health
assessments and utilize risk assessment tools for prioritizing community-based actions.
Legal and Policy Action (dark shade)
Projects requiring the most rigorous level of quality assurance and documentation fall into this category. The
purposes of these projects may include regulatory decision-making at a local, state or national level, and often
use approved federal methods, or acceptance testing to demonstrate equivalence to these methods.
The overall project purpose may include specific objectives for individuals, communities, and institutions (such as
government agencies or universities). Also, the purpose of a project may evolve, or fall into more than one
category of data use. A project may start as a community-based educational project but evolve into a more
rigorous scientific study that evaluates baseline conditions. For example, an effort to engage local communities in
measuring water quality may produce information indicating a need to curb pollution from specific sources. In
these situations, more stringent quality assurance and documentation may be needed and pursuing the highest
level of quality assurance that will meet a project's intended purpose is recommended. A QAPP is not a static
document and should be updated whenever an aspect of the project changes.
3. Factors to Consider
How do you determine the right level of quality assurance and documentation? One factor is whether the data
collected are qualitative or quantitative. In general, quantitative projects (i.e. how much?), as well as projects
using statistical hypothesis tests, require a higher level of quality assurance than qualitative projects (Table 2).
Table 2. Level of detail of quality assurance and documentation for different project purposes. The darker the shading, the
more rigorous the quality assurance level of detail.
Categories of Data Use
Intended Project Purpose
Quantitative
Qualitative
Level of
Detail
Increasing public
Community engagement





understanding
Education





Scientific studies and
research
Environmental condition indicators
(screening, exposure)









Studies and research









Legal and policy action
Regulatory decisions









Projects whose primary purpose is to engage the public might be collecting qualitative information. The
environmental question might address presence or absence of specific classes of plants or animals at targeted
geographic locations or in a watershed. Or, citizen scientists might collect quantitative data on abundance of
certain species, but summarize the data using descriptive measures, such as "low", "medium" or "high".
Page 8 of 32

-------
Handbook for Citizen Science Quality Assurance and Documentation - Version 1
In contrast, projects that provide information for measuring exposure, or for regulatory decision-making,
typically require a quantitative estimation of an important condition indicator. For these projects, the study
question often includes a statistic, such as the mean or median, and a measure of variability, estimated from the
collected data, which can be visually displayed on a graph or map. These projects are often conducted in a well-
defined study area that represents a potentially impacted population (EPA, 2006). Suppose that citizen scientists
are interested in the effectiveness of a pollution control device at a nearby smelter. The lead project scientist
should establish a statistical hypothesis test to measure, estimate, and compare the median concentrations of
air toxicants at specific locations (e.g. upwind and downwind) before and after a new pollution control device
has been installed.
Again, you should pursue the highest level of quality assurance and documentation that will meet your project's
intended purpose.
4.	Link the Intended Use of Data to Recommended Quality Assurance Templates1
Like all scientific projects, citizen science projects employ specific strategies, or activities, to improve the
credibility of data. These activities are grouped into 24 distinct elements found in standard EPA guidance (EPA,
2001; EPA, 2002). For this Handbook, however, we have consolidated these into 19 key elements shown in Table
3 (and Figure 1). The companion document, Templates for Citizen Science Quality Assurance and Documentation,
helps organizations complete these elements. Your organization should fill out the templates based on how the
data are intended to be used and on guidance received from a state and/or EPA regional office, if applicable.
For each template, it is important that you also choose the right level of quality assurance and documentation to
match the project purpose. So, projects addressing regulatory decision-making would require more quality
control (QC) activities, such as more frequent instrument calibrations. Similarly, all projects require some level of
training of volunteers, but projects to address regulatory decision-making would require more intensive training.
5.	Just Getting Started?
The guidance and concepts in the Handbook and Templates can be applied to citizen science projects at any
scale, but we recognize that some groups may be in the early stages of learning how to document data quality or
may be new to data collection entirely. Even at an early stage, EPA recommends projects provide some level of
data quality documentation to help you make use of your data.
As a coordinator of a citizen science project, you may be involved in many aspects of project planning, sample
collection, laboratory analysis, data review, and data assessment and data management. Therefore, it is
important to consider quality assurance (e.g. planning activities you perform to manage the project and collect,
assess, and review data) and quality control (e.g. technical activities you conduct to limit error from instruments
1 The distribution of quality assurance documentation for each of the data uses follows the alignment from tiered guidance
of the EPA Office of Air Quality Planning and Standards, 2017.
Page 9 of 32

-------
Handbook for Citizen Science Quality Assurance and Documentation - Version 1
or in measurements) in every stage of your project. But, this doesn't need to be overwhelming. You can address
the essential elements of quality assurance and documentation by answering these key questions:
1)	What is the purpose of the project, and the question you want to answer?
2)	How and where are you planning to collect samples, data, or other information?
3)	How are you training the volunteers to collect samples, data or other information?
4)	How will you control for errors in the field, in the laboratory, or during data analysis?
5)	How will you check your data and determine if it is useful?
6)	Where do the data go and who will look at the data?
Science is great! Collecting data for a big project is fun and valuable but to make conclusions from the data, you
need to carefully document these activities. We think that as you answer these questions by filling out the
templates in this Handbook, you will be able to better communicate the quality and utility of your project's data.
Page 10 of 32

-------
Handbook for Citizen Science Quality Assurance and Documentation - Version 1
Table 3. EPA QAPP elements and quality assurance templates recommended for citizen science projects. The
templates are organized into four major Quality Assurance Project Plan element listed in EPA guidance
documents.
Template
Increase Public
Understanding
Science/Research
Legal/Policy
A. Managing the Project
1. Title and Preparer Page
X
X
X
2. Table of Contents

X
X
3. Problem Definition, Background and
Project Description
X
X
X
4. Data Quality Objectives and Indicators
X
X
X
5. Project Schedule

X
X
6. Training and Specialized Experience
X
X
X
7. Documents and Records
X
X
X
B. Collecting the Data
8. Existing Data

X
X
9. Sampling Design and Data Collection
Methods
X
X
X
10. Sample Handling and Custody

X
X
11. Equipment/Instrument Maintenance,
Testing Inspection and Calibration

X
X
12. Analytical Methods
X
X
X
13. Field and Laboratory Quality
Control
X
X
X
14. Data Management

X
X
C. Assessing the Data
15. Reporting, Oversight and
Assessments
X
X
X
D. Reviewing the Data
16. Data Review and Usability
X
X
X
Managing the Project (continued)
17. Organization Chart

X
X
18. Project/Task Organization

X
X
19. Project Distribution List

X
X
Page 11 of 32

-------
Handbook for Citizen Science Quality Assurance and Documentation - Version 1
Figure 1. Quality assurance and documentation templates recommended for citizen science projects.
Increasing Public Understanding
Community Engagement, Education
Use of Templates 1,3,4,6,7,9,12,13,15,16 is recommended.
Scientific Studies and Research
Environmental Conditions Indicators, Screening, Studies and Research
Use of Templates 1 -19 is recommended.
Legal and Policy Actions
Regulatory Decisions
Use of Templates 1 through 19 is STRONGLY recommended.
This document does not define, or otherwise limit, the purpose to which citizen scientists may seek to
apply their data or information. EPA retains the discretion to determine the extent it will use data or
information produced or resulting from use of this document. Please note, however, that data used in
regulatory and policy decision making often must be collected using approved methods, which may
include acceptance testing to demonstrate equivalence to these methods. To ensure these studies
meet the sampling, siting, quality assurance and all other requirements, consult the target data end
users for your project.
6. In Summary
This Handbook is meant to convey common expectations for quality assurance process and documentation, and
best management practices for organizations that train and use volunteers in the collection of environmental
data. This Handbook and its companion Templates and Examples will help the citizen science organization select
the appropriate level of quality assurance and documentation to fit the intended use of the data.
For over twenty years, consistent with national consensus standards for collection of environmental data, EPA
has encouraged organizations to use the QAPP format to document the transparency of the project's scientific
methods, and the quality and usefulness of the collected data or information. The process of developing the
QAPP is vital to the success of the project and often just as important as the final written QAPP document. The
QAPP gives structure to the development of a project and helps when writing the conclusions reached at the end
of the project. In addition, evaluation of the project always involves comparing what was actually done with the
requirements in the QAPP.
Page 12 of 32

-------
Handbook for Citizen Science Quality Assurance and Documentation - Version 1
Instructions for Citizen Science Quality Assurance
and Documentation Templates
To develop the QAPP using the templates, complete each of the recommended templates
applicable to your project. Templates may also be combined if the information is clearly
distinguishable.
Page 13 of 32

-------
Handbook for Citizen Science Quality Assurance and Documentation - Version 1
Corresponding Template #1: Title and Preparer Page
This page provides a title, an effective date of the plan, and confirms that key parties agree with the plan.
Corresponding Template #2: Table of Contents
For QAPPs longer than a few pages, the Table of Contents helps ensure all the information has been included
and is easy to find. The table of contents should include a list (as appendices) of Standard Operating Procedures
(SOPs), Instrument Manuals, and any checklists or forms, as applicable.
Corresponding Template #3: Problem Definition, Background and Project Description
A.	Problem Definition
This section describes the environmental problem, question or threat to be addressed, explains why this work
needs to be done, and provides a framework for determining the project purpose, the use of the data, and the
project objectives (see below). As described earlier, the QAPP development process is iterative and the problem
definition can be revised as you gather new information (e.g. limitations of available methods, or results from
complementary studies).
B.	Background
This is an opportunity to describe the history of the project (or environmental problem), relevant previous
studies, and how this project fills in a data gap (including from existing data) or complements existing
information. If the project is related to other on-going projects, then it is important to show how and why it is
related. The more specific you can be, the more favorably it will be received when being considered as an
addition to your (or other) datasets. If similar work is being done by others, then collaboration could possibly lead
to even stronger results. Working within the scope of existing projects may enable you to build on or possibly use
an existing QAPP. A reference to papers or studies that inspired your project are also useful as it may
demonstrate that you have researched and identified appropriate methods for collecting data that apply to the
environmental problem.
C.	Project Description
This section is an opportunity to succinctly describe work to be performed, the data you plan to collect, the
technologies or methods used to collect the data, and the decisions you plan to make with the data. For
example, you may state that volunteers for your project will visually record the number of fish passing through a
fish ladder in two segments of a river during the spring over a three-year period, to determine whether fish
passage increases after the removal of a dam. Additional project information as listed below will help your
organization plan quality assurance activities to meet the project objectives.
Page 14 of 32

-------
Handbook for Citizen Science Quality Assurance and Documentation - Version 1
First, you need to identify the project objectives. These address the problem or answer the environmental
questions and link data results with possible actions. One way to answer this is using "If...then..." statements,
such as "If the result for parameter x in this area is above the regulatory standard, then we will...". These
objectives form the foundation for the entire study. The more precise you are in defining the problem and the
project objectives, the more likely it is that the project will successfully meet those objectives.
This section should also include a brief description of the project site or study area, and locations as they relate
to the environmental questions to be addressed. The definition of a target study area or population ensures that
the samples taken are representative of the intended population, and that the locations are selected based on
the project objectives.
It is also essential to identify the time period for data collection, because your project objectives may apply only
to a specific temporal window. For example, a project recording migratory fish through an estuary depends on
the timing of the life cycle of the species of interest.
Finally, you should include information on the data users. These are the individuals, groups, or agencies who are
interested in, or who will make decisions based on, the data and information.
Corresponding Template #4: Data Quality Objectives and Data Quality Indicators
These are key elements of a QAPP and they translate project objectives into specific quality assurance and
quality control activities.
A. Data Quality Objectives
Data quality objectives (DQOs) are quantitative or qualitative statements describing the degree of the data's
acceptability (i.e. performance criteria) for making decisions described in the project objective. For example, if
you plan to compare results from a continuous air monitor for ozone to the national 8-hour standard, you can
state that the data gathered must be able to measure ozone on an hourly basis, and that variability and
uncertainty are minimized (and reduce the likelihood of "decision errors") to determine statistically whether the
national standard is being met. (Please note that ozone data collected by a citizen science organization would
most likely be used for screening purposes and not for regulatory decision-making.) Qualitative DQOs, however,
do not need to make a statistical statement. For example, a community engagement project may provide a DQO
that plant species in the study area's impacted wetlands are accurately identified by the volunteers.
The process for setting DQOs can be very involved and relies on the information you provide in Templates 3 and
4. It can be summarized in the flowchart in Figure 2.
Page 15 of 32

-------
Handbook for Citizen Science Quality Assurance and Documentation - Version 1
Figure 2. The DQO Planning Process.
For more information on DQO planning, refer to EPA 2006 as found in the References at the back of the
Handbook.
B. Data Quality Indicators
To determine whether the data quality objectives are being met, you should evaluate Data Quality Indicators
(DQI) for each parameter measured. These DQIs are more typical for projects where quantitative measurements
of parameters, such as bacteria or particulate matter in air, or abundance of species, are collected. DQIs are
attributes of the data being collected, specifically related to minimizing the uncertainty for each measurement or
set of measurements. They are listed below:
Precision is the ability of a measurement to consistently be reproduced. Repeated measurements are
usually used to determine precision. In the case of repeated measurements, one would see how close
those measurements agree. Precision is often measured as the relative percent difference or the relative
standard deviation.
Bias is any influence in the project that might sway or skew the data in a particular direction. Taking
samples from one location where a problem is known to exist, instead of taking samples evenly distributed
over a wide area, is one example of how data can be biased. Bias can result from a non-representative
sampling design, calibration errors, unaccounted-for interferences and chronic sample contamination.
Accuracy is a degree of confidence in a measurement. The smaller the difference between the
measurement of a parameter and its "true" or expected value, the more accurate the measurement. Also,
the more precise or reproducible the result, the more reliable or accurate the result. Accuracy can be
determined by comparing an analysis of a chemical standard to its actual value.
Representativeness is how well the collected data depict the true system.
Comparability is the extent to which data from one data set can be compared directly to another data set.
The data sets should have enough common ground, equivalence or similarity to permit a meaningful
analysis.
Completeness is the amount of data that must be collected to achieve the goals and objectives stated for
the project. It is determined by comparing the amount of valid, or usable, data you collected to what you
originally planned to collect.
Sensitivity is essentially the lowest detection limit of a method, instrument or process for each of the
measurement parameters of interest.
Measurement range is the range of reliable readings of an instrument or measuring device, or a
laboratory method, as specified by the manufacturer or the laboratory.
Page 16 of 32

-------
Handbook for Citizen Science Quality Assurance and Documentation - Version 1
Each DQI has an associated activity, such as a Quality Control (QC) check. For example, volunteers recording pH
in water using an electronic pH meter typically perform a calibration of the instrument before and after a
sampling event with known standards. Or, an analytical instrument has a defined sensitivity, or detection limit.
And, all DQIs have a quantitative goal. For an instrument measuring a specific analyte, a rule of thumb is that the
detection limit should be at least three times less than the regulatory action level for that analyte.
If the project has a single use, or is limited in scope, then the data quality objectives will apply to just your
project. If, however, data from your project contributes to a team or network of similar projects, then you should
take into account the data quality objectives for the other projects as well.
More information on selecting and calculating these indicators are available in many of the documents listed in
the references including: EPA, 1996 (The Volunteer Monitor's Guide to Quality Assurance Project Plans); EPA,
2002a (Guidance for Quality Assurance Project Plans. EPA QA/G-5); National Water Quality Monitoring Council
(NWQMC), 2006; and Williams, et al., 2015 (Citizen Science Air Monitor (CSAM) Quality Assurance Guidelines).
Corresponding Template #5: Project Schedule
A project schedule helps identify timeframes necessary for planning sampling events, including preparing or
procuring equipment, informing participants about timing of data collection, and expectations for distribution of
data for review and reporting. Signing off on the QAPP implies that the participants agree to the planned
schedule.
Corresponding Template #6: Training and Specialized Experience
The training of citizen data collectors is a very important part of a project. Training ensures consistency and
should be recorded and documented. Note that records of training can be as simple as an email summarizing
what was taught, by whom and for whom. Training records are essential to evaluate whether procedures are
performed correctly and to document the qualifications of the people involved. Some projects rely on
professional scientists or trainers with specialized experience that greatly assist the project.
Corresponding Template #7: Documents and Records
It is critical that the key project personnel are aware of the location and status of important documents (e.g.
QAPPs, Standard Operating Procedures (SOPs), field data sheets and records (e.g. databases, quality control (QC)
checklist). By attaching copies of procedures and checklists to the QAPP, you can provide consistency for the
whole project. This also assists in training, data analysis and reporting.
Corresponding Template #8: Existing Data and Data from Other Sources
It is often valuable (or necessary) for projects to use existing data. Existing data can include sampling and testing
data collected during previous investigations; historical data, background information; interviews; modeling
Page 17 of 32

-------
Handbook for Citizen Science Quality Assurance and Documentation - Version 1
estimates; photographs; aerial photographs; topographic maps; and published literature, including from Federal
and State agencies. The project team should determine whether the quality of the data collected (even by
reliable sources or reported in a peer reviewed journal article) are acceptable for the objectives of your project.
You should consider the source of the data, the time period during which the data were collected, data collection
methods, potential sources of uncertainty, and the type of supporting documentation available including quality
assurance documentation such as precision, bias, representativeness, comparability, and completeness. Some of
this information may not be available, and you will need to make a judgement on the limitations of the use of the
data. Note that there are many sources of existing data that are recognized as reliable, such as National Weather
Service data, or other national monitoring networks (e.g. United States Geological Survey).
You should complete this template if your project will be using existing data. If there are no existing data being
used, state "no existing data is being used".
Corresponding Template #9: Sampling Design and Data Collection Methods
A. Sampling Design
You need to design a sampling regime that meets the project objectives. For example, you might consider a
statistical, or probability-based design to make inferences over a large geographic area, a watershed, or a
community. Or you may want to target specific areas using a judgmental design to identify "hot-spots" and
contrast "reference" versus "impacted" populations. And, you need to consider the sampling methods or
technologies you will employ, e.g. discrete sampling, in situ sensors, continuous monitoring, or vegetation
transects.
Details about the sampling design and quality control activities should also be documented so that if other
organizations repeat the project, they would generate similar results. Some of this information may occur
elsewhere in the QAPP and can be referenced here.
This template should also list the quantity and type of quality control (QC) samples collected during the project.
QC samples are needed to evaluate whether your data quality objectives and indicators are met (see Template
#4). For example, contamination is a common source of error in both sampling and analytical procedures, so
blank samples are collected to identify when and how contamination might occur. An organization should always
document that blanks are consistently free of contamination. A general rule is that 10 to 20% of field collected
samples should be QC samples. The laboratory must also run its own QC samples (such a positive and negative
controls for microbiological analysis) because they also need to document that their operations do not cause
increases in error. For a new monitoring project or for a new analytical procedure, it is a good idea to increase
the number of QC samples (up to 20%) until you have full confidence in the procedures you are using. Types of
QC samples are described in Table 4. Projects that collect qualitative information, such as general abundance of
specific species, can evaluate quality by having more than one individual make the same assessment. Vegetation
surveys that rely on correct identification of species often send voucher specimens to regional experts.
Page 18 of 32

-------
Handbook for Citizen Science Quality Assurance and Documentation - Version 1
Note that most of these QC samples are designed primarily for discrete water samples and other types of
samples may be required for air or water continuous monitoring. For air monitoring projects, the organization
should co-locate sensors at or near sites where reference instruments have been deployed by regulatory
agencies. If you are mapping vegetation, or conducting a faunal survey, it is good practice to perform replicate
transects to evaluate field variability or have more than one person evaluate the same transect. To assess
accuracy of your taxonomic identification, some scientists recommend use of voucher specimens. A voucher
specimen is any specimen that serves as a basis of study and is retained as a reference. It should be in a publicly
accessible scientific reference collection.
Page 19 of 32

-------
Handbook for Citizen Science Quality Assurance and Documentation - Version 1
Table 4. Typical QC sample types, descriptions and uses for discrete sampling
QC Sample
Type
Description
Useful for
(examples)
Field Blank
A "clean" sample, produced in the field, used to detect or document
contamination during the whole process (sampling, transport, and lab
analysis). Examples include clean sampling containers, blank filters, etc. that
are treated the same as field samples, except no sample is collected in/on
them.
Water, sediment
and soil sampling;
air sampling onto
filters
Equipment
or rinse
blank
This type of blank is used to evaluate if there is carryover contamination
from reuse of the same sampling equipment. A sample of distilled water (or
other solvent, per the method) is collected in a sample container using
regular collection equipment and analyzed as a sample.
Water, sediment
and soil sampling
equipment; filter
air sampling
equipment
Split
sample
Sample that is divided equally into two or more sample containers and then
analyzed by different analysts or laboratories. Used as a measure of
precision or to measure the variability in results between laboratories or
samples independently analyzing the same original sample.
Water, sediment,
soil and fish tissue
sampling
Co-located
samples
Applicable to air and water sampling. For air sampling, two or more sample
collection devices, located together in space and operated simultaneously,
to supply a series of duplicate or replicate samples for estimating precision
of the total measurement system/process and verification with established
methods. Co-located samples must be exposed to the same environment
and exposed to the same air. For water sampling, discrete sampling could
be applied to compare with continuous monitoring stations.
Water, sediment
and soil
Replicate
Samples
Obtained when two or more samples are taken from the same site, at the
same time, using the same method, and independently analyzed in the
same manner. When only two samples are taken, they are sometimes
referred to as duplicate samples. These types of samples are representative
of the same environmental condition. Replicates (or duplicates) can be used
to detect both the natural variability in the environment and that caused by
field sampling methods.
Water, sediment
and soil
Spiked
Samples
Samples to which a known concentration of the analyte of interest has been
added. Spiked samples are used to measure accuracy. If this is done in the
field (which is unusual), the results reflect the effects of matrix,
preservation, shipping, laboratory preparation, and analysis. If done in the
laboratory, they reflect the effects of the analysis from the point when the
compound is added, e.g. just prior to the measurement step. Percent
recovery of the spike material is used to calculate analytical accuracy.
Water, sediment
and soil
Page 20 of 32

-------
Handbook for Citizen Science Quality Assurance and Documentation - Version 1
B. Data Collection Methods
A successful project relies on a consistent protocol for sample collection. Organizations should be able to train
their volunteers in these methods so that data collection is feasible and repeatable. Information on
recommended sample collection methods, specialized containers, or technologies for ambient water and air,
drinking water, and vegetation sampling, may be found in the references and at the following resources:
>	Stream, estuary and lake monitoring: https://www.epa.gov/nps/nonpoint-source-volunteer-monitoring
>	Drinking water monitoring: https://www.epa.gov/sites/production/files/2Q15-
11/documents/drinking water sample collection.pdf
>	New England States Drinking Water Monitoring: https://www.epa.gov/sites/production/files/2Q15-
Q6/documents/N E-States-Sample-Collection-Manual.pdf
>	Air Sensor Monitoring: https://www.epa.gov/air-seiisor-toolbox
>	Continuous water quality monitoring: https://pybs.ysgs.gov/tm/2006/tmlD3/pdf7TlV11D3.pdf
Additional resources are available at the following resources targeted for citizen scientists:
>	https://www.citizenscience.gov/toolkit/resource-library/
>	http://citizenscience.org/
If there are Standard Operating Procedures (SOPs) used, either attach them to the QAPP or cite the publication
where they can be found.
When possible, obtain the exact position (using GPS or smartphone applications) of where the sample came
from, including sample depth, height, and air or water temperature (if appropriate). If during data review, the
reported value appears to be a potential outlier, it may be possible to resample exactly at that point where the
initial sample was taken. It could be that it really was an anomaly, but it could be that something else was
interfering with the physical conditions at that particular point, for example, an unexpected "hot spot" that
should be considered differently from the other values.
Corresponding Template #10: Sample Handling and Custody
To ensure that the sample is not altered after collection or meets the laboratory holding time requirements, this
template describes your efforts to have each collected sample retain its original physical form and chemical
composition through collection to final disposal. It also identifies maintenance of custody (or possession) of the
sample. These principles also apply to continuous monitoring measurements. Chain-of-custody procedures are
especially critical for projects where the data may be used in court as evidence or for making regulatory
decisions. A chain-of-custody log (which may be composed of shipping manifests or receipts, as long as a person
is identified at the point of shipping and receipt at the lab) may be vital to defend the integrity of the data in the
case of suspected tampering. Some methods include seals or other configurations to detect tampering.
Page 21 of 32

-------
Handbook for Citizen Science Quality Assurance and Documentation - Version 1
Corresponding Template #11: Equipment List, Instrument Maintenance, Testing, Inspection and
Calibration
Many citizen scientists use electronic (in situ) sensors in water and air matrices and these instruments need to be
calibrated. All calibrations for a project should be planned for and documented. Calibration records should be
kept on calibration data sheets specific to each piece of equipment and include date, time, name of individual
doing calibration, and the calibration results themselves. Acceptance criteria for calibration checks should also
be included on the data sheets. Acceptance criteria are usually quantitative measures and goals that evaluate
whether the results can be used. For example, you may check your instrument after sampling many sites against
a calibration standard. This is a QC check and called verification (see Template #16). If the difference is within
the acceptance criteria (or DQI goals, see Template #4), usually expressed as a percentage, you can accept the
results.
Calibration is important because it helps prevent bias. Depending on the specifications of the equipment
manufacturers, some equipment needs more attention than others and this template helps in making sure
equipment does not 'drift' over time. This applies especially to continuous monitoring instruments in water and
air. Water sensors are especially prone to drift caused by biofouling and is often distinct from sensor drift.
Analytical and field instruments differ in precision and sensitivity (i.e. lower limit of detection). To compare or
merge datasets, or if instruments are changed during the course of a project, it is critical to document these
specifications.
Corresponding Template #12: Analytical Methods
This template lists in one place all the specifications for the laboratory or field measurement method organized
by each parameter, or analytical group (such as metals, or nutrients). It also helps establish the criteria, such as
reporting limits, needed for the project, based on the data quality objectives identified in Template #4. Often,
the most important specifications here are the actual sample volume the laboratory needs to make the
appropriate analysis and the holding time for the sample. To complete this template, an organization should
consult with the analytical laboratory, or field method manufacturer or authorized representative to ensure that
the amount of material, water, or air sampled is sufficient to meet the target detection (or reporting) limits.
Most information regarding limits and sample volumes for methods can be found online, for both laboratories
using standard methods and for field measurement equipment.
Corresponding Template #13: Field and Analytical Laboratory Quality Control Summary
The tables in this template summarize information from Template Us 4 and 9 on the types of quality control (QC)
samples that will be collected in the field and by the laboratory.
Page 22 of 32

-------
Handbook for Citizen Science Quality Assurance and Documentation - Version 1
Corresponding Template #14: Data Management
As was done for sample handling, the information in this template traces the path of the data, from field
collection and laboratory analysis to final use or storage. This ensures that project personnel use data of known
status or of a defined stage of review or validation. You will need to describe record-keeping procedures, your
document control system, and the approach used for data storage and retrieval on electronic media. Any forms
or checklists should also be included as attachments.
Corresponding Template #15: Reporting, Oversight and Assessments
Assessments and project oversight include reviews (if possible, by an outside reviewer) to identify shortcomings
or departures from the QAPP, corrective actions taken, and limitations of the use of the data. Reporting of
results is critical for ensuring that any deviations are corrected if possible, and if none are found, to document
the successful implementation of procedures. It is also important to report results and interpretations so that
project participants are engaged, and understand the project's progress and results, and limitations or setbacks,
if found. Data and interpretive reports might be distributed to project partners and government agencies, but
also posted to web sites for public access.
Corresponding Template #16: Data Review and Usability
Use this section to describe how your organization will review, verify, and validate data to determine whether
your project objectives were met, and that your data are fit for use. The level of detail and frequency for
performing data review depends on the intended use of the data. You should describe how project personnel
(such as the project coordinator and the Quality Assurance Manager, if available) will review data as they are
obtained and documented (e.g., in checklists, logbooks, emails, and QA reports).
Although data verification and validation are typically conducted sequentially, it may be beneficial (and more
cost effective) for smaller projects to combine steps. For example, the personnel conducting the verification
could also conduct the first step of the validation process concurrently. When multiple people are involved in a
project, the initial data collector typically conducts the first verification at the time of collection and later, if a QA
Manager is available (see Template #17), he or she should review the QC checks recorded on the data sheets.
The first step in review is verification. Verification is determining whether an activity conforms to the stated
requirements, (such as acceptance criteria) for that activity. For example, if you are using a hand-held multi-
meter to determine pH and conductivity at certain points in a stream, you should have established calibration
procedures and QC checks (associated with data quality indicators) for those two parameters. While you are
collecting your data, you would record the results (e.g. on a datasheet or in a notebook) and then review them
to ensure that QC checks such as blanks and calibration checks are within the goals (or acceptance criteria) that
you have set (see Template Us 4, 9 and 11). This verification process indicates that your multi-meter is operating
correctly. Verification also applies if your data are qualitative, such as checking to see if a fish or plant species
identification was performed correctly.
Page 23 of 32

-------
Handbook for Citizen Science Quality Assurance and Documentation - Version 1
The next step in review is validation. Validation is determining whether the activities conform to the user needs
for the overall project. This step is a higher-level activity that introduces additional (mostly quality assurance)
activities such as reviewing sampling locations, training, chain-of-custody, documentation, and appropriate
methods. These activities tie back to the Data Quality Objectives to determine whether the data can be used.
(See Figure 3 below for the distinction between verification and validation.)
Regarding the example used above, if you were measuring conductivity in a stream to evaluate the impacts of
road salt, you might consider the following questions:
1.	Did we collect the right types of data? Can we make a decision based on pH and conductivity alone, or
should we have also taken other measurements such as temperature, nitrate, or dissolved oxygen?
2.	Did we collect data from the right locations in the stream(s)? If we had planned to collect data from all
the appropriate locations, did we meet our completeness goal? Can we compare our results statistically
to a regulatory standard?
3.	Did all the datasets meet 100% of the QC criteria? If not, what is the impact to the overall study? Are
data qualifiers needed to describe certain specific QC issues that did not meet their planned QC
acceptance criteria but do not necessitate complete rejection of the dataset?
All data issues, such as outliers, or batches of samples that do not meet acceptance criteria, should be discussed
with the project lead or QA Manager (if available) and explained in project reports. In some cases, data qualifiers
may be needed to "flag" data that do not meet QC acceptance criteria, but do not require rejection of the full
dataset.2
Figure 3. The distinction between verification and validation.
Verification
Am I doing this process right?
Validation
2 Data qualifiers (sometimes informally called "flags") are a set of codes that are applied to your data to describe various
aspects of an analysis that did not meet the QC goals (or acceptance criteria) described in the QAPP or SOPs. Typical flags
are for high blanks, low check standards, or measurements below the detection limit. For additional information about
common types of data qualifiers, see Appendix C of EPA QA/G-8 Guidance (EPA, 2002b) on Environmental Data Verification
and Data Validation.
Page 24 of 32

-------
Handbook for Citizen Science Quality Assurance and Documentation - Version 1
Finally, you should ensure that your review will evaluate whether you have met the project objectives, and the
data are usable, or whether you have identified and documented limitations in the use of the data. You may
identify significant departures from the QAPP, or incorrect assumptions established in the planning phase of data
collection. You may have uncovered unique qualities of the sample matrix; use of non-standard analytical
methods; or a sampling design that is not representative or does not allow for planned statistical comparisons.
Any potential limitations of the data should be documented in records associated with these data (i.e., metadata)
and included in the final project report.
Corresponding Template #17: Project Organization Chart
The organization chart shows the lines of communication and reporting for the project, like a chain of command.
The chart should represent the personnel responsible for ensuring data quality, such as the project lead, trainers,
and field data collectors. Some projects require a team of people with differing responsibilities and titles; others
may require the services of a single person on a part-time basis. In general, organization charts are not needed
for small projects.
We recommend that you identify one person responsible for the overall integrity of the project, sometimes
referred to as the Quality Assurance Manager (QAM). To ensure the inadvertent introduction of bias, this person
should be independent of the staff working on the project and should have sufficient knowledge of the
methodologies of the project, and the confidence to ask questions. The role of the QAM is to evaluate the
project activities, methods and results and determine whether the data collected are meeting the project
objectives. Many small projects may not have the capability to employ an independent person, so a qualified,
experienced, and trained individual can fill this role.
Corresponding Template #18: Project Organization
This shows everybody's role in the development of the project. The responsibilities section provides an outline of
the work that will be done for the project. Project-specific details are addressed in many of the other templates.
Corresponding Template #19: Project Distribution List
The distribution list ensures everyone involved with the project receives a copy of the QAPP or other documents
and is aware about the work being conducted. It also provides the contact information for those involved with
the project, so everyone is aware if changes to the project are made.
Page 25 of 32

-------
Handbook for Citizen Science Quality Assurance and Documentation - Version 1
Definitions3
Acceptance criteria. Quantitative measures and goals that evaluate whether the results can be used.
Accuracy. A data quality indicator, accuracy is the extent of agreement between an observed value (sampling
result) and the accepted, or true, value of the parameter being measured. High accuracy can be defined as a
combination of high precision and low bias.
Analyte. Within a medium, such as water, an analyte is a property or substance to be measured. Examples of
analytes would include pH, dissolved oxygen, bacteria, and heavy metals.
Assessment. The evaluation process used to measure the performance or effectiveness of a system and its
elements. As used here, assessment is an all-inclusive term used to denote any of the following: audit,
performance evaluation, management systems review, peer review, inspection, or surveillance.
Audit (quality). A systematic and independent examination to determine whether quality activities and
related results comply with planned arrangements and whether these arrangements are implemented
effectively and are suitable to achieve objectives.
Bias. Often used as a data quality indicator, bias is the degree of systematic error present in the assessment or
analysis process. When bias is present, the sampling result value will differ from the accepted, or true, value of
the parameter being assessed.
Blind sample. A type of sample used for quality control purposes, a blind sample is a sample submitted to an
analyst without their knowledge of its identity or composition. Blind samples are used to test the analyst's or
laboratory's expertise in performing the sample analysis.
Calibration. Comparison of a measurement standard, instrument, or item with a standard or instrument of
higher accuracy to detect and quantify inaccuracies and to report or eliminate those inaccuracies by
adjustments.
Chain-of-custody. An unbroken trail of accountability that ensures the physical security of samples, data,
and records.
Citizen science. When the public participates voluntarily in the scientific process, addressing real-world
problems in ways that may include formulating research questions, conducting scientific experiments, collecting
and analyzing data, and interpreting results.
3 EPA, 2001 - EPA Requirements for Quality Assurance Project Plans, March 2001: QA/R-5. EPA/240/B-01/003. Office of
Environmental Information. U.S. Environmental Protection Agency. Washington, DC 20460.
https://www.epa.gov/sites/prodnction/files/2016-06/ciocnments/r5-finai O.pdf
Page 26 of 32

-------
Handbook for Citizen Science Quality Assurance and Documentation - Version 1
Contractor. Any organization or individual that contracts to furnish services or items or perform work; a
supplier in a contractual situation.
Comparability. A data quality indicator, comparability is the degree to which different methods, data sets,
and/or decisions agree or are similar.
Completeness. A data quality indicator that is generally expressed as a percentage, completeness is the amount
of valid data obtained compared to the amount of data planned.
Data quality assessment. A statistical and scientific evaluation of the data set to determine the validity and
performance of the data collection design and statistical test, and to determine the adequacy of the data set
for its intended use.
Data quality indicators. Attributes of the data being collected, specifically related to minimizing the uncertainty
for each measurement or set of measurements. These typically include precision, bias, accuracy,
representativeness, comparability, completeness, sensitivity and measurement range.
Data quality objectives (DQOs). Data quality objectives are quantitative and qualitative statements describing
the degree of the data's acceptability or utility to the data user(s). They include indicators such as accuracy,
precision, representativeness, comparability, and completeness. DQOs specify the quality of the data needed to
meet the monitoring project's goals. The planning process for ensuring environmental data are of the type,
quality, and quantity needed for decision making is called the DQO process.
Data usability. The process of ensuring or determining whether the quality of the data produced meets the
intended use of the data.
Data users. The group(s) that will be applying the data results for some purpose. Data users can include the
monitors themselves as well as government agencies, schools, universities, businesses, watershed organizations,
and community groups.
Detection limit. Applied to both methods and equipment, detection limits are the lowest concentration of a
target analyte that a given method or piece of equipment can reliably ascertain and report as greater than zero.
Duplicate sample. Used for quality control purposes, duplicate samples are two samples taken at the same time
from, and representative of, the same site that are carried through all assessment and analytical procedures in
an identical manner. Duplicate samples are used to measure natural variability as well as the precision of a
method, monitor, and/or analyst. More than two duplicate samples are referred to as replicate samples.
Environmental conditions. The description of a physical medium (e.g., air, water, soil, sediment) or biological
system expressed in terms of its physical, chemical, radiological, or biological characteristics.
Page 27 of 32

-------
Handbook for Citizen Science Quality Assurance and Documentation - Version 1
Environmental data. Any measurements or information that describe environmental processes, location, or
conditions; ecological or health effects and consequences; or the performance of environmental technology.
Environmental data also includes information collected directly from measurements, produced from models,
and compiled from other sources such as data bases or the literature.
Environmental sample. An environmental sample is a specimen of any material collected from an
environmental source, such as water or macroinvertebrates collected from a stream, lake, or estuary.
Equipment or rinse blank. Used for quality control purposes, equipment or rinse blanks are types of field blanks
used to check specifically for carryover contamination from reuse of the same sampling equipment (see field
blank).
Field blank. Used for quality control purposes, a field blank is a "clean" sample (e.g., distilled water) that is
otherwise treated the same as other samples taken from the field. Field blanks are submitted to the analyst
along with all other samples and are used to detect any contaminants that may be introduced during sample
collection, storage, analysis, and transport.
Instrument detection limit. The instrument detection limit is the lowest concentration of a given substance or
analyte that can be reliably detected by analytical equipment or instruments (see detection limit).
Matrix. A matrix is a specific type of medium, such as surface water or sediment, in which the analyte of interest
may be contained.
Measurement range. The measurement range is the extent of reliable readings of an instrument or measuring
device, as specified by the manufacturer.
Metadata.4 The simplest definition of metadata is "structured data about data." Metadata is structured
information that describes, explains, locates, or otherwise makes it easier to retrieve, understand, use or manage
an information resource.
Method detection limit (MDL). The M DL is the lowest concentration of a given substance or analyte that can be
reliably detected by an analytical procedure (see detection limit).
Performance evaluation (PE) samples. Used for quality control purposes, a PE sample is a type of blind sample.
The composition of PE samples is unknown to the analyst. PE samples are provided to evaluate the ability of the
analyst or laboratory to produce analytical results within specified limits.
Precision. A data quality indicator, precision measures the level of agreement or variability among a set of
repeated measurements, obtained under similar conditions. Precision is usually expressed as a standard
deviation in absolute or relative terms.
4 EPA Classification No.: CIO 2135-S-01.0 - Enterprise Information Management (EIM) Minimum Metadata Standards
Page 28 of 32

-------
Handbook for Citizen Science Quality Assurance and Documentation - Version 1
Protocols. Protocols are detailed, written, standardized procedures for field and/or laboratory operations.
Quality assurance (QA). QA is an integrated management system designed to ensure that a product or service
meets defined standards of quality with a stated level of confidence. QA activities involve planning quality
control, quality assessment, reporting, and quality improvement.
Quality Assurance Manager (QAM). The individual designated as the principal manager within the organization
having management oversight and responsibilities for planning, documenting, coordinating, and assessing the
effectiveness of the quality system for the organization.
Quality assurance project plan (QAPP). A QAPP is a formal written document describing the detailed quality
control procedures that will be used to achieve a specific project's data quality requirements.
Quality control (QC). QC is the overall system of technical activities designed to measure quality and limit error
in a product or service. A QC program manages quality so that data meets the needs of the user as expressed in
a quality assurance project plan.
Relative standard deviation (RSD). RSD is the standard deviation of a parameter expressed as a percentage and
is used in the evaluation of precision.
Relative percent difference (RPD). RPD is an alternative to standard deviation, expressed as a percentage and
used to determine precision when only two measurement values are available.
Replicate samples. See duplicate samples.
Representativeness. A data quality indicator, representativeness is the degree to which data accurately and
precisely portray the actual or true environmental condition measured.
Sensitivity. Related to detection limits, sensitivity refers to the capability of a method or instrument to
discriminate between measurement responses representing different levels of a variable of interest. The more
sensitive a method is, the better able it is to detect lower concentrations of a variable.
Spiked samples. Used for quality control purposes, a spiked sample is a sample to which a known concentration
of the target analyte has been added. When analyzed, the difference between an environmental sample and the
analyte's concentration in a spiked sample should be equivalent to the amount added to the spiked sample.
Split sample. Used for quality control purposes, a split sample is one that has been equally divided into two or
more subsamples. Splits are submitted to different analysts or laboratories and are used to measure the
precision of the analytical methods.
Page 29 of 32

-------
Handbook for Citizen Science Quality Assurance and Documentation - Version 1
Standard reference materials (SRM). An SRM is a certified material or substance with an established, known
and accepted value for the analyte or property of interest. Employed in the determination of bias, SRMs are
used as a gauge to correctly calibrate instruments or assess measurement methods. SRMs are produced by the
U. S. National Institute of Standards and Technology (NIST) and characterized for absolute content independent
of any analytical method.
Standard deviation(s). Used in the determination of precision, standard deviation is the most common
calculation used to measure the range of variation among repeated measurements. The standard deviation of a
set of measurements is expressed by the positive square root of the variance of the measurements.
Standard operating procedures (SOPs). An SOP is a written document detailing the prescribed and established
methods used for performing project operations, analyses, or actions.
True value. In the determination of accuracy, observed measurement values are often compared to true, or
standard, values. A true value is one that has been sufficiently well established to be used for the calibration of
instruments, evaluation of assessment methods or the assignment of values to materials.
Validation. Confirmation by examination and provision of objective evidence that the particular requirements
for a specific intended use are fulfilled. In design and development, validation concerns the process of examining
a product or result to determine conformance to user needs.
Variance. A statistical term used in the calculation of standard deviation, variance is the sum of the squares of
the difference between the individual values of a set and the arithmetic mean of the set, divided by one less
than the numbers in the set.
Verification. Confirmation by examination and provision of objective evidence that specified requirements have
been fulfilled. In design and development, verification concerns the process of examining a result of a given
activity to determine conformance to the stated requirements for that activity.
Page 30 of 32

-------
Handbook for Citizen Science Quality Assurance and Documentation - Version 1
References
American Society of Quality/American National Standards Institute, 2014 - Specifications and Guidelines for
Quality Systems for Environmental Data Collection and Environmental Technology Programs. ASQ/ANSI E4-2014.
EPA, 1996 - The Volunteer Monitor's Guide to Quality Assurance Project Plans. EPA 841-B-96-003.
https://www.epa.gov/sites/production/files/2Q15-Q6/docurnents/vol qapp.pdf
EPA, 2001 - EPA Requirements for Quality Assurance Project Plans, March 2001: QA/R-5. EPA/240/B-01/003.
Office of Environmental Information. U.S. Environmental Protection Agency. Washington, DC 20460.
https://www.epa.gov/sites/production/files/2Q16-Q6/documents/r5-final Q.pdf
EPA, 2002a - Guidance for Quality Assurance Project Plans. EPA QA/G-5. EPA/240/R-02/009. Office of
Environmental Information. United States Environmental Protection Agency. Washington, DC 20460.
https://www.epa.gov/sites/production/files/2Q15-Q6/documents/g5-final.pdf
EPA, 2002b - Guidance on Environmental Data Verification and Data Validation. EPA QA/G-8. EPA/240/R-02/004.
Office of Environmental Information. United States Environmental Protection Agency. Washington, DC 20460.
https://www.epa.gov/sites/production/files/2Q15-Q6/documents/g8-final.pdf
EPA, 2006 - Guidance on Systematic Planning Using the Data Quality Objectives Process. EPA QA/G-4.
EPA/240/B-06/001. Office of Environmental Information. U.S. Environmental Protection Agency. Washington, DC
20460. https://www.epa.eov/qyalitv/gyidaiice-svstematic-plaririirig-ysirig-data-qyalitv-obiectives-process-epa-
.ggg-:4
EPA, 2013 - US EPA Region 2 Guidance for the Development of Quality Assurance Project Plans for Citizen Science
Projects, April 2013. https://www3.epa.gov/regionQ2/citizenscience/pdf/citsci air attach b form.pdf
EPA, 2015 - Village Green Project - Self-Powered, Real-Time Air Monitoring, Quality Assurance Project Plan -
Original Draft, Category IV/Measurement Project, February 2015.
EPA Office of Air Quality Planning and Standards, 2017 - Quality Assurance Handbook for Air Pollution
Measurement Systems Volume II Ambient Air Quality Monitoring Program. EPA-454/B-17-001 Appendix C, Using
the Graded Approach for the Development of QMPs and QAPPs.
https://www3.epa.gov/ttn/amtic/files/ambient/pm25/qa/Final%2QHandbook%2QDocument%2Ql 17.pdf
Freitag, A, Meyer, R. and Whiteman, L., 2016 - Strategies Employed by Citizen Science Programs to Increase the
Credibility of Their Data. Citizen Science: Theory and Practice, 1(1): 2, pp. 1-11, DOI:
http://dx.doi.Org/lQ.5334/cstp.6
Page 31 of 32

-------
Handbook for Citizen Science Quality Assurance and Documentation - Version 1
IDQTF, 2012 - Uniform Federal Policy for Quality Assurance Project Plans (UFP-QAPP Manual, 2005) - March 2012
Optimized UFP-QAPP Worksheets. Intergovernmental Data Quality Task Force. EPA-505-B-04-900A.
https://www.epa.gov/sites/production/files/documents/ufp qapp worksheets.pdf
National Advisory Council for Environmental Policy and Technology (NACEPT), 2016 - Environmental Protection
Belongs to the Public. A Vision for Citizen Science at EPA. EPA 219-R-16-001. https://www.epa.gov/faca/nacept-
2016-report-environmental-protection-belongs-public-vision-citizen-science-epa
National Water Quality Monitoring Council, 2006 - Data Elements for Reporting Water Quality Monitoring
Results for Chemical, Biological, Toxicological, and Microbiological Analytes. Technical Report No.3. April 2006.
New Jersey Department of Environmental Protection Division of Water Monitoring and Standards.
http://www.state.nj.us/dep/wms/bears/cwm qapps.htm. Accessed May 23, 2018.
Williams, R., Vasu Kilaru, E. Snyder, A. Kaufman, T. Dye, A. Rutter, A. Russell, and H. Hafner, 2014 - Air Sensor
Guidebook. U.S. Environmental Protection Agency. Washington, DC. EPA/600/R-14/159 (NTIS PB2015-100610).
https://www.epa.gov/air-serisor-toolbox/how-yse-air-serisors-air-sensor-gyidebook
Williams, R. W., T. M. Barzyk, and A. Kaufman, 2015 - Citizen Science Air Monitor (CSAM) Quality Assurance
Guidelines. U.S. Environmental Protection Agency. Washington, DC. EPA/600/R-15/008.
https://www.epa.gOv/air-seiisor-toolbox/air-sensor-toolbox-resoyrces-and-fyridiiig#TER
Page 32 of 32

-------