a EPA
United States
Environmental Protection
Agency
Office of Environmental
Information
Washington, DC 20460
EPA/240/B-07/001
March 2007
Systematic Planning: A Case
Study of Participate Matter
Ambient Air Monitoring
EPA QA/CS-2
-------
EPA QA/CS-2 ii March 2007
-------
FOREWORD
This document shows the use of the Data Quality Objectives (DQO) Process in the form
of a case study involving particulate matter ambient air monitoring. The U.S. Environmental
Protection Agency (EPA) has developed the DQO Process for project managers and planners to
help them collect the appropriate type, quantity, and quality of data needed to support Agency
actions.
Systematic Planning Using the Data Quality Objectives Process: A Case Study of
Particulate Matter Ambient Air Monitor ing \$ one of a series of quality management documents
that the EPA Quality Staff has prepared to assist users in implementing the Agency-wide Quality
System. Other related documents include:
EPA QA/G-4 Systematic Planning using the Data Quality Objectives Process
EPA QA/G-5 Guidance for Quality Assurance Project Plans
EPA QA/G-9R Data Quality Assessment: A Reviewer's Guide
EPA QA/G-9S Data Quality Assessment: Statistical Methods for Practitioners
This document provides guidance to EPA program managers and planning teams as well
as to the general public as appropriate. It does not impose legally binding requirements and may
not apply to a particular situation based on the circumstances. EPA retains the discretion to
adopt approaches on a case-by-case basis that differ from this guidance where appropriate.
This case study is one of the U.S. Environmental Protection Agency Quality System
Series documents. These documents describe the EPA policies and procedures for planning,
implementing, and assessing the effectiveness of the Quality System. These documents are
updated periodically to incorporate new topics and revisions or refinements to existing
procedures. Comments received on this version, will be considered for inclusion in subsequent
versions. Please send your comments to:
Quality Staff (2811R)
Office of Environmental Information
U.S. Environmental Protection Agency
1200 Pennsylvania Avenue N.W.
Washington, DC 20460
Phone: (202)564-6830
Fax: (202)565-2441
E-mail: quality@epa.gov
Copies of the EPA's Quality System documents may be downloaded from the Quality Staff
Home Page: www.epa.gov/quality.
EPA QA/CS-2 iii March 2007
-------
EPA QA/CS-2 iv March 2007
-------
PREFACE
Systematic Planning: A Case Study of Paniculate Matter Ambient Air Monitoring
describes how the Data Quality Objectives (DQO) Process is applied in a decision-making
situation. Through a detailed case study, this document shows how systematic planning, and the
DQO Process in particular, leads to sound data collection techniques, selection of proper
sampling methods, and the ability to analyze the collected data to make necessary decisions. As
noted by this case study, systematic planning is conducted by a planning committee whose
activities and discussion provide the basis for defining the problem, phrasing study questions to
be addressed, and designing a method for obtaining and analyzing information to address these
questions.
While this case study focuses primarily on the systematic planning process, it also
addresses the implementation stage where data are collected according to an approved QA
Project Plan, as well as how the collected data would be assessed relative to their intended use.
It explains how the choice of statistical technique for data analysis was made as a result of
discussions held between members of the planning committee and a consulting statistician.
The case study is intended for all EPA and extramural organizations that 1) have quality
systems based on EPA policies and specifications, 2) may periodically assess these quality
systems for compliance to the specifications, or 3) may be assessed by EPA. The use of the
DQO Process is consistent with EPA Order 5360.1 A2 (issued May 5, 2000) which calls for the
use of systematic planning in the collection of environmental data.
The techniques discussed in this case study are non-mandatory, and the case study is
intended to help project managers and staff understand how the DQO Process should be applied
in practical situations. The techniques discussed in the case study are appropriate for this
particular situation but should not necessarily be used as a template or recommendation for the
investigation of similar studies. The techniques discussed in this case study are real, although
the location and identifying characteristics of the actual power plant have been obscured to
protect its identity.
EPA QA/CS-2 v March 2007
-------
EPA QA/CS-2 vi March 2007
-------
TABLE OF CONTENTS
1.0 INTRODUCTION 1
1.1 Case Study Background 2
1.2 Site History and Description 4
2.0 PRELIMINARY ACTIVITIES PRIOR TO FIRST MEETING 5
2.1 Preliminary Documentation 6
2.2 Identification of Key Participants 7
2.3 Identification of Other Interested Parties 7
2.4 Schedule of Meetings 7
3.0 FIRST MEETING OF PLANNING COMMITTEE 9
3.1 Step 1: Define the Problem 9
3.2 Step 2: Identify the Goal of the Study 10
3.3 Step 1 Revision: Define the Problem 10
3.4 Step 2 Revision: Identify the Goal of the Study 12
3.5 Step 3: Identify Information Inputs 13
3.6 Step 4: Define the Boundaries of the Study 15
3.7 Identifying Information Needed for Subsequent Steps 16
4.0 INTERIM ASSIGNMENTS PRIOR TO THE SECOND MEETING 17
4.1 Reviewing Sampling and Analysis Methods 17
4.2 Determining Numbers of Samples for a Fixed Cost,
Considering a Fixed Sampler Versus a Mobile Sampler 17
4.3 Investigating Use of Existing Data 17
4.4 Identifying Regulatory Guidelines 18
5.0 SECOND MEETING OF PLANNING COMMITTEE 19
5.1 Discussion of the Findings of Interim Assignments 19
5.1.1 Step 2 Revision: Identify the Goal of the Study 19
5.1.2 Step 3 Revision: Identify Information Inputs 20
5.1.3 Step 4 Revision: Define the Boundaries of the Study 21
5.2 Step 5: Develop the Analytic Approach 22
5.3 Step 6: Specify Acceptance or Performance Criteria 24
5.4 Step 7: Develop the Detailed Plan for Obtaining Data 25
5.5 Identifying Areas for Further Review and Additional Information Needs 25
6.0 INTERIM ASSIGNMENTS PRIOR TO THE THIRD MEETING 27
6.1 Performance Criteria for Tolerance Intervals 27
6.2 Options for Sample Sizes 27
7.0 THIRD MEETING OF PLANNING COMMITTEE 29
7.1 Discussion of the Findings of Interim Assignments 29
EPA QA/CS-2 vii March 2007
-------
TABLE OF CONTENTS (cont.)
7.1.1 Finalizing Steps 1 through 4 29
7.1.2 Step 5 Revision: Develop the Analytic Approach 29
7.1.3 Example Performance Criteria, Corresponding Sample Size,
and Budget Impact 29
7.2 Specific Sample Collection Design and Alternatives 30
7.3 Completion of Steps 6 and 7: Selection of Final Sample Design 32
7.4 Review of Final DQOs 32
8.0 FROM PLANNING TO IMPLEMENTATION AND ASSESSMENT 36
8.1 Completion of the Planning Stage 36
8.2 Implementation Stage 39
8.3 Assessment Stage 39
8.4 Conclusion 40
9.0 REFERENCES 42
LIST OF FIGURES
Page
Figure 1. The Project Life Cycle 1
Figure 2. The Data Quality Objectives (DQO) Process 5
Figure 3. Iteration of Steps Within the DQO Process 35
LIST OF TABLES
Table 1. Relationship Between Tolerance Interval Performance
Criteria and Minimum Sample Size 30
Table 2. DQOs for the Post-Construction Monitoring Study of PMio
Concentrations in Ambient Air in the Vicinity of
the Emmerton Power Plant 33
EPA QA/CS-2
Vlll
March 2007
-------
LIST OF ACRONYMS USED IN THIS CASE STUDY
AQIA Ambient Air Quality Impact Assessment
CFR Code of Federal Regulations
DQA Data Quality Assessment
DQO Data Quality Objectives
EPA Environmental Protection Agency
NAAQS National Ambient Air Quality Standard
NIST National Institute of Standards and Technology
PM Particulate Matter
PMio Particulate Matter less than 10 microns in diameter
PM2.s Particulate Matter less than 2.5 microns in diameter
PSD Prevention of Significant Deterioration
QA Quality Assurance
QAPP Quality Assurance Project Plan
SOP Standard Operating Procedure
WINS Well Impactor Ninety Six
EPA QA/CS-2
IX
March 2007
-------
EPA QA/CS-2 x March 2007
-------
SYSTEMATIC PLANNING: A CASE STUDY OF PARTICIPATE MATTER
AMBIENT AIR MONITORING
1.0 INTRODUCTION
This case study, Systematic Planning: A Case Study of Particulate Matter Ambient Air
Monitoring, represents an investigation of ambient air particulate matter concentrations
following the installation of upgrades to a large coal-fired power plant. Its purpose is to
demonstrate the importance of a systematic planning process in the use of existing data as well
as the collection of new data to address an environmental monitoring problem. This case study
demonstrates how use of the iterative Data Quality Objectives (DQO) Process can ensure that
data to be obtained for such a study will be of sufficient quality and quantity to address the
study goal.
The Project Life Cycle consists of three project stages - planning, implementation, and
assessment - each of which contains activities and tools that are applied or prepared on
individual data collection projects to ensure that project objectives are achieved. These stages
and their primary components are illustrated in Figure 1. (A fourth stage, reporting and the
improvement process, is often added.) This case study illustrates how systematic planning can
be effectively implemented on a project, emphasizing its iterative nature, and how the Project
Life Cycle proceeds following its completion.
Systematic
Planning
(e.g., DQO Process)
t
OA
Project Plan
\v Expe
r
Standard
Operating
Procedures
;t Studv/V ^
riment jS
t
Technical
Assessments
Data Verification
& Validation
1
Data Quality
Assessment
Figure 1. The Project Life Cycle
The monitoring project described in this case study will generate data that must
demonstrate a known and appropriate level of quality due to its intended use in supporting
conclusions on public health risk. Thus, the systematic planning process will adhere to EPA's
Information Quality Guidelines (as detailed in Guidelines for Ensuring and Maximizing the
Quality, Objectivity, Utility, and Integrity of Information Disseminated by the Environmental
Protection Agency (U.S. EPA 2002a) in ensuring that the data to be collected will meet basic
quality standards on objectivity, utility, and integrity. The systematic planning process also
allows for collected environmental information to achieve the following General Assessment
EPA QA/CS-2
March 2007
-------
Factors, which EPA's Science Policy Council published in^4 Summary of General Assessment
Factors for Evaluating the Quality of Scientific and Technical Information (U.S. EPA 2003):
• Soundness: The extent to which the scientific and technical procedures, measures,
methods, or models employed to generate the information are reasonable for, and
consistent with, the intended application.
• Applicability and Utility: The extent to which the information is relevant for the
intended use.
• Clarity and Completeness: The degree of clarity and completeness with which the
data, assumptions, methods, quality assurance, sponsoring organizations and
analyses employed to generate the information are documented.
• Uncertainty and Variability: The extent to which the variability and uncertainty
(quantitative and qualitative) in the information or the procedures, measures,
methods, or models are evaluated and characterized.
• Evaluation and Review: The extent of independent verification, validation, and
peer review of the information or of the procedures, measures, methods, or models.
This case study illustrates application of systematic planning using the DQO Process
for a specific project.
1.1 Case Study Background
A large coal-fired power plant located just southwest of the small Midwestern city of
Emmerton planned to expand its generating capacity by installing a new boiler. When this
expansion was announced, the ambient air quality in Emmerton was considered to be in
attainment with the 24-hour National Ambient Air Quality Standard (NAAQS) for particulate
matter less than ten microns in diameter (PMio). Therefore, the power plant owners, A&B Inc.,
were required to obtain a Prevention of Significant Deterioration (PSD) permit from the State
EPA before initiating the expansion.
A&B Inc. proposed that the power plant's expansion would increase the plant's coal
combustion rate. Therefore, the expansion also included a new add-on pollution control device
for reducing the plant's sulfur oxide and nitrogen oxide emissions. Nevertheless, the
expansion still had the potential to yield a significant net emission increase of PMio levels in
Emmerton. For this reason, and as dictated by PSD regulations, A&B Inc. monitored pollutant
levels that had the potential to increase in ambient air as a result of the expansion (including
PMio), and collected meteorological data, for a period of 12 months prior to submitting the
PSD permit application (\.Q.,pre-construction monitoring).
The NAAQS primary 24-hour PMio standard is 150 ng/m3, and the allowable PSD
increment, or the maximum amount that the ambient PMio concentration could increase from
the expansion without resulting in significant deterioration of air quality, was determined to be
15 |J,g/m3. For its PSD permit application, A&B Inc. conducted an ambient air quality impact
assessment (AQIA) to demonstrate that allowable emission increases following the proposed
upgrade would not cause or contribute to violations of the NAAQS or the PSD increment. The
AQIA used the pre-construction monitoring data and an applicable air quality dispersion model
EPA QA/CS-2 2 March 2007
-------
(specified in US EPA's Guideline on Air Quality Models) to calculate the post-construction
ambient PMi0 concentration, or the expected average concentration that would occur following
the proposed expansion.
Based on the pre-construction monitoring data, the estimated average 24-hour pre-
construction ambient PMi0 concentration was 80 |J,g/m3. The air quality dispersion model
calculated that average 24-hour ambient PMio concentration would increase by 9 ng/rn3
following the expansion. Therefore, the AQIA estimated a post-construction average ambient
air concentration of 89 |J,g/m3. Because this average concentration was less than the NAAQS
PMio standard (150 |j,g/m3), and the size of the expected average increase (9 ng/rn3) was less
than the allowable PSD increment (15 ng/m3), the State EPA continued processing A&B Inc. 's
PSD permit application and published a draft permit approval action for public comment. The
draft permit received by the plant specifies allowable emission rates (e.g., Ibs/hour) for each
emission source within the facility that would be affected by the expansion. The draft permit
requires that the plant demonstrate adherence to these rates through stack testing, but does not
require the plant to conduct post-construction ambient air monitoring to confirm conformance
with the NAAQS and the allowable PSD increment. Although the permit does not specify the
PSD increment nor the NAAQS for ambient air, it is implied that compliance with these
ambient air standards will occur if the plant achieves the permitted emission rates.
During the public comment period for the draft permit approval action, an
environmental citizens' group voiced concern to the State EPA regarding the planned upgrades
to the power plant. The group was concerned that the results of dispersion modeling were not
sufficiently accurate to be able to state with high confidence that the health of Emmerton's
citizens would be protected following the upgrades. This would imply that the PSD provisions
on performing an acceptable AQIA, which would demonstrate that allowable post-upgrade
emission increases would not cause or contribute to violations of the NAAQS or the PSD
increment, were not satisfied. The health concern was elevated by the U.S. EPA's recent
revisions to its NAAQS for parti culate matter (Code of Federal Regulations, 40 fCFRJ Part
50) that would apply to particles from industrial sources. As the proposed NAAQS revisions
were based on the U.S. EPA's review of the scientific literature relating to the health risks
associated with exposure to particulate matter pollution, the citizens' group was particularly
concerned that the power plant upgrades would put Emmerton's more vulnerable citizens,
including the elderly and asthmatic children, at risk for respiratory health effects. The citizens'
group urged that A&B Inc. should be required by its PSD permit to conduct post-construction
monitoring of PMio to:
• Assist in determining the impact of the plant upgrades on local air quality,
• Verify that the area in which Emmerton is located will remain in attainment for
PMio NAAQS upon A&B Inc. 's implementation of the upgrades and that the
allowable increment will not be exceeded, and
• Verify the accuracy and correctness of the assumptions and methods applied in the
plant's AQIA.
Upon learning of the concerns raised by the citizens' group, A&B Inc. agreed that once
the facility expansion was completed and brought online, they would continue monitoring
EPA QA/CS-2 3 March 2007
-------
concentrations to demonstrate that ambient air levels remained within acceptable levels
following the expansion. They agreed to organize a planning committee to develop a post-
construction monitoring program. The draft PSD permit was subsequently approved.
1.2 Site History and Description
During coal combustion, a number of gaseous pollutants (sulfur oxides, nitrogen oxides,
and others) are produced. In addition to gaseous pollutants, fly ash (soot) paniculate emissions
are generated. Coal-fired power plants utilize filters and electrostatic precipitators to trap
particles produced during coal combustion with good efficiency. However, those particles that
are not retained at the power plant are emitted to the air and, when in sufficient concentration,
can lead to reduced visibility and a potential human health risk. Secondary paniculate matter can
also be formed in power plant plumes from the reactions of sulfur and nitrogen oxides in the
atmosphere to produce aerosol particles, made of sulfur and nitrate salts.
A&BInc. 's power plant is located approximately 10 miles southwest of Emmerton's city
limits. A&BInc. conducted pre-construction PMio monitoring just south of Emmerton according
to EPA's Reference Method for Determination of Paniculate Matter as PM10 in the Atmosphere
(40 CFR Part 50, Appendix M), and the sampling location was determined according to
guidelines established in EP'A's Ambient Monitoring Guidelines for Prevention of Significant
Deterioration (PSD) (U.S. EPA1987). For pre-construction monitoring, A&B Inc. arranged to
locate the monitoring equipment on the grounds of a nearby private golf course. Given the
historical predominant wind direction in Emmerton (southwest), any emissions by the power
plant would be expected to impact ambient air quality at this location. The location also had the
logistical features (e.g., shelter, power, easy access for operators) needed to perform the pre-
construction monitoring of PMio and meteorological parameters.
EPA QA/CS-2 4 March 2007
-------
2.0 PRELIMINARY ACTIVITIES PRIOR TO FIRST MEETING
As the first step in designing its post-construction monitoring effort, A&B Inc.
determined that it would utilize EPA's systematic planning seven-step DQO Process (Figure 2)
to produce a cost effective monitoring design that fully addressed the public concerns. A&B Inc.
decided to form a planning committee and charged it with implementing the DQO Process.
Information on the practical implementation of the DQO Process was obtained from Systematic
Planning: A Case Study for Hazardous Waste Site Investigations (EPA QA/CS-1)
Step 1. Define the Problem.
Describe the problem that motivates the study;
identify the planning team, examine budget, schedule
Step 2. Identify the Goal of the Study.
State how environmental data will be used in solving the problem;
identify study questions, define alternative outcomes
Step 3. Identify Information Inputs.
Identify data & information needed to answer study questions.
Step 4. Define the Boundaries of the Study
Specify the target population & characteristics of interest,
define spatial & temporal limits, scale of inference
Step 5. Develop the Analytic Approach.
Define the parameter of interest, specify the type of inference,
and document the logic for drawing conclusions from findings
I
Statistical
hypothesis testing
I
; i
Estimation and other
analytic approaches
I
Step 6. Specify Acceptance or Performance Criteria
1 1
Specify probability limits for
false rejection and false
acceptance decision errors
i
Specify performance metrics
and acceptable levels
of uncertainty
r i
A
1
L
r
r
Step 7. Develop the Detailed Plan for Obtaining Data
Select the most resource-effective sampling and analysis plan
that satisfies the performance criteria specified in Steps 1-6
Figure 2. The Data Quality Objectives (DQO) Process
The chief environmental engineer for the Emmerton plant was assigned the duty of
identifying and recruiting members of the planning committee and chairing the committee
EPA QA/CS-2
March 2007
-------
meetings. Along with getting the commitment of committee members to participate, he
conducted several preparatory steps prior to the first meeting. He also identified other
stakeholders who might have an interest in the work of the planning committee and would be
either available for consultation or routinely updated on the committee's progress. A&B Inc.
hired an environmental consulting firm to provide technical expertise in designing and
executing the monitoring program and to prepare review documents.
The following sections review the types of preparations that the plant's chief
environmental engineer made for the planning committee's initial meeting. These activities
contribute toward Step 1 of the DQO Process.
2.1 Preliminary Documentation
The chief environmental engineer for the Emmerton plant prepared a packet of
preliminary documentation that was mailed to each member of the planning committee to review
prior to the first meeting. This packet included:
• A brief owner-supplied background narrative on the history of the plant, the need for
plant upgrades, the types of upgrades that were planned, and the expected impact of
these upgrades on emission sources;
• A conceptual model (including exposure scenarios) of particulate matter in ambient
air within the vicinity of Emmerton that is associated with plant emissions;
• Selected information gathered in the process of obtaining and holding the PSD permit
for construction (including an overview of the types of pre-construction ambient air
data collected and the results of air dispersion modeling);
• The State EPA's published PSD permit approval action;
• A summary of written comments received during the public comment period in
response to the State EPA's permit approval action (including those submitted by the
environmental citizens' group);
• A draft narrative describing the purpose and goals of the monitoring study;
• An overview of the DQO Process; and
• A draft of the problem statement, which the chief environmental engineer prepared
under Step 1 of the DQO Process. This statement read as follows:
A&B Inc. recently expanded its coal-burning power plant near the town of
Emmerton to better meet the needs of its customers. The plant received a PSD
permit from the State EPA that specified allow able emission rates following
expansion. As the expansion would increase the plant's coal combustion rate,
several members of the general public raised the concern about the lack of post-
construction monitoring in the permit process that could confirm whether the
plant continued to operate within the limits on PMw emissions established in the
permit. Thus, it is necessary to address this concern by verifyingparticulate
matter concentrations through post-construction emissions monitoring.
EPA QA/CS-2 6 March 2007
-------
2.2 Identification of Key Participants
A&B Inc. hired a consulting firm to conduct the study and designated one of its experts in
air particulate matter monitoring to be the principal investigator. The chief environmental
engineer determined that the planning committee should also include a regulator from the district
office of the State EPA who was knowledgeable in air quality regulations, and a member of the
environmental citizens' group whose concerns led to performing the study. The citizens' group
selected the leader of a local environmental organization to be its representative on the
committee. In addition, the consulting firm would provide an individual to take notes at each
meeting. Thus, the planning committee tasked to conduct the DQO Process consisted of the
following:
• A&B Inc.'s chief environmental engineer (committee chair);
• The study's principal investigator (an employee of the environmental consulting
firm);
• A regulator from the district office of the State EPA;
• A local environmental organization leader, representing the environmental citizens'
group.
The chair and principal investigator were also given the freedom to bring other members
of their organizations to participate in meetings as needed, such as a statistician, field sampling
technician, quality assurance specialist, and meteorologist.
2.3 Identification of Other Interested Parties
In addition to determining membership on the planning committee, the study's principal
investigator suggested that a list of potential stakeholders and other interested parties be
identified prior to initiation of the planning committee meetings. These stakeholders would not
be involved in the meetings to develop DQOs, but they would be given the opportunity to review
and comment on the final study plan. The organizations identified as potential stakeholders and
interested parties included:
• Emmerton public officials (mayor and town council), as well as officials from several
neighboring towns that are downwind of the power plant;
• County health department officials (to coordinate communication with the public
from a health perspective);
• Members of the Emmerton Senior Citizens' Organization; and
• Health or environment reporters from Emmerton's newspaper and from media in
neighboring cities.
2.4 Schedule of Meetings
Based on the principal investigator's past experience in implementing the DQO Process
on similar studies, it was expected that the planning committee would have to meet three or four
times to plan the study, with the meetings to be held in a conference room at the Emmerton
EPA QA/CS-2 7 March 2007
-------
power plant. The committee chair cleared the date for the first meeting with each of the
committee's members and provided the following agenda to the members for the first meeting:
1:00 Introduction of committee members
1:15 Background Material for Study
1:45 DQO Step 1: Define the Problem
2:30 DQO Step 2: Identify the Goals of the Study
3:15 Break
3:30 DQO Step 3: Identify Information Inputs
4:00 DQO Step 4: Define the Boundaries of the Study
4:30 Review of progress
4:45 Identify needs for next meeting
5:00 Adjourn
In providing the preliminary documentation packet to the planning committee members,
the chair of the committee suggested that the members be prepared to share their own ideas on
study goals and important issues that the committee should consider within this first meeting.
EPA QA/CS-2 8 March 2007
-------
3.0 FIRST MEETING OF PLANNING COMMITTEE
The first meeting of the planning committee for the post-construction PMio monitoring
study of the Emmerton power plant focused on initial execution of the first four steps of the
DQO Process (Figure 2):
Step 1: Define the Problem
Step 2: Identify the Goal of the Study
Step 3: Identify Information Inputs
Step 4: Define the Boundaries of the Study
The following sections document the discussions that were held in this first meeting and
the outputs from each of these four steps.
3.1 Step 1: Define the Problem
The primary focus of Step 1 is to assemble the planning committee, to prepare a problem
statement, and to examine available resources for investigating this problem.
The committee chair was able to begin work under Step 1 prior to the first committee
meeting. For instance, he compiled the list of planning team members and identified roles for
each member, including who would be making certain types of decisions. He also prepared
drafts of a conceptual model of particulate matter in ambient air within the vicinity of the plant
and of a draft problem statement associated with the monitoring program (Section 2.1) and
included them with the preliminary documentation given to the committee members for their
review before the first meeting. Therefore, discussion on Step 1 focused on agreeing on a
problem statement among the members, determining the scope of the study, and determining
initial information on the program's potential cost, duration, and technical needs. This
discussion featured the following:
• As the plant's chief environmental engineer, the committee chair presented the budget
allocated to the study by the plant's owner: $250,000.
• The principal investigator provided information on line-item costs associated with
conducting a monitoring study, including samplers, sample filters, sample collection,
laboratory analysis, and data analysis. While she was prepared to discuss various
monitoring options depending on the direction the committee was willing to take, her
presentation focused on stack testing to address the committee chair's initial draft
problem statement on measuring post-construction emissions (Section 2.1). The
principal investigator pointed out that stack testing would be less costly for the plant
to perform compared to other types of sampling, thereby allowing them to collect
additional data.
• The citizens' group representative asked whether it was possible to expand the study
beyond measuring particulate matter to also measuring other toxic chemicals that may
exist within the power plant's exhaust. Others argued that this was beyond the scope
EPA QA/CS-2 9 March 2007
-------
of what A&B Inc. had originally agreed to do and would require an additional
commitment of resources (including cost) from the company.
• On the basis of the discussions held, the committee chair decided to proceed to Step 2
of the DQO Process with the originally proposed problem statement (Section 2.1),
after supplementing it with the following statement:
The study is to involve stack testing that would be conducted over a one year
period within a total budget of $250,000.
3.2 Step 2: Identify the Goal of the Study
In Step 2, the committee uses the problem statement to identify a principal study question
and a statement of the study goals, and then considers potential alternative actions that
may be made upon answering this question and their implications. This leads to making
either a decision statement or an estimation statement, whichever is relevant to the
particular problem.
To initiate Step 2 of the DQO Process, the committee chair proposed the following as the
principal study question:
Using the results of stack testing as input to an air dispersion model, are post-construction
ambient PM levels projected to be higher than the values that are calculated by the air
dispersion model under the allowable PM emission rates?
As this and other follow-up study questions and study goals began to be discussed, a couple of
committee members began raising questions on whether stack testing would really be addressing
what citizens wished to learn from the collected data: that whether post-construction ambient air
levels in the vicinity of the power plant (where most of Emmerton's citizens lived) were not
appreciably different from pre-construction levels, and in particular, were within the standards
set to protect public health. This led to a debate, led by the environmental citizens' group
representative, on whether the committee was, in fact, addressing the right question. After all,
the plant's requirement to collect pre-construction ambient air samples yielded concentration
data against which post-construction levels could be compared. It soon became evident that
performing stack testing to verify that the emission rate limits specified in the PSD permit were
being achieved was not sufficient to fully address citizens' concerns that levels in ambient air
had been negatively affected. Thus, the committee decided that it needed to go back to Step 1 of
the DQO Process to redefine the problem they were trying to address.
3.3 Step 1 Revision: Define the Problem
The committee determined that the post-construction monitoring program needed to
perform ambient air monitoring, similar to (and perhaps at the same location from) the pre-
construction ambient air monitoring that the plant performed as part of its PSD permit
application. Therefore, they asked the principal investigator to resume her presentation on costs
associated with conducting a monitoring study, but she should now focus on the information she
EPAQA/CS-2 10 March 2007
-------
prepared relating to ambient air monitoring. For example, she addressed the ramifications of
using fixed versus mobile ambient air samplers on the study costs, as well as how costs
associated with sample filters, sample collection, laboratory analysis, and data analysis for
ambient air sampling may change from the stack testing approach. Her presentation implied that
under a $250,000 budget, several hundred ambient air samples could be collected using the same
sampler that was used to collect pre-construction ambient air samples, allowing the study to
extend longer than one year if necessary. If a new sampler were purchased and used, the number
of samples that could be collected within the available budget would be reduced to roughly 150.
In the ensuing discussion, the committee realized that the amount of time spent to collect
a fixed number of samples, whether it is one or two years, would have a somewhat minimal
effect on overall study costs. Rather, costs would be dominated most heavily by the number of
samples and number and location of samplers. While most group members preferred to limit the
monitoring effort to one year, the citizens' group representative lobbied to extend the study for
more than one year, as a one-year study would not be able to account for annual variation in
weather. While the committee agreed that a two-year sampling program could address some of
that concern, extending the sampling to two full years would likely not be acceptable to
stakeholders and others in the general public who were anxiously awaiting the outcome of the
monitoring study. The committee chair also noted that A&B Inc. still had an agreement in place
with the golf course that served as the location for pre-construction monitoring, allowing them to
use the same location for post-construction ambient air sampling, but the agreement would need
to be extended if the study were to extend for two years. There was also discussion about
whether meteorological variability that occurs within a single year would be a sufficient
surrogate for year-to-year variation, although no conclusion in that regard was reached.
Given the discussion and the principal investigator's presentation, the planning
committee worked together to revise the problem statement, in order to have it centered on
performing ambient air monitoring rather than stack testing, and giving some flexibility to extend
sampling to beyond a year if necessary. The result, agreeable to all on the committee, was the
following:
A&B Inc recently expanded its coal-burning power plant near the town ofEmmerton to better
meet the needs of its customers. The plant received a PSD permit from the State EPA prior to
construction. As the expansion would increase the plant's coal combustion rate, several
members of the general public raised the concern about the lack of post-construction
monitoring in the permit process that could confirm whether the increase in actual PMw
emissions from the expanded operation would result in unhealthy ambient concentrations.
Thus, it is necessary to address this concern by verifyingparticulate matter concentrations
through post-construction monitoring. The study is to be conducted in one year, if possible,
extending to no more than two years, and within a total budget of $250,000.
Before moving on to Step 2, the committee chair initiated a brief discussion on whether
additional expertise needed to be represented on the committee in order to complete the DQO
Process. The study's principal investigator noted that her consulting firm employed experts in
air sample collection and handling, air transport modelers, and statisticians to provide assistance
EPA QA/CS-2 11 March 2007
-------
in addressing areas where their expertise was needed. They would be called upon to provide
background reference material and participate in discussions when needed.
3.4 Step 2 Revision: Identify the Goal of the Study
Working with the revised problem statement, the committee members prepared a list of
specific questions that the study should be designed to answer. Two initial questions that
members suggested were the following:
• Are ambient PM levels higher than those considered safe by EPA?
• Are ambient PM levels higher after construction than before construction?
This led to drafting the following primary study question:
• Can it be verified that the allowable emissions from the power plant's upgrades result in
post-construction ambient particulate matter concentrations that are within acceptable
levels?
After further discussion, however, the statement "within acceptable levels" was considered too
vague. The committee made this question more specific and unanimously adopted it as the
primary study question:
Can it be verified that the allowable emissions from the power plant's upgrades result
in post-construction ambient particulate matter concentrations that are within
regulatory levels defined as protective of human health?
Considering the possible answers to this question, the committee identified the possible
alternative actions that could be taken based on the findings of this study:
• If the answer to this question is "Yes", then the study would conclude that PM levels are
within safe levels (as represented by the allowable PSD increment and the NAAQS).
This result would be reported to public health officials, other stakeholders, and the
general public through a press release to the local media.
• If the answer to this question is "No", then the study would conclude that the air
dispersion modeling, which was used in the permit application process to estimate the
impact that the upgrades may have on ambient air quality based upon the allowable
particulate matter emission rates, was not sufficiently accurate to ensure with high
confidence that the health of the citizens of Emmerton would be protected. The results
would be forwarded to the State EPA for further review, and they would determine an
appropriate course of action.
The problem statement and study questions led the committee to establish the following primary
study goal:
EPAQA/CS-2 12 March 2007
-------
Determine whether levels ofparticulate matter (PM) in post-construction ambient air
samples are considered hazardous to the health of the population ofEmmerton and its
surroundings, and therefore, require the plant owner to take additional action to reduce
emissions ofparticulate matter from the plant. Specific questions to be addressed are
• Are ambient PM concentrations greater than levels defined by the U.S. EPA
(through NAAQS standards) and the State EPA as being "safe " (i.e., protective of
human health)?
• Are ambient PM levels measured after construction higher than those measured
before construction, as well as what was expected by dispersion modeling?
PM concentrations in air samples will be collected once the upgrade is brought online,
and these concentrations will provide the scientific information that is necessary to
answer these questions.
Before the committee chair could move to Step 3 of the DQO Process, the principal
investigator and State EPA regulator noted that simply having the study questions and the study
goal refer to "paniculate matter" in ambient air was too vague. A discussion ensued as to
whether only fine particular matter less than 2.5 microns in diameter (PM^.s) should be
measured, versus PMio (i.e., total particulate matter less than 10 microns in diameter). The
citizens' group representative suggested that the type of particular matter known to have the
greater link to adverse health effects should be measured. The principal investigator noted that
permit levels were expressed relative to PMio, which includes PM2 5, and there is evidence from
several studies of a link between PMio and various respiratory problems in sensitive
subpopulations. Thus, the committee decided that the study should focus on measuring PMio,
and therefore, references to "PM" in the above study questions and the study goal were revised
to "PMio".
3.5 Step 3: Identify Information Inputs
Given the study goals and questions prepared under Step 2, the committee now begins to
identify the different types of information that are needed to answer these questions and
whether appropriate sampling and analytical methods are available to obtain this
information.
Based on the decision to focus ambient air monitoring on measuring PMio levels, the
committee identified the following necessary information inputs:
• To determine whether PMio concentrations are greater than "safe" levels, action
levels specified as "safe" by the U.S. EPA and State EPA would be needed. The
study's principal investigator noted that the NAAQS 24-hour standard of 150 |J,g/m3
for PMio is the appropriate standard to adopt as the action level, because it represents
EPAQA/CS-2 13 March 2007
-------
a level set by the U.S. EPA to provide protection of public health. The State EPA
regulator noted that the state has accepted this national level.
• To determine if PMio levels are higher after construction than before construction, it
would be necessary to obtain PMio concentration data collected before the
construction. These data were collected as required by the PSD permit and are
maintained by the plant's owner. Therefore, the plant's chief environmental engineer
agreed to provide the environmental consulting firm with the pre-construction
monitoring data.
• Post-construction PMio concentration data obviously do not exist, and therefore,
would need to be newly collected under this study. The committee decided that the
study will follow the U.S. EPA requirements for monitoring of PMio concentrations
(40 CFR Part 58). The principal investigator was charged with reviewing these
requirements and providing the committee with recommendations for sampling and
analysis approaches at the next meeting.
• A discussion was held among committee members on the need for generating and
interpreting predictions from air transport models, because modeling had been used
within the AIQA to generate allowable emission levels. If such prediction data were
needed, there would be an implicit need to collect meteorological data for use as input
to the model. The committee concluded that computer modeling would not be
needed to generate information to address the study goals and questions.
• The committee discussed the extent to which measured PMio ambient concentrations
could possibly be affected by emissions from other sources in the area. The principal
investigator noted that pre-construction monitoring data were used as input to the
modeling that evaluated allowable PMio emission rates for the plant's upgrades. Any
contributing sources to ambient PMio levels would be assumed to impact both pre-
and post-construction levels. Therefore, expected effects associated with the plant's
permit levels, which were assessed based on pre-construction ambient air levels and
the allowable emission rates associated with the upgrades, would already have been
adjusted for other PMio sources. Thus, any observed increases in PMio
concentrations at the sampling location (i.e., the golf course site) should be the
exclusive result of the plant's modification. It was suggested, however, that
meteorological data would assist in the analysis and interpretation of the PMio
concentration data should any questions arise about the source of PMio present in
ambient air at the monitoring site.
EPAQA/CS-2 14 March 2007
-------
3.6 Step 4: Define the Boundaries of the Study
In addressing Step 4, the committee addresses how the target population should be
defined, along with determining the geographic (spatial) and temporal boundaries
associated with the population, and whether any practical constraints exist to collecting
data.
• The committee began by discussing the consequences of using only the single
sampler that the plant used in pre-construction monitoring. Using this sampler for the
post-construction monitoring would facilitate the interpretation of comparisons
between pre- and post-construction measurements. However, committee members
noted that a single sampler provides limited spatial coverage and questioned how the
representativeness of the site to other locations in Emmerton would be assessed. In
response, the principal investigator noted that variations in wind direction will occur
at the fixed location, which would simulate greater spatial coverage. Also, the
committee reviewed the plan and results from the pre-construction monitoring
analysis distributed by the committee chair, which included the State's detailed
assessment of the representativeness of the single site for that analysis. Based on this
information, the committee members tentatively agreed to the use of a single sampler,
pending further consideration of the benefits and limitations of using a fixed-location
sampler versus a mobile sampler.
• The committee defined the target population to consist of all possible air samples that
could be collected from the sampler's location (at the nearby golf course) during the
period of time that the post-construction monitoring study will cover.
• The committee specified in the problem statement that the study should be completed
in less than two years, and within a budget not exceeding $250,000. The principal
investigator will determine how this budget should be divided among several tasks
within the study, such as project management, planning, field sampling, laboratory
analysis, and data analysis.
• Some committee members raised questions about whether the quality of the pre-
construction sampling data would be sufficient to allow comparisons to be made to
post-construction. The plant's chief environmental engineer (committee chair) will
investigate the quality criteria that were placed on the pre-construction data and the
extent to which these criteria were achieved, and will share this information with the
committee in the next meeting
The output prepared under Step 4 in this first meeting was as follows:
EPA QA/CS-2 15 March 2007
-------
The target population consists of all possible samples that might be collected during post-
construction from the sampling location used for the pre-construction monitoring. The
spatial boundaries of the study consist of the single location, which represents all points the
same distance from the source within the cone defined by the prevailing winds. The
temporal boundaries of the study are from the end of construction until the end of the post-
construction monitoring program.
3.7 Identifying Information Needed for Subsequent Steps
Prior to adjourning the first meeting, the committee chair set a date for the second
meeting that would work in the schedule of all members of the committee. The chair then listed
a series of action items for committee members that will provide additional important
information for use in the next meeting. These action items were as follows:
• The principal investigator will explore alternative sampling and analysis methods for
use in the post-construction monitoring study.
• In order to better determine whether a mobile sampler would be feasible to consider
as an alternative to a fixed-location sampler, the principal investigator will provide
some initial estimates for the number of samples that could be collected within a
budget of $250,000, first assuming a single fixed sampler is used, and then assuming
a mobile sampler is used.
• Steps 6 and 7 of the DQO Process will require some expertise in statistical sampling
design. Thus, upon request by the committee chair, the principal investigator will
arrange to have a statistician assigned to the planning activities. This statistician will
review the outcome of the first meeting and will plan to participate in the second
meeting.
• The State EPA representative will provide additional information on regulatory
guidelines concerning PMi0.
• The plant's chief environmental engineer (committee chair) will provide information
(e.g., quality criteria, summary) on the pre-construction monitoring data.
• The committee chair, with the assistance of the principal investigator's firm, will
compile notes of the first meeting and propose an agenda for the second meeting
within seven days of the first meeting. Each committee member will review the notes
for accuracy.
EPAQA/CS-2 16 March 2007
-------
4.0 INTERIM ASSIGNMENTS PRIOR TO THE SECOND MEETING
The action items resulting from the first meeting (Section 3.7) required certain committee
members to do some additional research work and to gather information that all members would
review and discuss in the second meeting. For example, minutes of the first meeting would be
prepared, distributed, and reviewed prior to the second meeting. Other assignments given to
specific committee members and performed before the start of the second meeting are discussed
in the following sections.
4.1 Reviewing Sampling and Analysis Methods
The principal investigator was asked to review the PMi0 sampling and analysis methods
used for pre-construction monitoring and to determine if post-construction monitoring should
involve the same or a different method. Ambient air monitoring requirements for criteria
pollutants, including PMio, are available in 40 CFR Part 58; the method provides for the
measurement of the mass concentration of PMi0. Particles are inertially size-selected by a
specially shaped inlet and collected on a filter over the 24-hour sampling period. The filter is
weighed, and the net mass gain due to the collected PM is used to calculate the PMio
concentration. The filters could then be saved for subsequent physical or chemical analysis. The
committee had already agreed that it would be appropriate to follow the methods and
requirements thatA&B Inc. used for the pre-construction monitoring.
4.2 Determining Numbers of Samples for a Fixed Cost, Considering
a Fixed Sampler Versus a Mobile Sampler
The principal investigator was also asked to provide initial estimates of the number of
samples that could be collected for a cost of $250,000 under two scenarios:
• Using the sampler that collected pre-construction monitoring samples, at the same
(fixed) location as in the pre-construction monitoring; and
• Using a mobile air sampler.
Upon gathering information on laboratory analysis costs, equipment costs, and operating
costs, she concluded that for a fixed cost of $250,000, approximately 500 samples could be
collected using the fixed sampler, and approximately 100 samples could be collected using the
mobile sampler. The number of samples is lower for the mobile sampler due to the cost required
to obtain the sampler and the higher operational cost associated with moving the sampler.
4.3 Investigating Use of Existing Data
The plant's chief environmental engineer was asked to provide a summary of the pre-
construction PMio data thatA&B Inc. collected during the permitting process. He gathered this
information from the data summaries that were included in the permitting report that A&B Inc.
supplied to EPA in support of the plant upgrade. These summaries included:
EPAQA/CS-2 17 March 2007
-------
• the starting and ending sampling dates;
• the number of samples collected during that period;
• the minimum, maximum, median, and mean PMio concentrations;
• the standard deviation of the PMio concentrations; and
• a plot of the PMio concentrations over time that shows trends and any unusual
measurements.
In addition, the chief environmental engineer was asked to provide information on the
quality assessment of the pre-construction data and its analysis, including all quality assurance
requirements and summaries. The committee members would use this information to determine
whether the pre-construction data were of sufficient quality to warrant the data's use in the post-
construction investigation. Also, in his capacity as committee chair, the chief environmental
engineer considered an initial list of possible alternatives to propose to the committee if the
quality assessment led the committee to conclude that the pre-construction data should not be
accepted for use in this investigation.
4.4 Identifying Regulatory Guidelines
The regulator from the State EPA was asked to provide details on State regulations
concerning PMio. He worked with staff in his office to obtain information on the State
Implementation Plan for PMio and found that it was in agreement with the national regulatory
standard promulgated by the U.S. EPA.
EPA QA/CS-2 1 g March 2007
-------
5.0 SECOND MEETING OF PLANNING COMMITTEE
The second meeting of the planning committee was held about two weeks after the first
meeting. The first item on the agenda was to review and refine the outcome of Steps 1 through 4
of the DQO Process (as determined from the first meeting). Then, the agenda featured initial
discussion on Steps 5 through 7 (Figure 2):
Step 5: Develop the analytic approach
Step 6: Specify acceptance or performance criteria
Step 7: Develop the detailed plan for obtaining data
The participants of the second meeting included all of the original planning committee members,
along with a statistician from the principal investigator's environmental consulting firm who had
reviewed the committee's work to date and was prepared to address statistical-related issues that
were expected to be encountered in Steps 6 and 7.
5.1 Discussion of the Findings of Interim Assignments
The first item on the agenda of the second meeting was to review the findings of the
interim assignments made in the first meeting. This was done by revisiting each of Steps 2
through 4 of the DQO Process to determine if the additional information should lead to making
revisions or more specifics to the outcome of these steps. The following subsections highlight
the discussions and outcome of this review.
5.1.1 Step 2 Revision: Identify the Goal of the Study
• The State EPA regulator proposed that there should be less of an emphasis on
comparing post-construction PMio concentrations, to be collected in this new study,
with the existing pre-construction monitoring levels. Rather, the study should focus
on determining whether post-construction levels were below a specified level, such as
a health-based level. Upon further consideration and discussion, the committee
agreed to this change in focus.
• A follow-on discussion addressed whether the primary interest should be 1) verifying
that post-construction PMio concentrations meet health-based levels, or 2) the level
implied by the permit (i.e., the PSD increment) was not exceeded. The State EPA
regulator noted that the PSD increment was lower (i.e., more stringent) than the
NAAQS. The citizens' group representative thus argued for considering the PSD
increment as the threshold to which post-construction PMio concentrations would be
compared. However, the remainder of the committee argued that ambient air samples
should be collected assuming comparison to PSD.
• The specific PMio measurement techniques had not been specified in the first
meeting. The principal investigator suggested that 24-hour ambient air samples
should be collected and analyzed, because the NAAQS assumes a 24-hour sampling
EPAQA/CS-2 19 March 2007
-------
period. This would result in daily average PMio concentrations being measured and
reported from post-construction monitoring. The committee was in agreement.
Based on these discussions, the committee chair proposed the following revision to the primary
goals statement, which was approved by the committee:
Determine whether levels of paniculate matter (PM10) in 24-hour post-construction ambient
air samples exceed the levels dictated by PSD regulations and used to establish emission
limits in the permit, and therefore, require the plant owner to take additional action to
determine if the emissions of paniculate matter from the plant are contributing to a violation
of the NAAQS in ambient air. The specific question to be addressed is:
• Are PMw concentrations greater than the levels defined in PSD regulations?
5.1.2 Step 3 Revision: Identify Information Inputs
• With the revision made to the study's primary goal in Step 2, the committee requested
additional information on the AQIA used to establish the allowable PMio emission
limits for the upgrade. The committee chair noted that the AQIA was based on:
o the estimated average 24-hour PMio ambient air concentration observed by pre-
construction monitoring (80 |j,g/m3);
o the plant's estimate of the post-construction PMio emission rate;
o the expected post-construction average 24-hour PMio ambient air concentration,
as calculated by dispersion modeling during the permit process (89 ng/m3),
o the maximum PMio level in ambient air considered to be protective of human
health (the NAAQS primary 24-hour PMio standard of 150 ng/m3), and
o the allowable PSD increment associated with the upgrade (15 ng/m3).
• The chief environmental engineer for the power plant presented information on the
pre-construction data summary and quality assessment. He noted that the AQIA
depended on the pre-construction data solely through its average (80 |j,g/m3). The
committee determined that this average was based upon data of sufficient quality to
be used in this effort, and the entire pre-construction database would not be required.
• In reviewing the results of Step 3 from the first meeting, the committee concluded
that computer simulation modeling would be unnecessary to address the primary
study goals and questions. However, meteorological data may still be useful to
acquire, particularly in helping to interpret the PMio concentration data to be
collected, such as explaining extreme measurements or apparent outliers. The
meteorological data could also be compared between pre- and post-construction
monitoring periods to determine the extent to which data collected within the two
periods are comparable. Several meteorological parameters that could be of
EPA QA/CS-2 20 March 2007
-------
importance were average wind direction and speed, total precipitation, and average
temperature.
5.1.3 Step 4 Revision: Define the Boundaries of the Study
• The principal investigator proposed that because the AQIA utilized data from pre-
construction monitoring, and the threshold to which post-construction monitoring
data would be compared would depend in part on the outcome of pre-construction
monitoring, the post-construction data should be collected in as similar a manner as
possible to the pre-construction data. In particular, she recommended that the
monitoring study use the same fixed pre-construction location for post-construction
monitoring. The committee chair noted that the plant owner had made prior
arrangements with the owner of the site (the golf course) to continue ambient air
monitoring from this site for the foreseeable future.
• While the committee agreed to conduct post-construction ambient air monitoring at
the golf course site, the debate continued on whether monitoring should be performed
at other sites as well. In particular, some committee members raised the concern of a
lack of spatial variability in the data if monitoring occurred only at a single site. The
principal investigator noted that a single site would be much easier to manage within
the study's available budget, and in particular, fewer samples could be collected if
multiple sites were considered, in part due to the additional labor and expense
associated with maintaining multiple samplers and multiple sites. Furthermore, the
principal investigator and her statistician noted that spatial variability would be
represented within the sample results by variability attributable to meteorological
conditions (i.e., wind speed and direction). Recognizing that a sampling approach
that featured the single golf course site would be most cost effective, the committee
agreed to proceed with this approach.
• The committee reviewed the definition of the target population that was drafted in the
last meeting. While actual post-construction sampling would only occur over a finite
period of time, the committee did not want to limit the target population to only those
samples that could be collected during the study. Thus, the definition of the target
population was slightly revised to be all ambient air samples that could be collected at
the site under those conditions that exist over the duration of post-construction
monitoring (i.e., 12 to 24 months), for as long as those conditions remain in effect
(i.e., until the various contributors to PMi0 in ambient air at this site change in some
way, such as the addition of a PMio source or a change in emission rates from
existing sources).
From these discussions, the revised output from Step 4 was as follows:
EPAQA/CS-2 21 March 2007
-------
The target population consists of all possible 24-hour ambient air samples that might be
collected from the sampling location used for pre-construction monitoring, during the period of
time in which conditions that can affect PM10 levels in ambient air are unchanged from those
present during the first 12-24 months following construction. The spatial boundaries of the
study consist of the single location, which represents all points the same distance from the
source within the cone defined by the prevailing winds. The temporal boundaries of the study
are from the end of construction until the end of the post-construction monitoring program.
5.2 Step 5: Develop the Analytic Approach
In developing Step 5, the committee addresses the methods that will be used to draw
conclusions from the study results that address the specific objectives of the study,
including specifying the population parameters that will be examined and creating a
decision rule or estimator based on the population parameters and the data collected.
At this point in the second meeting, the committee shifted from reviewing and revising
the outputs from the first meeting to addressing the final three steps of the DQO Process. In
addressing Step 5, the committee accomplished the following:
• The principal investigator presented the findings of her review of PMi0 sampling and
analysis methods to be used. She noted that the methods used by A&B Inc. for pre-
construction monitoring met the requirements described in 40 CFR Part 58, and
therefore, the same approaches should be used in this study. She also discussed
methods for characterizing the chemical make-up of the particles in the samples, for
use in verifying that the PMio in the collected sample filters actually originated from
plant emissions. The committee agreed that additional research into chemical
analysis of the collected PMio, focusing on fingerprinting coal combustion sources,
should be investigated only if the results could show that the PSD increment was
exceeded more than once per year.
• A discussion on the appropriate population parameters to consider began by noting
that the study would be measuring average 24-hour post-construction PMio
concentrations in ambient air. Some committee members were unclear on exactly
how the average concentration could be used to verify the assumptions used in the
AQIA, and ultimately, to determine the appropriate action that the plant may need to
take. The State EPA regulator explained that the draft permit - like all similar PSD
permits for PMio that are issued in situations such as this - specifies that the plant
would be in compliance with the 24-hour PSD increment, as indicated by the air
dispersion model results, as long as the 24-hour post-construction PMio concentration
in ambient air did not exceed the increment more than once per year.
• As a result of this explanation on how to determine compliance with the 24-hour
PMio PSD increment, the citizens' group representative suggested that the parameter
of interest should not be the average concentration, but instead, the observed
EPA QA/CS-2 22 March 2007
-------
proportion of days in a given year during the monitoring period that the daily average
concentration exceeded the PSD increment. Of interest would be whether or not this
observed proportion exceeds 1/365 = 0.00274 (i.e., the maximum proportion of days
in a year that the PSD increment could allowably be exceeded). The statistician
pointed out that this approach would require that 24-hour samples be collected over a
large number of days, such as daily for one to two years.
• The committee members agreed that the proportion of days exceeding the PSD
increment would be an appropriate measure for determining compliance. They asked
the statistician whether a decision on compliance could be made by performing a
statistical hypothesis test rather than simply noting whether or not the observed
proportion of days exceeds 0.00274. The statistician suggested an approach that was
based on calculating tolerance intervals, or confidence intervals placed on percentiles
of a probability distribution. He suggested that a tolerance interval could be
calculated for the (1-0.00274) x 100 = 99.726th percentile of the distribution of 24-
hour PMio sample concentrations at the fixed site. The average pre-construction
PMio concentration (80 |J,g/m3) would be subtracted from the upper bound of this
tolerance interval, as calculated from the actual sample results.
o If this result is less than the PSD increment, then it would be reasonable to expect
(with a specified degree of statistical confidence) that under current post-
construction conditions, daily average PMio levels in ambient air would exceed
the PSD increment no more than one day per year, on average.
o If this result exceeds the PSD increment, then under current post-construction
conditions, daily average PMio levels in ambient air are expected to exceed the
PSD increment more than one day per year, on average, implying that sufficient
evidence exists that the assumptions used for the AQIA did not accurately
represent the upgrades to the power plant.
The committee agreed that the parameter of interest would be the 99.726th percentile
of the distribution of 24-hour PMio concentrations, and that a decision rule would be
based on calculated tolerance intervals for this percentile.
As a result of the suggestions and discussion made in support of Step 5, the committee
prepared the following statement:
The parameter of interest is the 99.726th percentile of the distribution of 24-hour post-
construction PMw concentrations. The type of statistical inference to be used to address the
study questions is a tolerance interval associated with this percentile. If the upper bound on
this tolerance interval calculated from the actual monitoring data, minus the average pre-
construction PMw concentration, exceeds the PSD increment, then decide that the AQIA did
not accurately represent the upgrades to the power. If the upper bound is less than the PSD
increment, then decide that the plant is in compliance with the PSD regulations.
EPA QA/CS-2 23 March 2007
-------
5.3 Step 6: Specify Acceptance or Performance Criteria
In this step, the committee will establish "performance or acceptance criteria " that the
post-construction monitoring data need to achieve in order to achieve their intended use in
the approach specified within Step 5. The committee now accepts the fact that that the
collected data will represent one of many possible outcomes from the (infinite) target
population, and as a result, must understand the various types of "random error " (or
uncertainty) to which the data will be subject, such as errors due to sampling and analysis
techniques. As some degree of error is inevitable in this process, the committee needs to
agree upon how much random error is "acceptable, " relative to the extent that this error
may lead to making erroneous conclusions upon analysis and interpretation of the
collected data.
In beginning efforts under Step 6, the principal investigator and her statistician explained
the concept of acceptance and performance criteria. As noted in the U.S. EPA's Systematic
Planning using the Data Quality Objectives Process (EPA QA/G-4J (U.S. EPA 2006a),
performance criteria represent the full set of specifications that are needed to design a data or
information collection effort such that, when implemented, will generate newly-collected data
that are of sufficient quality and quantity to address the project's goals (determined from Step 2).
Acceptance criteria are specifications intended to evaluate the adequacy of one or more existing
sources of information or data as being acceptable to support the project's intended use. As this
study will be collecting new post-construction ambient air monitoring data, the committee will
focus on establishing performance criteria that these new data will need to achieve in order to be
used to determine compliance with the established PSD increment. These criteria are then
adopted as the study's Data Quality Objectives (DQOs).
The discussion to initiate Step 6 featured the following:
• Given that the committee agreed in Step 5 to take the tolerance interval approach to
determining compliance, the committee's next task was to determine the level of
confidence that they would be willing to accept in ensuring that the tolerance interval
correctly contained the 99.726th percentile of the distribution. The citizens' group
representative argued vehemently for 100% confidence. However, the statistician
convinced him that 100% confidence was impossible to achieve, due to inherent
variability present in the sampling and measurement process. Furthermore, the
statistician warned that extremely high confidence levels (e.g., 99.5%) would require
very large numbers of samples. After further discussion, the committee agreed that a
confidence level of 95% would serve as a reasonable value for this performance
criterion, as this level is routinely used in similar efforts.
• The statistician pointed out that the DQO Guidance also required a second
performance criterion on the width of the tolerance interval, which is an indicator of
the degree of acceptable uncertainty in estimating the 99.726th percentile. The
citizens' group representative demanded that it be as short as possible - perhaps on
the order of 10 ng/m3 - so that a clear idea of the ambient PMi0 levels would be
EPA QA/CS-2 24 March 2007
-------
available. The statistician requested that the discussion of tolerance interval width be
tabled because he needed more time to research the appropriate methodology before
making recommendations. The committee agreed and asked the statistician to
provide additional guidance at the next meeting regarding methods for calculating
tolerance intervals and their associated performance criteria.
Thus, the committee chair determined that Step 6 could not be completed in this meeting, and
appropriate action items would be assigned to provide additional information that is needed to
complete this step in the next meeting.
5.4 Step 7: Develop the Detailed Plan for Obtaining Data
In this final step of the DQO Process, the committee will decide upon a sampling and
analysis design that will generate the post-construction data that will achieve the
performance criteria developed in Steps 1 through 6 andean be implemented within the
available budget.
The committee realized that the sampling design requires a specification of the sample
size, which is pending upon finalizing the performance criteria in Step 6. Thus, various design
alternatives, and selection of the final design, cannot be reviewed and agreed upon until the next
meeting. However, based on discussions in prior steps, the committee did agree on several
features of the sampling plan:
• 24-hour ambient air samples would be collected to obtain daily average
concentrations;
• The same sampler used in the pre-construction monitoring would be used, and it
would continue to be located at the golf course;
• Samples will be collected daily for at least one year, but no more than two years.
Rather than wait until the next meeting to clarify performance criteria, the statistician was
asked, with the help of the principal investigator, to provide some candidate sample designs for
the committee to consider at the next meeting. If none were acceptable to the committee, then
further guidance could be given to the environmental consulting firm on preparing other options,
and another meeting would be required to approve the final design.
5.5 Identifying Areas for Further Review and Additional Information Needs
Prior to adjourning the second meeting, the committee chair set a date for the third
meeting that would work in the schedule of all members of the committee. The chair then
established the following action items for committee members, to be completed by the next
meeting, that will provide additional important information for use in that meeting:
• All members will review the current outputs from each step of the DQO Process and
determine the need for further revision.
EPA QA/CS-2 25 March 2007
-------
The statistician will provide more details on the tolerance interval approach,
including methodology and related performance criteria.
The statistician and principal investigator will prepare candidate sample designs
based on the use of a 95% tolerance interval.
The principal investigator would report on costs associated with sampling and
analysis.
The committee chair, with the assistance of the principal investigator's firm, will
compile notes of the second meeting and propose an agenda for the third meeting
within seven days of the second meeting. Each committee member will review the
notes for accuracy.
EPA QA/CS-2 26 March 2007
-------
6.0 INTERIM ASSIGNMENTS PRIOR TO THE THIRD MEETING
Between the second and third meetings of the planning committee, selected committee
members investigated the topics introduced in the sections that follow.
6.1 Performance Criteria for Tolerance Intervals
As requested by the committee, the statistician researched procedures for calculating
tolerance intervals, as well as proper interpretation of tolerance intervals, in order to determine
how appropriate performance criteria can be established, and therefore, how sample sizes would
be determined based on these criteria. He determined that in this situation, the following two
performance criteria were relevant:
• The tolerance interval should contain the true 99.726th percentile of the distribution
with high probability; and
• The tolerance interval should contain a specified percentage (to be determined) of the
distribution that is higher than the 99.726th percentile of the distribution with low
probability. For example, the probability that the tolerance interval actually contains
the 99.9th percentile should be small.
The first performance criterion ensures that the goal of containing the specified percentage is
achieved, while the second criterion specifies that the goal need not be exceeded by an unusually
large margin. In other words, the sampling design must specify that a sufficient number of
samples will be collected to allow the first performance criterion to be met, but the sample size
should not be so unnecessarily large as to overshoot this goal by a large margin.
The planning committee would be tasked with concurring on versions of the above
performance criteria in which numeric probabilities are specified rather than simply "high" and
"low." If the collected data satisfy the criteria established by the committee, then the plant
would be considered to have demonstrated compliance with the PSD increment.
6.2 Options for Sample Sizes
In estimating the number of samples required to achieve the performance criteria given
above, the statistician considered a series of different values for the probabilities specified within
these criteria, then calculated the sample sizes required for each set. For example, the statistician
considered the following specific performance criteria, which he planned to propose to the
committee:
• There should be a 95% chance that the collected data will (correctly) demonstrate that
99.726% of the population falls below the upper bound of the tolerance interval.
• There should be a 5% chance that the collected data will demonstrate that 99.9% of
the population falls below the upper bound of the tolerance interval.
EPA QA/CS-2 27 March 2007
-------
His statistical analysis concluded that 588 samples would be required to meet both of these
performance criteria. For the sample sizes calculated by the statistician, the principal
investigator estimated the associated sampling and analysis costs, which she planned to present
at the next meeting. She noted that in many cases, including the example above, the projected
costs exceeded the budgeted amount for the study. After discussions with representatives of the
plant owners, it was decided that an initial recommendation to reduce sample sizes may be to
increase the probability underlined in the second criterion.
EPA QA/CS-2 28 March 2007
-------
7.0 THIRD MEETING OF PLANNING COMMITTEE
The goal of the third meeting was for the planning committee to arrive at a final set of
DQOs and a final sampling design. The statistician also attended this meeting to provide input
on statistical issues. The agenda called for a review of outputs for all seven steps of the DQO
Process. The sections that follow document the discussions that were held, with the final outputs
from each step given at the end.
7.1 Discussion of the Findings of Interim Assignments
7.1.1 Finalizing Steps 1 through 4
The committee began this meeting by reviewing the most recent outputs from Steps 1
through 4 and accepting these outputs as written. Thus, there was no need to further consider
these steps, and their outputs could be considered final.
7.1.2 Step 5 Revision: Develop the Analytic Approach
There was additional discussion about the analytic approach that was adopted in the
previous meeting within Step 5. Given the 99.726th percentile was adopted as the primary
parameter of interest, some committee members had some questions about the calculation of a
tolerance interval for this percentile, including whether alternative methods may be available that
are more straightforward or "mainstream." The committee chair tabled the discussion until after
the statistician presented his information on the tolerance interval methodology. At that time, it
would be determined whether iteration back to Step 5 would be needed to redefine the analytic
approach.
7.1.3 Example Performance Criteria, Corresponding Sample Size, and Budget
Implications
• The statistician presented the example performance criteria introduced in Section 6.2
to illustrate the application of tolerance interval method to the problem. This
example was as follows:
o There should be a 95% chance (0.95 probability) that the collected data will
(correctly) demonstrate that 99.726% of the population falls below the upper
bound of the tolerance interval.
o There should be a 5% chance (0.05 probability) that the collected data will
demonstrate that 99.9% of the population falls below the upper bound of the
tolerance interval.
His presentation included a detailed discussion of the two performance criteria and
justification for their relevance. Using this example as a basis for discussion, the
committee was able to better understand the underlying principles of the tolerance
EPA QA/CS-2 29 March 2007
-------
interval methodology, but the citizens' group representative continued to voice
concern about whether the general public would understand this approach.
• The statistician noted that in order to meet these two example performance criteria,
the sampling design would need to specify a minimum of 588 samples to collect.
• The principal investigator gave a report on expected rates associated with sampling
and analysis. Labor costs for field technicians could range from $40 to $70 per hour,
sample filters would cost about $40/filter for the sampler used by the plant to conduct
pre-construction monitoring, and laboratory analysis costs associated with these
filters (including labor) were approximately $65/sample.
• The principal investigator noted that under these assumed rates, collecting enough
samples to yield 588 valid 24-hour averages would require more budget than what
was available for the study. Thus, she pointed out that above example performance
criteria would need to be revised to be less stringent, so that fewer samples would be
required, and therefore, the study could be completed within budget. The citizens'
group representative argued that the confidence level of 95% in the first criterion
should not be made less stringent, in order to sufficiently protect human health.
7.2 Specific Sample Collection Design and Alternatives
• As part of his interim assignment, the statistician calculated minimum sample sizes
for different sets of values for the two percentages underlined in the example criteria
given above. He summarized these sample sizes in Table 1 and presented this table
to the committee. Note that the sample size for the example criteria (588) is specified
in the row labeled 0.05 and the column labeled 0.95.
Table 1. Relationship Between Tolerance Interval Performance Criteria and Minimum
Sample Size
Probabilities
within the
Second
Performance
Criterion2
0.25
0.20
0.15
0.10
0.05
0.01
Probabilities within the First Performance Criterion1
0.75
100
125
158
205
285
472
0.80
127
155
191
243
330
539
0.85
162
194
234
291
385
599
0.90
213
249
295
357
462
694
0.95
301
343
397
469
588
846
0.99
506
561
629
719
865
>1000
1 Probability of correctly concluding that 99.726% of the population falls below the upper bound of the tolerance
interval.
2 Probability of concluding that 99.9% of the population falls below the upper bound of the tolerance interval.
EPA QA/CS-2
30
March 2007
-------
The committee reviewed and evaluated the different sample sizes in Table 1,
considering which sample sizes would lead to performance criteria that could be
considered acceptable, and could also be collected within the available budget (based
upon the cost estimates which the principal investigator provided earlier in the
meeting). The committee chair reminded the committee that in an earlier meeting
they agreed that the monitoring program needed to be less than two years in duration,
with a one-year duration being preferable, to ensure that results could be reported
back to the public within an acceptable period of time.
The citizens' group representative reminded the committee of concerns about being
able to account for variations in weather from year to year and noted that the problem
statement offered the possibility of extending the sampling into a second year. The
committee agreed that up to 18 months of sampling would be acceptable in keeping
the study to within a two-year time frame, as the laboratory can still analyze the
collected samples, the collected data could be statistically analyzed, and a final report
prepared, within this time frame. This would also allow for some year-to-year
variability due to weather to be addressed. If daily sampling occurred seven days per
week, an 18-month sampling period would yield approximately 550 samples. She
also reiterated her objection to any confidence levels within the first performance
criterion being below 95%, and therefore, would only support sample sizes selected
from one of the last two columns of Table 1.
The principal investigator noted, however, that overtime costs associated with
sampling on weekends and holidays may not be acceptable within the study budget,
and therefore, it may be necessary to limit sampling only to business days (e.g., 5
days per week). If this occurred, only about 380 samples could be collected in an 18-
month period.
The principal investigator also mentioned that it was unlikely that 100% of the
samples would yield valid results, due to various problems that could occur with air
sampling and sample collection and analysis. In fact, only about 75% of collected
samples were predicted to yield valid data. Thus, the maximum number of valid data
points that could be obtained in an 18-month period was about 285 if sampling
occurred only on business days, and about 400 if sampling occurred seven days per
week.
Given the principal investigator's presentation on sampling and analysis costs, the
committee determined that the study budget could afford paying the extra labor cost
associated with daily sampling for seven days per week (i.e., including sampling on
weekends and holidays) in order to accelerate the sampling period.
The principal investigator noted that she had held discussions with representatives of
the plant owners in advance of the meeting, and they appeared to be willing to
increase the percentage specified in the second performance criterion in order to meet
the budget.
EPA QA/CS-2 31 March 2007
-------
7.3 Completion of Steps 6 and 7: Selection of Final Sample Design
Using the tolerance interval methodology, the committee decided that 24-hour ambient
air samples would be collected using the same sampler that was used for pre-construction
monitoring. According to Table 1, the committee noted that approximately 400 data points
would be needed to achieve the following performance criteria:
There should be a 95% chance that the collected data will (correctly) conclude that
99.726% of the population falls below the upper bound of the tolerance interval;
and
There should be a 15% chance that the collected data will conclude that 99.9% of
the population falls below the upper bound of the tolerance interval
(Note that these criteria differ from the example criteria specified by the statistician only in the
percentage given in the second criterion, which increased from 5% to 15%.) Upon further
discussion, all members of the committee agreed that these appeared to be acceptable
performance criteria. At that time, while the committee members took a break, the committee
chair and principal investigator held a brief conference call with representatives of the plant's
owner to present the pros and cons associated with these criteria and to determine if they would
be acceptable. The plant's owner agreed that, given the endorsement of the planning committee
and the ability to meet the original specified budget, these criteria would be acceptable.
If daily monitoring were to occur seven days per week, and assuming that 25% of
samples would yield results that were unusable or invalid, then it would take 18 months to
collect enough daily samples to yield 400 valid data points (i.e., 550 samples would be collected
and this would correspond to the 397 valid samples needed to satisfy the 15% criterion). Given
the principal investigator's presentation of sampling and analytical costs, the costs associated
with collecting and analyzing 550 samples were approximately $95,000, which was within the
portion of the allocated budget. Therefore, the committee agreed that taking 550 samples over
an 18-month time frame, expected to yield a minimum of 400 valid sample points, would be
acceptable from both a time and cost standpoint.
7.4 Review of Final DQOs
Upon the conclusion of the systematic planning process, the planning committee released
the final set of outputs on the DQO Process that is provided in Table 2.
As noted in Section 1 and throughout this particular case study, efforts to complete the
DQO Process can often involve iterating between its various steps, where information gathered
in later steps of the process leads to revisiting earlier steps and revising their output. Figure 3
illustrates the degree of iteration in the DQO Process that occurred by the planning committee in
this case study. The freedom to iterate among steps as needed is one feature of the flexibility
that is inherent in the DQO Process.
EPAQA/CS-2 32 March 2007
-------
Table 2. DQOs for the Post-Construction Monitoring Study of PMi0 Concentrations
in Ambient Air in the Vicinity of the Emmerton Power Plant
DQO Step
Statement
1. Define the
Problem
The power plant in Emmerton is planning an expansion and is going through the
permitting process to operate the updated facility. During a public hearing, a
citizens' action group voiced concern that the permitting process does not require
post-construction monitoring to verify that the plant's emissions do not result in
unacceptably high ambient concentrations of particulate matter. The plant owner,
A&B Inc., has agreed to conduct post-construction monitoring of ambient particulate
matter (specifically, PM10) concentrations to assuage the group's concerns that
unhealthy levels of ambient PMi0 within city limits may result upon implementing the
plant upgrades. Daily monitoring data will be needed that can be used to isolate the
effect of plant emissions on ambient PM10 concentrations.
The planning committee for the study consists of the following individuals:
• the chief environmental engineer from the power plant (chair);
• an expert in air sampling from a firm hired by A&B Inc. to perform post-
construction monitoring (principal investigator);
• an environmental regulator from the State EPA District Office; and
• a local environmental organization leader, representing the environmental
citizens' group.
A&B Inc., has set a total budget for the monitoring program at $250,000. Initial
estimates are that the monitoring program will last between one and two years, with
an additional 6 months for data compilation, analysis, and reporting.
2. Identify
the Goal of
the Study
The environmental data to be collected consists ofPM10 concentrations in air
samples collected on a daily basis. These concentrations will be examined to
determine if they are too high compared to regulatory levels. The primary question
of interest is whether the power plant is operating within the requirements of the
PSD regulations after completion of construction (specifically regarding PM10
concentration). The data will be used to determine how the ambient PM10 levels
compare to the NAAQS or PSD increment. Possible outcomes of the study are 1)
that requirements of the PSD regulations are being met (meaning no further action is
needed) or 2) that they are not being met (meaning that the ambient air quality
impact assessment that provided justification for the allowable PM10 emission rates
must be re-evaluated).
3. Identify
Information
Inputs
• Daily PM10 concentrations after construction (new data).
• Pre-construction PM10 concentration used for the permitting process (to be
provided by the plant's owner).
• EPA-published NAAQS standard for PM10.
• Permitted PSD increment (provided by State EPA District Office).
• Allowable PSD increment for PM10 concentration.
• Expected post-construction performance (in terms of PM10 concentration
predicted by the dispersion model during the permitting process)
• Daily meteorological measurement data (new data).
4. Define the
Boundaries
of the Study
The target population is the set of all possible 24-hour air samples obtained at the
fixed (pre-construction) sampling location, and the characteristic of interest for the
air samples is their PM10 concentrations. The study is bounded spatially to the area
from which the air samples are collected. Temporal restrictions for the historical
data are to the period during which data were collected for the permitting process.
While the monitoring program will operate for 18 months, the temporal boundaries
for the inference will be to the entire length of time that the power plant will be
operating under the post-construction conditions.
EPA QA/CS-2
33
March 2007
-------
Table 2. (cont.)
DQO Step
Statement
5. Develop
the Analytic
Approach
The parameter of interest is the 99.762 percentile of the estimated distribution of
post-construction PM10 concentrations minus the average pre-construction PM10
concentration. The type of inference is an upper tolerance bound (i.e., an upper
confidence bound on the percentile of interest). If the upper tolerance bound
exceeds the PSD increment, then decide that the air quality impact assessment
conducted as part of the PSD permitting process may not have adequately
represented the PM10 emissions from the plant's upgrades; if the upper tolerance
bound is less than the PSD increment, then decide that the plant is likely in
compliance with the PSD regulations.
6. Specify
Acceptance
or
Performance
Criteria
A tolerance interval is defined to be a confidence interval that contains a fixed
percentage of the population of observations. In this study, the fixed percentage is
99.726% (1-1/365) and the population is all 24-hour PMi0 concentrations. Often,
there is also interest in controlling the probability that a higher percentage is also
covered. For example, in this study we will control the probability that the tolerance
bound covers more than 99.9% of the population of 24-hour PM10 concentrations.
Thus, the performance criteria for this study are
• 95% likelihood that 99.726% of all 24-hour PM10 concentrations are less
than the PSD increment; and
• less than 15% chance that 99.9% of all 24-hour PM10 concentrations are
less than the PSD increment.
7. Develop
the Detailed
Plan for
Obtaining
Data
• Total number of valid data points required to meet the performance criteria:
397;
• One location (same as the pre-construction monitoring location);
• 18-month sampling duration (to allow for seasonal yearly differences);
• Collection of samples 7 days per week;
• Expected data completeness target of 75% (i.e., 75% usable data)
• Total number of samples to be collected: 550
EPA QA/CS-2
34
March 2007
-------
(Refine outputs
upon considering
additional
information)
Step 1: Define the Problem
Step 2: Identify the Goal of the Study
Step 3: Identify Information Inputs
Step 4: Define Boundaries of the Study
Step 5: Develop the Analytic Approach
Step 6: Specify Acceptance or
Performance Criteria
Step 7: Develop Detailed Plan for
Obtaining Data
(Change approach to
ambient air sampling)
(Evaluate options for
acceptable criteria and
sample sizes)
Figure 3: Iteration of Steps Within the DQO Process for the PMi0 Ambient Air
Monitoring Situation
EPA QA/CS-2
35
March 2007
-------
8.0 FROM PLANNING TO IMPLEMENTATION AND ASSESSMENT
As the planning committee completes the DQO Process, the project begins to move
from the planning stage to the implementation stage (Figure 1). The DQO Process yields a set
of specifications that are needed to support both the qualitative and quantitative components of
the design for data collection. As such, they serve as the starting point for the QA Project Plan
(QAPP), which is prepared and approved within the implementation stage. The systematic
planning process also leads to Standard Operating Procedures (SOPs) which ensure
conformance with required practices, reduction in error occurrences, and improved data
comparability, credibility, and defensibility. As the implementation phase proceeds, the
targeted data are collected according to the methods and procedures documented in the QA
Project Plan and SOPs. Once all data are collected, the project enters the assessment stage,
where the data are verified and validated for adherence to the QAPP and that the data are
appropriate for their intended use. Then a Data Quality Assessment is performed to determine
the extent to which the collected data meet the intended DQOs. This section addresses some
of the methods that will occur on this PMw monitoring program to complete its planning,
implementation, and assessment stages, and to make conclusions from the collected data.
Upon completion of the DQO Process, the planning committee had completed its
objective and no longer needed to meet formally. However, certain committee members,
including the study's principal investigator from the environmental consulting firm, andA&B
Inc.'s chief environmental engineer, continued as members of the project team assembled by
A&B Inc. to complete the planning stage of the Project Life Cycle (Figure 1), and to conduct
activities through the study's implementation and assessment stages. The remaining
committee members showed an interest to serve as study consultants or as reviewers of the
study findings. The following sections summarize the remaining activities on this study and
the conclusions made from the collected data.
8.1 Completion of the Planning Stage
The project team was aware that several additional planning steps were required before
data collection could begin. These steps included:
• Preparation of a Quality Assurance Project Plan (QAPP) and approval of this plan by
project and State EPA officials before post-construction sampling and data collection
begins. As noted in Figure 1 of Section 1, this follows the completion of systematic
planning and serves to document its outcome (e.g., the DQO Process statements noted
in Table 2). The QAPP describes the quality assurance procedures, quality control
specifications, and other technical activities that must be implemented to ensure that the
results of the project will meet the specifications that are represented in the DQOs. It
integrates all technical and quality aspects of the project in order to provide a
"blueprint" for obtaining the type and quality of data needed to address the principal
study questions. Further information on the content of a QAPP was obtained from
Guidance for Quality Assurance Project Plans (EPA QA/G-5) (U.S. EPA 2002b).
EPAQA/CS-2 36 March 2007
-------
One duty of the principal investigator was to oversee QAPP preparation and approval.
QAPP preparation was greatly facilitated by the review, discussion, and documentation
that occurred within the planning committee while executing the DQO Process. As a
result of these activities, information on several of the QAPP elements had been
gathered earlier, and therefore, could be placed directly within the QAPP. The plant's
chief environmental engineer assigned a QA Manager to the project, whose
responsibilities were to ensure that the QA activities specified within the QAPP were
being performed properly, to establish and follow up on any corrective action that may
be required, to oversee performance reviews and audits, and to control the distribution
of the proper version of the QAPP.
• Acquisition of Standard Operating Procedures (SOPs) for the sampling and analysis
process. SOPs were already available from the principal investigator and the owner of
the plant, who used them to conduct pre-construction monitoring. They were modeled
after the specifications given in the EPA publication Guidance for Preparing Standard
Operating Procedures (EPA QA/G-6) (U.S. 2001). The instruction manuals for all
EPA Reference or equivalent methods were included in the SOPs. The monitoring
consulting firm's field technicians were trained using these SOPs and instruction
manuals. The Code of Federal Regulations was also consulted to review the specified
method for PMio -- CFR Title 40, Part 50, Appendix M, Reference Method for the
Determination of Particulate Matter as PM10 in the Atmosphere.
• Selection of laboratories to supply and process 47 mm diameter Teflon filters in
accordance with the guidance and SOPs supplied by the nationwide paniculate matter
(PMio) monitoring network. The project team contracted with a public health
laboratory on the basis of cost, its relative closeness to Emmerton, (to eliminate the
expense associated with air shipment of samples), its experience in analyzing ambient
air samples for PMio that were collected from other sites in the area, and its ability to
demonstrate successful adherence to quality assurance and quality control procedures
required under the project.
• Training project personnel_to service the PMio sampler, including installation and
calibration of the PMio sampler and the electronic data acquisition and transmission
systems, conducting performance audits, and operating the sampler. To save travel
time and expenses, the monitoring consultants were asked to train two power plant
employees in the operation of the sampler and in the process of handling and shipment
of filter samples to the laboratory. Each trainee set up and operated the sampler three
times while being observed by the trainer, completed field data sheets, and reviewed
typical field data supplied by the laboratory.
• Creation of schedules for regular monitoring calibration and NIST standard traceability
checks, to verify that the sampling system is operating within specified guidelines and
standards. Checks of the ambient air sampler included pump speed, cleanliness of
internal flow path surfaces, the handling and placement of the Teflon filter in its
EPAQA/CS-2 37 March 2007
-------
cassette, and the presence of oil in the Well Impactor Ninety Six (WINS) impactor
well.
• Development of an implementation program for performance evaluation, which
includes on-site visits to check accuracy of calibrations and operator knowledge of
procedures. Supervised by the project's QA Manager, these evaluations include checks
of the sampler, operation of the contracted public health laboratory, performance
evaluation data, and chain-of-custody procedures.
8.2 Implementation Stage
Once planning activities were completed and the QAPP and SOPs were formally
approved, the post-construction ambient air monitoring study moved into the implementation
stage of the Project Life Cycle (Figure 1), where A&B Inc. collected samples and generated data
according to specifications and procedures documented in the QAPP and in the SOPs. Several
steps were involved in the sample collection and analysis process, including:
• Installing, calibrating, operating, and maintaining the PMio ambient air sampler,
collecting and replacing the filters each day, and shipping the filters to the laboratory for
PMio analysis.
• Performing chemical analysis of the filters to measure PMio concentrations.
• Conducting technical assessments to verify field and laboratory adherence to QAPP
specifications.
• Creating, populating, and maintaining a managed database containing the newly-
collected sample measurements, sample identifiers, and other information used to assess
the quality and utility of the measurements. The database was constructed in a format
that would easily allow the data to be input to programs and functions associated with the
statistical software that could be used for data analysis.
8.3 Assessment Stage
Once all samples were collected and analyzed and the data were stored in the project database,
the project moved from the implementation to the assessment stage of the Project Life Cycle
(Figure 1). In the assessment stage, the following two activities took place:
• Verifying and validating the data stored within the database, where the data were
generated according to specifications noted in the QAPP and are appropriate and
consistent for their intended use. EPA's Guidance on Environmental Data Verification
and Data Validation (EPA QA/G-8) (U.S. EPA 2002c) provided useful information on
how to perform these activities. As part of this effort, a chemist employed by the power
plant reviewed the data to verify that the measurements achieved the quality assurance
criteria, including specifications on data quality indicators (e.g., precision, accuracy,
representativeness, consistency, completeness, sensitivity) that are documented in the
EPAQA/CS-2 38 March 2007
-------
QAPP. The project database also included selected validation functions to ensure quality
of data entry and processing operations, including simple range and consistency checks
and checks for data completeness within a given sampling record. Any specific data
points (or batches of data) for which problems were identified through the validation
procedure were either appropriately corrected or invalidated. Quality assurance validity
flags, documented within the QAPP, were included as separate fields from the numerical
data within the project database, with values for these flags being properly paired with
the original data values.
• Performing statistical analyses on the daily average PMio concentrations within the study
database, as part of a data quality assessment (DQA) to determine the extent to which the
collected data meet the DQOs that were generated during the DQO Process and are
therefore suitable for use in addressing the study questions. (EPA's guidance document
entitled, Data Quality Assessment: A Reviewer's Guide (EPA QA/G-9R) (U.S. EPA
2006b), provided general information on how to assess data quality criteria and
performance specifications.) These data analyses began with a preliminary review that
evaluated data assumptions, identified potential statistical outliers, calculated descriptive
summary statistics (e.g., means, standard deviations, selected percentiles, ranges), and
prepared graphical representations of the data. The analyses then proceeded to
investigating the relationship between the newly-collected PMio concentration data and
meteorological data to determine the impact of climate-related factors on the
concentration data, and ultimately, to calculating tolerance intervals on the 99.726th
percentile of the 24-hour PMio concentration data, using the techniques specified by a
statistician, and determined whether the upper limit of the tolerance interval (after
subtracting off the average pre-construction PMio concentration) fell below the PSD
increment. These analysis techniques used in the data quality assessment were
documented in a separate statistical analysis report.
8.4 Conclusion
The study collected a total of 528 filter samples over the 18-month post-construction
monitoring period, slightly less than the 550 samples that were targeted to be collected. (The
number was reduced due to various field sampling problems that were encountered and an
occasional mix-up in field workers successfully collecting and shipping the filters.) Of the 528
samples, 452 were deemed to yield valid analytical measurements for PMio concentration. Thus,
approximately 18% fewer sample measurements were available for statistical analysis, relative to
the 550 measurements that were targeted. This percentage was below the 25% upper limit that
was considered acceptable for completeness and therefore deemed satisfactory.
To draw the final study conclusion, the upper bound of the tolerance interval on the
99.726th percentile of the daily 24-hour monitoring data was calculated from the newly-collected
PMio data. Then, the average pre-construction PMio concentration at the golf course site was
subtracted from this upper bound value, resulting in 12.6 |J,g/m3. This value was less than the
allowable PSD increment of 15 ng/m3. As a result, the study concluded that it was reasonable to
expect (with 95% confidence) that under conditions represented by the 18-month post-
construction monitoring period, daily average PMio levels in ambient air would exceed the PSD
EPAQA/CS-2 39 March 2007
-------
increment no more than one day per year, on average. This implied that PMio levels following
the plant upgrades were, in fact, found to be within safe levels (as represented by the allowable
PSD increment).
These findings, along with a summary of the study methods and the study data, were distributed
to the original planning committee members and other members of the project team for review.
Upon hearing back from these personnel, the plant's owners reported the study findings to public
health officials and the general public through a press release to the local media.
EPA QA/CS-2 40 March 2007
-------
9.0 REFERENCES
U.S. Code of Federal Regulations. 40 CFRPart 50, [Appendix M], Reference Method for
Determination of Particulate Matter as PMw in the Atmosphere
U.S. Code of Federal Regulations. 40 CFR Part 58, Ambient Air Quality Surveillance
U.S. Environmental Protection Agency, 1987. Ambient Monitoring Guidelines for Prevention of
Significant Deterioration, EPA-450/4-87-007
U. S. Environmental Protection Agency, 2001. Guidance for Preparing Standard Operating
Procedures (EPA QA/G-6), EPA/240/B-01/004
U.S. Environmental Protection Agency, 2002a. Guidelines for Ensuring and Maximizing the
Quality, Objectivity, Utility, and Integrity of Information Disseminated by the Environmental
Protection Agency, EPA/260/R-02/008
U.S. Environmental Protection Agency, 2002b. Guidance on Environmental Data Verification
and Data Validation (EPA QA/G-8), EPA/240/R-02/004
U.S. Environmental Protection Agency, 2002c. Guidance for Quality Assurance Project Plans
(EPA QA/G-5), EPA/240/R-02/009
U.S. Environmental Protection Agency, 2003. A Summary of General Assessment Factors for
Evaluating the Quality of Scientific and Technical Information, EPA 100/B-03/001
U.S. Environmental Protection Agency, 2006a. Systematic Planning using the Data Quality
Objectives Process, EPA QA/G-4, EPA/240/B-06/001
U.S. Environmental Protection Agency, 2006b. Data Quality Assessment: A Reviewer's Guide
(EPA QA/G-9R), EPA/240/B-06/002
U.S. Environmental Protection Agency, 2006c. Data Quality Assessment: Statistical Methods
for Practitioners (EPA QA/G-9S), EPA/240/B-06/003
EPAQA/CS-2 41 March 2007
------- |