Atmospheric Sciences Research Laboratory
CHEMISTRY, MATHEMATICS. METEOROLOGY. MODELING, PHYSICS
GUIDANCE FOR PREPARING
STATEMENT OF PROJECT QUALITY OBJECTIVES
June 2, 1987
Office of Acid Deposition, Environmental Monitoring and Quality Assurance
Office of Research and Development
U.S. Environmental Protection Agency
Research Triangle Park. North Carolina 27711
-------
~
A~HERIC OCIENCES RESFARCH IABORAroRY
GUIDAOCE FOR PREPARnl;
STA'I'EMENI' OF PROJECT QUALITY OBJECTIVES
Contract NUmber 68-02-4174
June 2, 1987
SUbmitted to:
Dr. Jack Durham, Project Officer
Atmospheric Sciences Research Laboratory (ASRL)
U.S. Environmental Protection Agency
Research Triangle Park, North Carolina 27711
SUbmitted by:
Research and Evaluation Associates, Inc.
1030 15th Street, N.W., Suite 750
Washington, D.C. 20005
(202) 842-2200
727 Eastowne Drive, Suite 200A
Chapel Hill, N.C. 27514
(919) 493-1661
PROPERTY OF
EPA UBRARY, RTP, NC
-------
NOl'ICE
THIS DCX:UMENI' IS A PRELIMINARY DRAFl'. It has not
been formally released by the u.S. EPA/ASRL
project Officer. It is being circulated for
comment on its technical accuracy and policy
inplications.
-------
DISCIAIMER
"Although the procedure described in this report
has been funded by the United States Environmental
Protection Agency through Contract Number
68-02-4174 to Research and" Evaluation Associates,
Inc., it has not been subjected to Agency review
and therefore does not necessarily reflect the
views of the Agency and no official endorsement
should be inferred."
Atm:>spheric Sciences Research Laboratory
Office of Research and Development
u.s. Enviromnental Protection Agency
Research Triangle park, North Carolina 27711
i i
-------
A~
The work on this project was performed by the QA/QC staff of
Research and Evaluation Associates, Inc. Mr. Don Cox, Project Leader,
and Ms. Sharron Rogers contributed to the project. As the external
reviewers of IX»s, we are fortunate to have had the support and advice
of numerous individuals who know the problems of research project
management, data collection, analyses, reporting, and quality control.
Dr. Jack Durham and Mr. Ron Patterson of u.S. EPA's AtrrDspheric
Sciences Research Laboratory, Research Triangle Park, NC directed our
efforts and provided technical advice. Ms. Brenda White graciously
typed this document and was willing to make our numerous revisions.
i i i
-------
ABSl'RACl'
This document presents a procedure for preparing a Statement of
Project Quality Cbjectives (SPQO). The procedure involves a dialogue
am::>ng clients, program/project managers, and technical implementers.
The SPQO evolves from the three stages of the Data Quality Cbjective
(IX)O) Developnent Process. . This guidance document is intended to
guide ASRL Project ~fanagers and project Officers in developing a
personalized process for gathering and conveying this information.
Supporting checklists are provided in appendices.
iv
-------
ABBREVIATIONS
AOOM
AES
AMS
APIOS
ASRL
CAPMON
CV
DQO
EPA
EPRI
Hi -Vol
HQ
ME
MOI
NAPAP
NCAR
Nl'N
OEN
a1E
PCA
PPA
PI
PM
PO
PQO
QA
QAMP
QAMS
QAO
QAPjP
QC
LIST OF ABBREVIATIONS
- Acid Deposition and Oxidant Model
- Atnospheric Environment Service
- American Meteorological SOciety
- Atnospheric Sciences Research Laboratory
- Canadian Air and Precipitation Monitoring Network
- COefficient of Variation
- Data Quality O:>jecti ves
- Environmental Protection Agency
- Electric Power Research Institute
- High-Volume
- Headquarters (EPA)
- ~~el Evaluation
- l-1eroc>randurn of Intent
- National Acid Precipitation Assessment Program
- National Center for Atnospheric Research
- National Trends Network
- Operation Evaluation Network
- Ontario Ministry of the Environment
- Principal Component Analysis
- Planned Program Accomplishments
- Principal Investigator
- Program Manager
- Project Officer
- Project Quality Q:>jectives
- Quality Assurance
- Quality Assurance Management Plan
- Quality Assurance l1anagement Staff
- Quality Assurance Officer
- Quality Assurance Project Plan
- Quality Control
v
-------
RADM
RFP
sav
- Regional Acid Deposition Model
- Request for Proposal
- Statement of ~rk
- Statement of Project Quality Cbjectives
- Standard Deviation
- Transition-Flow Reactor
- Task Officer
SPQO
S)
TFR
TO
vi
-------
TABLE OF CONrENrS
~tice . . . . . . . . . . . . . . . . . . . . . . . . . . . .
DisclaiIner . . . . . . . . . . . . . . . . . . . . . . . . . .
Acknowled.gement ..... . . . . . . . . . . . . . . . . . .
Abstract. . . . . . . . . . . . . . . . . . . . . . . . . .
List of Abbreviations . . . . . . . . . . . . . . . . . .
Introduction
. . .
. . . . . . .
.....
. . .
. . .
The SPQO Developnent Process. . . . . . . . . . . . . .
Steps in Preparing a SPQO . . . . . . . . . . . . . . . .
General Background. . . . . . . . . . . . . . . . . . .
Statement of the Problem. . . . . . . . , . . . . . .
Potential Application of the Product. . . . . . . . . .
Constraints of TiIne and Deli verables .........
Constraints of Budget. . . ,. . . . . . . . . . . . . . .
Alternatives and Selection of Approach. . . . . . .
Data Quality Statement. . . . . . . . . . . . . . . . .
Concurrences
........
.....
......
Appendices A - C
vii
~
i
ii
iii
iv
v
1
3
6
6
8
9
9
10
11
11
12
-------
LIS!' OF TABLES
Table
l.
2.
3.
4.
SPQO Responsibilities in Relation to DQO Process.
. . . .
Initial SPQO Checks. . . . . . . . . . . . . .
.....
~tions Available to ASRL Director. . . ~ . . . . . . . .
Sanple SPQO Format and Corresponding Roles. . . . . . . .
v;;;
~
3
5
5
7
-------
LIsr OF FIGURES
Fiaure
1.
Interaction of DQO Stages and SPQO Deyelopnent Process. . .
ix
~
2
-------
GUIDAOCE FOR PREPARnX;
A STATEMENr OF PROJECT QUALITY OBJECI'IVES
INl'ImUCXION
The primary purpose of the ASRL Statement of Project Quality
Cbjectives (SPQO) is to provide a logical framework for defining the
quality of data and the uncertainty inherent in the products to be
produced by a research project needed to support policy and regulatory
decision making. Develo~nt of the ASRL Statement of Project Quality
Cbjectives (SPQO) supports the Quality Assurance Management Staff's
development process for Data Quality Cbjectives (J:QO). Figure 1
illustrates the three-stage J:QO process in relation to ASRL' s SPQO
development process. A checklist is provided in Appendix A to
facilitate preparation of the SPQO.
.
A Statement of Project Quality Cbjectives will provide:
. General background about the national policy need for the
research project
Statement of the problem that the research project is to
solve, as seen by the client
Description of the product uses/users
Client's constraints of time, deli verables, and budget
Discussion of alternatives, selection of research approach
Data quality objectives
Recommendations and approvals.
.
.
.
.
.
The SPQO provides the ASRL Program Manager with measurement
standards to assess the quality of data produced during conduct of the
resulting research program. In addition to agency management uses,
the. coIIq;>leted SPQO, subsequent statement of work, and other quality
assurance support materials can be used by organizations responding to
a subsequent Request for Proposal (RFP) or proposing for a Cooperative
Agreement. The SPQO can provide critical information for development
of a work plan and quality assurance plan for the research project.
1
-------
I I Initial input
I
I by Decision L
--
I --
Maker. . . - -
L.- - - - - T- - - ~ .;,.j--...- '> Gath " Inf t"
. . - - '"'" " en.ng orma lon
r-- - - - - j - - - - - . . I Characterize Project Quality
'--- - - --""',
I II Clarifica- ~Fornulate the Problem
"
: tion of ,- - ---- --v. Define project objectives
I the problem. : Identify use and users
r----J-----'T
I III Development of '-------""> Develop alternatives,
I alternatives. r - - - - ~ approaches and initial DQOs
I I
I Selection of I
I . the approach ,
I ,
L. - - - to 1;e_USed. - -1
I
I
r---t----,
I
I
&- - - - - - - - - -.&
QAMS
DQO DEVELOPMENl'
PROCESS STAGES
r--~ --.-----,
L. - - - - - - - - '- _..J
I
Complete DQO
Figure 1.
ASRL
SPQO DEVELOPMENl' PROCESS
Define application of product,
limitations, opportunities,
goals, and deli verables
Develop project quality
objectives
Define constraints on budget
and deliverables
Prepare draft of SPQO
Selection of alternatives,
approach and DQOs
Complete SPQO
Interaction of DQO Stages and SPQO Development Process
2
-------
THE SPQO DEVELOPr-1ENI' PROCESS
The SPQO is developed in an interactive process involving the
client, HQ PM, ASRL Program Manager (ASRL PM), with sUPIX>rt from the
Project Manager, Project Officer (PO), Quality Assurance Officer
(QAO), and technical staff merrbers (Table 1). The ASRL Program
Manager is responsible for the execution of technical and management
functions for developnent of the SPQO with the approval of the ASRL
Laboratory Director and Division Director. Performance of these
functions is dependent on 1) the status of planning at the
client/decision maker levels and 2) the completeness of information
given in the early stages of the !XX> develo{neI1t process. Quality and
quantity of the background information provided in Stages I and II of
the !XX> process by the client/decision maker influence the selection
of applicable project quality objectives and the approach to the
project.
TABLE 1.
SPQO RESPONSmILITIES IN REIATION TO tQO PROCESS
DQO Stage Principal
Stage Description Resoonsibili ty Interactions
I Initial inp.lt HQ Pr.1 Client, HQ PM, ASRL
by Decision Lab Director and PM
Maker
II Clarification ASRL Pf.1 HQ PM, ASRL PM,
of the problem ASRL Division
Director
III Developnent ASRL Lab Director ASRL Project Manager,
of alternatives and Division Dir Principal Investigator
and selection ASRL PM and
of approach Project Manager
to be used
3
-------
To ensure an acceptable SPQO, the ASRL Program Manager rust
perform the following five tasks during the development of the SPQOs:
. Establish clients r quantitative project objectives
. Identify the users and uses of the product to be prod\Jced
. Identify the resources and/or technical requirements needed
to produce the product
Identify the duration of the project
Identify the approach suggested by the client.
.
.
To assist the ASRL PM and supporting technical and planning
staff, questions relating to the collection of initial information for
the SPQO process are given in Appendix A. The objective of these
questions is to aid the information gathering process and identify
where further information is needed.
The ASRL PM initially characterizes the overall project quality
specifications and identifies alternate research approaches that could
be taken to meet client or decision maker needs. The recomnended
approach should be the IroSt cost effective that can ensure an
acceptable level of performance by an, as yet, unspecified PrinciPal
Investigator. A descriptive statement of the research problem to be
addressed and a draft statement of work are written into an incomplete
draft of the SPQO and approved by the ASRL Director and HQ PM.
Questions relating to this process are found in Appendix A, Part 2,
Sections I and II and a checklist given in Table 2.
The ASRL PM, next, develops specific project quality objectives
in terms of accuracy, precision, representativeness, and completeness.
After an evaluation of the project quality objectives and alternative
approaches that can be taken, the ASRL Laboratory Director selects one
of the following options (Table 3) to be used by the PM.
4
-------
TABLE 2. INITIAL SPQO CHOCKS
I. Client's descriptions of:
A. General background
B. project background
C. Product uses
D. Product users
E. Qualitative product quality.
II. Client information included:
A. Statement of the problem
B. Sources of error
C. Quantitative project quality objectives
D. Project restrictions
E. Linkages to other projects.
III. Is it apparent that:
A. Planning staff participated in the development of the
above
B. Project objectives can be iIrplemented
C. Project use and/or needs are clearly identified by the
client?
TABLE 3. OPTIONS AVAILABLE TO ASRL DI~R
cption I
The Program r~ager defines the approach that will
be taken and the project quality objectives that
the PI must achieve during the project.
The PM provides the basic project quality
objectives and alternative research approaches
from which the PI must select. The PI develops
the data quality objectives, which are
incorporated into the ASRL SPQO.
cption II
cption III
The PM provides the SPQO and requires the PI to
respond with the approaches and project
quality objectives.
If cption III is followed, the ASRL PM and Division Director
review the PI's alternatives to the research approach, when received,
and select the research approach that best meets the client's or
decision maker's needs. No matter which option is selected the
product of this process is the SPQO.
5
-------
STEPS IN PREPARING A SPQO
A st~y-step process is described to provide guidance for
writing a statement of project quality objectives with enphasis on the
respective roles of the client and the proposed inplementer of the
anticipated research project. The SPQO format and the corresponding
client and iIrplernenter roles are given in Table 4. This suggested
format is designed .to produce a dOClm1eI1t that will set project
performance measurements and provide a planning tool for the PI to use
in developing the Quality Assurance Project Plan (QAPjP). In Appendix
B, a discussion of the relationship between project quality objectives
and QC criteria necessary for the project iIrplementation is provided.
Preparation of the SPQO for a new project/task is a highly
individualized process. The format and examples given are presented
to assist the writer of the statement in providing the detail and
content necessary for production of an acceptable statement.
Throughout the following narrative, reference will be made to a Data
Quality C1:>jective Statement for an existing project, "Deploy and
<:perate a Daily SUrface Monitoring Network," written following the
SPQO format. This example is presented in its entirety in Appendix c.
General Background
Typically, this section will describe to the project iIrplernenter
the background leading to the need for the product. This description
is written from the client's perspective on the issues and policies
requiring the product. This section presents the objectives for the
entire project in the broadest terms, as seen by the client or
decision maker. Usually the client's written description, the NAPAP
objectives, or EPA' s work plan provide a good starting point. In some
cases, EPA's PPAs, Peer Review, or workshop documentation are
excellent sources of material for this background. A written
statement by the decision maker or client should be incorporated into
this section.
6
-------
TABLE 4. SAMPLE ~ FORMAT AND CORRESPONDm; ROLES
.
GENERAL BACKGRroND about the source of the need for the products.
Client: Provides a perspective on the need for the product.
Written by: BJ PM.
srATEMENl' OF THE PROBLEM and quanti tati ve objectives that will
lead to a product or products needed by the client.
Client: Provides an overview of the problem to be solved, lists
uses and users of the product.
Written by: HQ PM, but may be Jrodified or compiled from
interviews by the ASRL PM.
.
.
Pal'ENI'IAL APPLICATION OF THE PROOUCT and potential users of
results of the project.
Client: Provides a description of the user's anticipated
product application, conclusions to be based on uses
of the product, and states deliverables.
Written by: BJ Program Office, HQ PM, and ASRL PM.
CONSTRAINrS OF TIME AND DELIVERABLES to serve as an indicator of
satisfactory performance to the client.
Client: Provides milestones and deliverables.
ID PM: Provides HQ milestones, deliverables, and decision points.
ASRL PM: Provides information on time constraints and
deliverables to the client and HQ for approval.
Written by: ASRL PM.
.
.
CONSTRAINrS OF BtJDGEl' set by the client to determine and assign
the resources required to reach the user's objectives.
Client: Provides funding information limitations.
ID PM: Provides allocation of funds.
ASRL PM: Provides information on budget constraints and specific
budget restrictions.
Wri tten by: ASRL PM.
.
DIOCUSSION OF ALTERNATIVES AND SELECTION OF APPRa\CH describes
the alternatives to the client's approach and the selection of
the approach to be used.
ASRL PM: Provides the alternatives to client approach and
recommends approach to be taken.
ASRL Lab Director: Selects which approach meets client needs.
Written by: ASRL PM.
.
DATA QUALITY srATEMEm' by the implementer of the project that
describes the project quality objectives required to meet the
client's, users', and implementer's needs.
ASRL PM: Provides specific data quality requirements for project.
ll: Provides detailed quality specifications for project.
Written by: ASRL PM.
7
-------
Appendix C illustrates how both a general and specific background
statement can be made. Most of the information was obtained from the
client, EPA's PPAs, and \tJOrkshop documentation by the HQ PM. Using
this initial input from the client, the ASRL PM will be to identify
and write materials answering the questions: why the product was
needed, who will be users, and what is needed by the client as a
product?
Statement of the Problem.
This section is designed to give the reader of the statement,
i.e., the implementer of the project, sufficient information about
what is needed by the client. The HQ PM and staff provide a
"statement of the Problem" and the quantitative project objectives.
The level of detail regarding the product that can be discussed at any
given phase of project definition will depend on how well the
information is gathered and how well the problem has been addressed by
the client. Specifically, the client should identify the following
points.
.
Sources of error or the acceptable uncertainty
Primary and secondary uses of the product
Primary and secondary users of the product
Problem(s) to be solved by the project
Quantitative objectives
Restrictions that apply
Linkages to other projects or products.
.
.
.
.
.
.
In Appendix A (Part II, Section II), a list of questions is
provided to assist in defining the content and writing of the
statement of the problem. Utilizing the questions as a checklist, a
simple or complex statement of the problem can be drafted. Detailed
points addressed in this section include:
. Project objectives in specific terms
. Product quality
. Reference to documents containing the scope of \tJOrk to be
performed.
8
-------
The background statement identifies the basic parameters of
interest to the users and the client. In the exanq;>le presented in
Appendix C, the BJ PM provided this level of detail based upon
documents produced by t\ttU \ttUrkshops held by the client. At this
stage, the BJ PM is writing from the client's perSPeCtive to identify:
1) what type of information is needed, 2) why this information is
needed, 3) the criteria and specifications for the product, and 4)
what type of products are being produced from the project outputs.
Identification of the fitness of the data and the intended use of
the products and quantitative objectives/goals of the project by the
BJ PM will lead to project quality objectives that can be
quantitatively measured and/or assessed. Fuzzy or qualitative
objectives/goals often lead to non-quantitative project quality
objectives. In stating the client's and users' goals, rernenber that
it is the BJ PM's responsibility to state the level of quality needed
for the intended use of the data and to lead the principal
Investigator to the eXPeCted precision and accuracy of each goal.
Potential Application of the Product
The HQ PM and HQ technical staff next provide a Statement of the
Potential Application of the Product (s) of the project. One or roore
statements can be given to define the conclusions and/or uses that
will be made from the product of the research. If this information
has been adequately stated, the following can be more fully detailed
. by the ASRL PM.
. Anticipated application of by-products from the project.
. Conclusions to be based on the product.
. Milestones or project deliverables for other projects
depending on this project.
Impact of project deliver abIes on the project products.
.
9
-------
Constraints of Time and Deliverables
An exarrple of a Constraints of Time and Deli verables is provided
in Appendix C. The BQ PM provides the information that identifies,
clarifies, and/or defines the critical constraints on the products
needed by the client. Questions 2 and 3 in Part 2, Section TV of
Appendix A provide the basic framework for this section. At this
point, the ASRL Program Manager with other ASRL staff should be able
to prepare a conplete statement of work for the anticipated research
effort. Major points addressed in this section are:
. Points for project "go" or "no go" decisions.
. Constraints imposed on scientific effort, data collection,
time frame, and deliver abIes.
Trade-offs that are acceptable and the resources available.
Basic project quality requirements.
.
.
Constraints of Budqet
As in every project, the Constraints of Budget for the project
nust be clearly specified. These constraints can be in the form of
detailing the allocation of funds to the eventual Principal
Investigator for specific COItp>nents of the project effort, e.g.,
start-up costs and QA/CC program costs. In some cases, the ASRL PM
may base financial decisions on the life time of the project and on
the quality of the work to be produced. Specific points to be
addressed are:
Planned budget for:
Life time of the project
First year start-up cost
First year operational cost
First year QA/OC cost
10
-------
Budget breakdowns for:
Project \YOrk plan
Peer review
Training
QA/QC activities
Financial report.
The following format is an exanple which could be used:
FY:
Constraints of Budget
87 88 89 90 91
92
93
94
m-HooSE
Manpower
Travel, $I{
(Fill in as required)
EX-HOOSE
Modeling, $I{
Fld Meas., $I{
Likewise the following statement could be used as appropriate:
NOI'E: Planning budgets are confidential. This section will be
completed upon award and negotiation of revisions to this SPQO.
Alternatives and Selection of APProach
This section, written by the ASRL PM, describes in detail the
alternatives to the client's approach and justifies selection of the
specific approach to be used. Appendix C (pp 18-22) provides an
exanple of this section. Appendix A (Part 2, Section TV question 2)
may provide assistance through a list of points to be checked by the
writer of the statement. Due to the detail of these points, they will
not be repeated here.
11
-------
Data Ouality Statement
This section is based upon negotiations between the client, HQ
Staff, ASRL staff, and the PI. The statement is usually written by
the ASRL PM, Project Manager, Project Officer, and/or the PI and is
based upon the ~rk plan for the project.. If option III has been
selected (Table 3) by the ASRL Laboratory Director, the section on
Project Quality will be conpleted upon approval of the quality
sPecifications submitted by the princiPal Investigator. Appendix A
(Part 2, Section VII, question 2) provides a list that can be used as
guidance to write this section. In Appendix C (pp 22-24), a detailed
exanple of a sPecific Data Quality Statement is given.
Concurrences
The conplexity and dynamic nature of ASRL research projects
demand the following concurrences on the content of the statement.
Recommended by ASRL
Project Officer:
Branch Chief:
Division Director:
Concurrences
(Signature)
(Date)
Project Manager:
QA Officer:
Laboratory Director:
12
-------
AI
-------
Appendix A
Checklist for Statement of Project Quality Cbjectives
THIS CHEX:KLIST IS DESIGNED 'ID AID IN ccm>IIATION AND PRESENl'ATION OF
INFORMATION. ccm>LETION OF THIS CliOCKLIST PERMITS ASRL MANAGERS TO
ASSESS THE s:mWS AND PROBLEMS IN DE.VELOPING THE SPQO. IN PARI.' 2,
SEX::TIONS OF THIS CtiJxKLIST CORRESPOND TO ~IONS OF A SUOOESTED
FORMAT FOR A WRITl'EN s:mTEMENr OF PRDJECr QUALITY OBJECTIVES.
Questions in the following materials are assigned one of b«> ratings,
"G" or "R". A question rated as "G" for guidance is provided to
assist or to provide guidance to the Project Officer during develop-
ment of the DQO. It will not necessarily be used in evaluation of the
conpleted process. A question with an "R" for recommended strong], v
addresses an issue or process that the Project Officer should complete
during this stage of developing the DQO inasnuch as such information
will be necessary to denonstrate the quality of the project.
IDENl'IFICATION
Program name:
Project name:
Provide names for:
a) Client
b) Decision Maker
c) Program Manager (HQ)
d) Project Manager
e) Project Officer
f) Quality Assurance Officer
i) Principal Investigator (PI)
Title of Statement:
Date:
Revision 0
Revision 1
Revision 2
13
-------
Part 1 - DQO Development Process
I.
INFORMATION PROVIDED BY HQ PR(X;RAM MANAGER
A. Information about the Client
1. Has the client been identified?
a) Name
b) Agency
c) Office
2. Bas the decision maker been identified?
a) Name
b) Agency
c) Office:
B. Information from the Client provided by HOs
1. Have the following been given or identified by the
client or decision maker?
a) Planned Program Accomplishments (PPAs)?
1) Goals?
2) Rationale?
3) Resources?
4) Description?
5) Deliverables?
6) Qualitative statements of use?
7) Program Quality Objectives?
b) The decision to be made or product needed?
c) The level of uncertainty that is acceptable to the
decision maker?
d) The anount of time available?
e) The level of resources available?
f) Intended use of the product?
g) Users of the product?
h) Why the product is needed?
i) Background information?
2. Has a statement of the quality of the product been
given?
a) In qualitative form?
b) In quantitative form?
c) As a hypothesis?
3. Has a written statement (qualitative) of the problem
been:
a) Described in sufficient detail to implement?
b) Approved by the:
1) Client?
2) Decision Maker?
4. Has the decision maker or client background information
on the context of the problem been given?
II. Clearances
. A. Clearances other than ASRL
1. Has the decision maker or client reviewed and commented
on:
a)
b)
c)
d)
Statement of problem and decisions to be made?
Level of uncertainty or quality goals?
Use and users of product?
Background information?
14
-------
2.
Has decision maker or client provided program quality
objectives?
Has the HQs' technical staff reviewed and approved the
QA Project Plan?
Has the client approved both the \\1Ork plan and
Quality Assurance Project Plan?
3.
4.
15
-------
Part 2. Statement of Project Quality Cbjectives
I. CLIEN!' I S DEOCRIPl'ION OF THE GENERAL MCKGRaJND
1. Has information been provided/acquired and a statement
written to address:
a) Key policy questions?
b) National goals that will be inpacted by
the product of this project? .
2. Has description of general background been provided:
a) Project background (including source of
the need for the products)?
b) Product uses clearly identified?
c) SUImnarY of project needs and objectives
clear to reader?
d) Basic qualitative statement of quality
given?
e) Interrelated projects given as reference?
f) Description written by decision maker
(client)?
II. CLIEN!' I S srATEMENl' OF THE PROBLEN AND PROJECT OBJECl'IVES
1. Has client provided or has project management staff
written:
a) General statement of the problem?
b) Statement on sources of error or
acceptable uncertainty?
c) Clear indication of who are the decision
maker and potential data users?
d) Clear identification of secondary user(s)
and use (s)?
e) SUfficient information on what is needed
from the Project Manager/project Officer?
f) Statement of the problem to be solved
by the project given?
2. Has client or project management staff provided
a statement of the project objectives in terms
that the Principal Investigator can utilize to
provide for a quality product:
a) Level of detail sufficient to determine
quantitative objectives?
b) Critical objectives identified by the
writer (client)?
c) Intended use (s) of product given?
3. Have the restrictions that apply to this project
been identified and/or described by the:
a) Client?
b) Program Manager?
c) Laboratory Director?
4. Have an identification of primary and/or secondary
user(s) needs been made and given in sufficient detail
to address:
a) List of primary users needs?
b) List of secondary users needs?
16
R( )
R( )
R( )
R( )
G( )
G( )
G( )
G( )
R( )
R( )
R( )
G( )
R( )
R( )
R( )
R( )
R( )
R( )
R( )
R( )
G( )
G( )
-------
5.
Are the following clearly understood:
a) Linkages to other projects?
b) Dependency of other projects on this
project's products?
Is a statement of product quality provided, specific
to which the PrinciPal Investigator can/IIUSt respond
to in:
a) Project ~rk Plan?
b) Project Quality Assurance Plan?
Is specific reference provided to support documents
containing product 'quality requirements or
criteria?
Has a specific statement of the problem that the
PrinciPal Investigator rust respond to in
the \\1Ork plan been provided/written?
Has specific reference been made to documents
containing the scope of \\1Ork to be conpleted?
Is it apparent that the decision maker and
planning staff participated in the developnent
of this section of the project quality statement?
6.
7.
8.
9.
10.
III. CLIENl" S DEOCRIPl'ION CF THE APPLICATION (USE) OF THE PRCDUCl'
1. Is the description of the application of
the product:
a) Clear?
b) In sufficient detail to be inplemented?
2. Are statements defining the conclusions of the
product:
a) Clear?
b) In sufficient detail to be inplemented?
3. Are limitations, opportunities, or options for any
by-products of this project specified?
4. Are goals for application given in quantitative
tenns?
5. Have milestones or deli verables for other projects/
tasks depending on this project been identified?
6. Has a clear statement of how the project deli verables
will inpact on the project products been given?
a) Are there critical inputs?
7. Is it apparent that the decision maker, planning
staff, and user (s) participated in the development
of this section of the project quality statement?
IV. CLIENl" S CONSTRAINTS OF TIME AND DELIVERABLEs
1. Are decision points clearly identified?
2. Do the statements of deli verables support the
initial client/users' needs?
3. Are the constraints inposed reasonable for:
a) Scientific effort?
b) Data collected?
c) Products to be delivered?
d) Time frame to be conpleted within?
17
G( )
G( )
R( )
R( )
G( )
R( )
G( )
R( )
R( )
R( )
R( )
R( )
G( )
G( )
R( )
R( )
R( )
R( )
R( )
R( )
R( )
R ( )
R( )
R( )
-------
4.
Is there sufficient information to draw
conclusions about what is needed, what
resources are available, and what tradeoffs
are acceptable?
Can a reader of the project quality statement
at this level of detail determine what can be
provided as a quality product from the infonnation
provided?
Will the statements derived from Stages I and
II (QAMS DQO Development) give the Principal
Investigator enough information in Stage III to
write an approach to .the client's needs/problems?
Is it apparent that the decision maker and planning
staff participated in the development of this
section of the project quality statement?
5.
6.
7.
v. CLIEN!" S CONSI'RAINl'S OF BUDGEr
1. Are the constraints of the project budget clearly
stated?
2. Is there sufficient information to draw
conclusions about what tradeoffs are acceptable?
3. Are financial decision points clearly identified?
4. Has a planned budget been given for:
a} Life time of project?
b} First year start up cost?
c} First year operational cost?
5. Does the budget breakdown include:
a} Project ~rk plan costs?
b} Peer Review costs?
c} Training costs?
d} Financial report costs?
e) Quality assurance and control costs?
6. Is it apparent that the decision maker and planning
staff participated in the development of this
section of the project quality statement?
R( }
G( }
G( }
R( }
R( }
R( }
R( }
R( }
R( }
R( }
G( }
G( }
G( )
G( }
G( }
R( }
VI. srAGE II IMPLEMENl'ER' S DIOCUSSION OF ALTERNATIVES AND SELECTION
OF APPRQ\CH
1. Identify who is serving as the implementer at
this stage (Stage II) of the development of
the project quality objectives
a) Is it the Project Officer?
b) Is it the PrinciPal Investigator?
2. Is it apparent that the Stage II implementer
understands and has responded to the:
a} ())jectives of the project?
b) Stated product needs?
c} Stated user needs?
d} Stated constraints?
e) Stated deliverables?
f} Stated data analysis procedures?
g} Stated database requirements?
h) Stated OC database requirements?
18
R( }
R( }
R( )
R( )
R( )
R( )
R( )
R( )
R( )
R( )
-------
Is it apparent that the Stage II implementer and the
decision maker have agreed on the approach taken?
Does the approach given best balance the stated
objectives, resources, needs, and constraints given? R()
Is it clear to a reader of this section of the project
quality statement how the needs of the decision maker
and user will be met?
Are there statements in this section concerning
what products will be provided?
Are these products:
a) Related to client/user needs?
b) Related to the secondary user needs?
c) Sufficient to meet the stated project
objectives?
Is the information provided sufficient in detail to
determine requirements for project quality objectives
to be given in the next stage?
Is it apparent that the planning staff was involved
in the development of the selected approach?
VII. PROOECT QUALITY OBJECTIvE srATEMENr FOR SELECrED APPRO!\CH
1. Is a qualitative statement of project quality
objectives given? If so, is there a clear statement
of the way in which each conclusion was reached?
2. Is a quantitative statement of the project quality
. objectives given?
If so, does the statement:
a) Relate to the approach given?
b) Relate to the stated product need?
c) Relate to the intended use of the
data/project?
d) Presented in terms of:
1) Precision and accuracy?
2) Frequency of measurement?
3) Representativeness?
4) Completeness?
3. will the project quality objectives, as represented
by the IXJOs be effective? Do they: R( )
a) Create passive quality assurance procedures? R( )
b) Create interactive quality assurance procedures? R()
c) Characterize the data quality as needed
by the client and user(s)?
d) Characterize the data quality for a
data cOllection/database effort?
e) Consider the use of EPAs databases?
f) Support the approach selected by the
Stage II implementer?
g) Require quality control data reports?
4. Are critical project objectives supported by the
D'JOs given?
5. Are the products needed by the project's secondary
users supported by the IXJOs given?
3.
4.
5.
6.
7.
8.
20
R( )
R( )
R( )
R( )
R( )
R( )
G( )
G( )
R( )
R( )
R( )
R( )
R( )
R( )
R( )
R( )
R( )
R( )
R( )
G( )
R( )
R( )
G( )
-------
6.
Are the DQOs presented in sufficient detail to
support definition of the accuracy and precision
values for the Quality Assurance Project Plan?
Are the DQOs given in sufficient detail to support
definition of Quality Assurance reporting require-
ments and evaluation?
7.
VIII. SIJMMARY
Provide as needed
IX. PROJECT QUALITY STATEMEN1' RECCJt1MENDED BY:
Provide names
x. PROJECT QUALITY STATEMENl' APPROVED BY:
1. Client
a) Name
b) Title
c) Signature
2. Decision Maker
a) Name
b) Title
c) Signature
3. Program Manager (HQ)
a) Name
b) Title
c) Signature
4. Laboratory Director
a) Name
b) Title
c) Signature
5. Project Manager
a) Name
b) Title
c) Signature
6. Quality Assurance Officer
a) Name
b) Title
c) Signature
7. Senior Program Staff
a) Name
b) Title
c) Signature
8. Senior Technical Staff
a) Name
b) Title
c) Signature
. 9. Principal Investigator (if known)
a) Name
b) Title
c) Signature
Date
Date
Date
Date
Date
Date
Date
Date
Date
21
G( )
G( )
-------
B
-------
APPENDIX B
REIATIONSHIP OF THE r.QO PRCX:ESS AND OC CRITERIA
The relationship of the Data Quality <1:>jective (r.QO) Development
Process, the Statement of Project Quality <1:>jectives (SPQO), and the
Quality Control (OC) criteria given in the QA Project Plan (QAPjP)
differ only in the anount of detail given in each. The SPQO draws
upon the initial Program Quality Objectives and enhances these
objectives to a higher level of sPecification for use in the Scope of
Work (sav). The result is the Project Quality Objectives (PQQs).
Depending on the project inplementation process selected by the
Project Officer (PO), the Principal Investigator (PI) will either
inplernent the PQOs specified by the PO or provide the his or her own
PQQs to the PO. Upon acceptance of the PIs ~rk plan, the PQQs agreed
upon become the DQOs for the project and represent conpletion of the
DQO development process. Table 1 illustrates this process.
mBLE 1. LEVELS OF QUALITY C>BJE:CrIVES IN r.QO PRCQ:SS
IXX> Process
Stage
I
Responsibility
Client and HQ
Program Manager
Statement of Project Quality Project Manager and
PO Objectives
PO
Oualitv Level
Program Quality Objectives
II
From II to III
sav
III
End of III
PIs ~rk Plan
QAPjP
PI and PO
PI
The DQOs provide a simple and straight forward statement of data
quality to be achieved by the project data collection, analysis, and
reporting activities. The relationship between the DQOs and
development of OC limits or criteria incorporates a series of sPecific
23
-------
ste~ from the given DQOs to the QAPjP. How this is achieved by the
PI will be an iterative and somewhat complex process. It is the role
of the PIs technical staff to refine the DQOs to the awropriate level
of detail enabling a specific quality control process to be
implemented, documented, and audited.
During the detailed. preparation of the OAPjP sections,
quantitative statements of the types of errors/biases that will be
controlled, the level of this control, and Supporting data that is
needed nust be given. These statements will characterize known
sources of error and bias that BUSt be controlled and the QC checks
that will be performed to ensure their control. To illustrate this
process, an example is provided.
PR(X;RAMMATIC EXAMPLE
A IIDdel for air quality acidic deposition is needed to perform an
assessment of the relative importance of local versus regional sources
to mesoscale acid deposition. The specific requirements of the IIDdel
are:
a.
To provide a means of predicting acidic deposition loading
within + 100% of observed field test values to economically
iITportant surfaces within urban areas;
b.
To determine the relative importance of local, as opposed to
long-range, transport to deposition loading at sensitive
receptors within 200 to 300 kIn of large point or area
sources;
c.
To determine the relative irnportance of the deposition of
primary, as opposed to secondary, sulfate near large point
and area sources.
The client 's requirements for field and laboratory measurements
state that only standard analytical procedures can be used for the
determination of rainfall and the ionic concentration related to
24
-------
acidity.
Table 2.
The objectives for these measurements are given below in
mBLE 2. PRCGRAM ~ GIVEN BY CLIENT
Precision:
+ 10% as a standard deviation of duplicate
analysis.
Accuracy:
~ 10% of standard reference values as differences.
Now that the overall measurement quality objectives for the
program (Table 2) have been defined, the next step in our example is
to establish the data quality indicators for each. During the
detailed planning and preparation of the PI IS QAPjP, the SPQOs and
DQOs are used as the starting point for developing explicit,
quantitative statements of the types of error that will be accepted
am controlled, and the QC information that will be collected in order
to characterize the quality of the data and identify errors or biases.
These indicators are needed to select the appropriate collection
method (s), field and/or laboratory analytical techniques, and data
processing and management approaches for the results. The indicators
also serve as the basis for selection o~ the PI's QA activities and QC
criteria and procedures for the project to be given in the QAPjP. For
example, the data quality indicator for sulfates is given in Table 3.
mBLE 3. EXAMPLE DA'I2\ QUALITY INDICA'IDR FOR SULFATE ANALYSIS
Precision:
~ 10% difference as a meaQ/RSD
Accuracy:
~ 10% difference from a known standard.
Both high volume and dichotomous samplers are required for
collecting sulfate samples. The PI has identified that total mass of
the filter and 002 gaseous sampler are also required for the
25
-------
evaluation of the accuracy for sulfate flux. At this point, a system
of project quality control objectives is required to provide an
assessment of the data being collected. The sanpler flows, transport
of the filter, QC checks, collocated sanplers, transport to the
laboratory, extraction analysis by ion cbronotography method, am data
processing will be part of the total data base for doannenting
achievement of project data quality objectives. In addition, the use
of the 502 nonitor provides the CO-analysis relationship needed by the
modelers for the model accuracy estimates of sulfate prediction.
Based on reconunendations of the PI I S technical staff for data
collection activities identified above, the Project Quality Cbjectives
for sulfate measurements (Table 4) are used in preparing the QAPjP.
Technical staff guidance in preparing the QAPjP quality objectives is
a critical source of information, necessary for the PI and PO to
support development and inplementation of sPecific plans for model
evaluation and use.
'mBLE 4. EXAMPLE P~ QUALITY OB.JE:CrIVES
SULFATE FOR DATA FROM FIELD Sl'ODY
Experimental
Conditions
Precision
Measurement
Parameter 1
002
004=
Mass (DICH
Balance)
(HI-VOL
Balance)
Atm:>. Sanples
Atm:> . Sanples
Atm:> . Sanples
=t. 10%
::!: 10%3
+ 4%3
Accuracy2
=t. 10%
+ 10%
+ 0.02mg/10Orng
:t 0.04ng/zero
+ 0.5ng/l,2,5 9
:t l.Orng/zero
Atm:>. Sanples
:t 4%3
1 Reference methods document is EPA standard method of collection
am analysis (EPA 600/4 series).
2 Difference from a known concentration.
3 Standard Deviation of (1) replicate analysis or (2) the percent
differences between collocated s~lers.
26
-------
The next step for the PI I S technical staff is to define data
collection and analytical quality control checks to support the
quality objectives for sulfates. Table 5 provides a basic list of OC
criteria related to the Project Quality Objective parameters.
Obviously, a system of quality control checks is needed to detennine
the quality of the field data being collected. The QAPjP and related
RPMs nust provide the. procedures, evaluation techniques, and
mechanisms for any corrective action needed to ensure that data of
good quality is being collected. A functioning QA plan or program
will minimize or eliminate surprise nonconformance problems. The goal
is prevention of the generation of bad data, rather than correction
after collection.
AUDIT QUFSl'IONNAIRE PROCESS
It is the role of the PI I S technical staff to refine the data
\
quality objective to the level of detail shown in Tables 4 and 5 of
the exanple. The goals of the staff menDers are to select specific
quali ty control and auditing approaches that will control the bias and
error of the data being collected in the field and to define the
criteria for the audit evaluation. The audit and OC approach selected
should be the one that is the best balance of the time requirements,
deliverables, and cost constraints required by those directing the
overall program.
The PI . s technical staff also is responsible for
initiating/implementing the rore technical phases of the audit
process. Their responsibility extends to the preparation of detailed
guidance for internal audits in the form of audit questionnaires which
provide the assessment and performance criteria for the data
collection, analysis, and precision assessment activities. During
detailed planning and preparation of the project, the technical staffs
guidance for project internal audits, the DQCs, and QC limits/criteria
are used as the starting p:>int for developing: 1) explicit,
quantitative statements of the types of errors;biases that will be
controlled, 2) the level of this control, and 3) the support data that
will be collected during the internal audits or control checks to
characterize all the known sources of error and bias.
27
-------
'mBLE 5. EXAMPLE OF QC CRITERIA
Related
Parameter
Collection/Analytical
Method
Dichotomus
Hi -Vol
9)4=
9)2
Con. 100ni tor
Mass
Hi-Vol/Balance
Dichotomus/Balance
9)4=
Ion Chrom.
Data Handling Strip chart
Data Acquisition
Input
1 Collocated sampler for precision
2 Every 5th filter
3 Every 10th filter
4
Compared to perfect curve
5 Also within + 2 S.D.
6 Also within + 2.8 CV limit
7
10% of samples
8
5 to 8% of samples
9 100% of data
Control
Check
Flow Rate
Flow Rate
Zero
precision
SPAN
Zero
1,2,5 gram wt.
Tare replicates
Gross replicates
Zero
100mg weight
Tare replicates
Gross replicates
Calibration slope
Calibration slope
Standard sample
Percent recovery
Replicates
Baseline, hr, Aug.
Baseline & time
Raw vs printout
28
Control
Limi t .
% 10% set pointl
~ 10% set pointl
zero % 0.025 ppm
+ 10% std. dev.
+ 15% difference2
~ 1.Omg of zero3
~ 0.5mg of wts.
~ 2.8mg difference
+ 5.Omg difference
+ 0.04mg of zero2
~ 0.02mg of wts.3
~ O.Olmg difference
+ 0.02mg difference
> + 15% slope diff.4
~ 2.0% differenceS
+ 10% difference6
90%
+ 4% as std. dev.6
< 1. 0% incorrect 7
< 1.0% incorrect8
< 2.0% incorrect9
-------
~I
-------
APPENDIX C
DATA QUALITY OBJECTIVE
DEPLOY AND OPERATE A DAILY SURFACE
MONITORING NETWORK
Revision:
Date:
3
31 July 1986
1.
Client's DescriDtion of the General Backaround
1.1 General Background
Over the past several years, various proposals have called for amelioration
of the adverse effects of acid deposition by controlling the emissions
of acid precursors. Each of these proposals has raised a number of
international and regional issues. The National Acid Precipitation
Assessment Program (NAPAP) has decided that numerical models provide the
most sCientifically defensible tool for describing existing source-receptor
relationships, for predicting the effect that emission changes might
have on these relationships and, thus, for resolving the various issues
in question. Specifically, the key policy questions that must be answered
for a given distribution of emissions include:
Decosition Loadinqs. What are the seasonal and annual averages of
the wet ana dry deposition of acidic species (esp., sulfate, nitrate,
hydrogen, and ammonium ions) and oxidants to specific states and
Canadian provinces or to portions of states and provinces?
b. Source Attribution. What is the net (i.e., wet and dry) deposition
of sulfur oxides, nitrogen oxides, and volatile organic compounds
(VOCs) to each receptor area from each geopolitical source area?
a.
c.
Chemical Nonlinearity. How effective are various strategies for
controlling sulfur oxides, nitrogen oxides, and VOCs in reducing
acid deposition to sensitive receptor areas? For example, is the
regional source-receptor response relationship "linear" for sulfur
oxides? Do nitrogen oxides and VOC play an important role in
governing the chemical response of the atmospheric system?
In order to produce scientifically defensible tools for analyzing the
consequences of proposed acid-deposition control policles, the
Environmental Protection Agency (EPA), the Ontario Ministry of the
Environment (OME), the Atmospheric Envlronment Service (AES) of Environment
Canada, and the Electric Power Research Institute (EPRI) have supported
the development of regional-~cale acid-deposition models. EPA has been
supporting the development of a Regional .Acid Deposition Model (RADM) at
the National Center for Atmospheric Research (NCAR). OME, AES and the
Umweltbundesamt (West Germany) have been supporting the development of
the Acid Deposition and Oxidant Model (ADOM) at ERT, Inc. The RADM and
ADOM are sophisticated state-of-the-science models specifically designed
1
-------
for application to the complex technical issues implied by the policy
issues. Although the RADM and ADOM have already been shown to be capable
of addressing the policy issues, their predictions will not gain wide
acceptance unless the models have been subjected to a credible model
evaluation program. Therefore, an important element of the modeling
effort must be to establish model credibility through comparison with
observed conditions. Without evaluation of the models, large uncertainty
on the reliability of model predictions limits the use of these predictions
in making policy decisions.
The decision to design a program for the evaluation of these models
initiated a sequence of activities that is expected to culminate in
evaluated models for use in making policy decisions. A series of
workshops, aimed at assisting EPA, OME, AES, and EPRI in this design
process, have been convened. The first workshop, sponsored by the EPA in
Raleigh, North Carolina on February 11-13, 1986, focused on identifying
various protocols for evaluating models against field observations.
Several field projects were identified to obtain data for comparison with
model predictions (Pennell 1986). The second workshop, which was sponsored
by EPRI, was held in Seattle, Washington on March 11-13, 1986. This
workshop focused on refining the data requirements and data quality
objectives for the field projects identified in the first workshop (Sarchet
1986). A third workshop, sponsored by OME, was conducted in Toronto,
Ontario on June 11-13, 1986, to consider quality auditing for the field
measurements and model exercises for the model evaluation program. Data
archival and coordination for all of the projects within the program, as
well as other related monitoring and research projects supported by NAPAP,
EPRI, and the Canadian agencies, were also considered at this workshop
(Olsen 1986).
Model evaluation has two functions:
Assess the ability of models to predict deposition and air
concentration patterns and amoants for effects applications and
policy analysis decisions.
. Provide scientific credibility to assure that model predictions are
correct for the right reasons.
.
These functions are manifested on different time and space scales and
result in evaluations with different functions. Integrated evaluation
(also referred to as operational evaluation) addresses primarily the policy
function. Diagnostic evaluation, on the other hand, addresses primarily
the credibility function and hence are also an important adjunct to the
integrated evaluation. These types of evaluations do not establish
confidence limits on model predictions. Rather, they focus on determining
whether the models simulate the salient features of atmospheric transport,
transformation, diffusion and deposition processes reasonably well and
whether the model is giving the correct predictions for the right reasons.
Integrated evaluation considers how well the model is able to predict
seasonal or annual mean values of deposition and air concentrations.
Field
2
-------
studies to support integrated evaluation must span a sufficient time for
these averages to be established over the simulation period. Frequently
this type of evaluation has been referred to as an operational evaluation
because it examines the model predictions that would be operationally
used in making policy decision or effects assessments. The variables
predicted by the models are the surface air concentration and deposition
patterns of the major species: S02' NOx' 03' and perhaps a few other
oxidants in air and H+, S042-, N03-' and NH4+ in precipitation. Model-
derived seasonal or annual means are to be compared to similar averages
of the field observations of these quantities.
Diagnostic evaluation has as its principal objective to determine if the
model, as a complete entity, and its various modules are functioning
properly. Model complexity requires that the evaluation treat each
individually simulated episode of 1- to 6-day duration as a separate case
study. Variables studied in a diagnostic evaluation include those obtained
for integrated evaluation, as well as those variables which are .sensitive
to the performance of specific modules. Furthermore, diagnostic evaluation
requires information on the three-dimensional nature of the observed
fields to determine if the models are performing correctly.
In the most detailed evaluation, at the sub-modular or mechanistic level,
the workings of individual modules within the complex regional-scale
acid deposition models are evaluated. Such evaluations require extremely
detailed observations of variables unique to the processes being simulated
within each module. High temporal and spatial resolution observations
in three dimensions are needed to determine if these modules satisfactorily
parameterize subgrid scale features of the observed fields.
Diagnostic evaluation spans the widest range of time scales. Sub-event
information, i.e., high temporal resolution, helps to establish that
modules interact correctly. At the other extreme, observed seasonal and
annual averages are needed to demonstrate that individual episodes can be
combined to form longer term means. All of these factors were taken
into consideration by the workshop participants in the identification of
field studies that were necessary for the various aspects of model
evaluation.
Field projects needed for model evaluation that were identified by the
workshop participants include:
INTEGRATED EVALUATION
1.
2.
3.
4.
5.
Deploy and Operate a Daily Surface Monitoring Network (Operational)
Conduct Vertical Profiles over Modeling Domain (Diagnostic)
Determine Subgrid Deposition Variability (Diagnostic)
Evaluate 1985 NAPAP Emission Inventory (Input, Diagnostic)
Evaluate Inflow Boundary Conditions (Input, Diagnostic)
3
-------
MECHANISTIC (MODULE) EVALUATION
6.
7.
8.
9.
Evaluate Wet Deposition Module
Evaluate Dry Deposition Module
Evaluate Atmospheric Transport Module
Evaluate Gas Phase Chemistry Module
The principal purpose of these field projects is to provide'a data base
.for RADM and ADaM evaluation (Pennell 1986; Sarchet 1986). The workshop
participants further recommended that the model evaluation field studies
need to span at least two full .calendar years in which the daily surface
network (Project 1) is producing quality data. Embedded within these two
years would be four periods in which more intensive (higher sampling
frequency) and extensive (additional variables) measurements would be
taken from aircraft and at special (enhanced) surface sites. The surface
network is capable of providing field observations at all spatial scales.
Its national coverage, with greater site density in the northeastern United
States, will yield national patterns of the geographic distribution of the
observed fields. In the area of higher site density, fields associated
with individual wet deposition episodes can be resolved for diagnostic
evaluations. Enhanced sites within the surface network serve as focal
points for diagnostic and subgrid scale studies. Special sites distributed
in clusters about selected surface network sites can support diagnostic
evaluation at a modular level and are needed for interpreting the spatial
variability associated with surface network observations. Evaluation of
certain modules may require clusters of closely spaced surface sites.
But the surface networks are not capable of obtaining vertical profile
da~a needed in the diagnostic evaluations. Sampling and measurement
systems carried aloft by free or tethered balloons, or on aircraft are
needed to acquire these data. Such observations mainly support the
diagnostic evaluations, but also contribute to modular evaluations, and
in special situations, to integrated evaluation. These measurements can
only be performed during intensive periods.
The intensive periods were suggested primarily to collect data for.process
module evaluation (Projects 6-9; these periods would also be used for the
vertical profiles (Project 2) and subgrid variability (Project 3) studies.
The intensive periods would be of two-month duration each and scheduled
to sample the important seasonal contrasts. At most, two intensives
would be scheduled in any year. A two-month intensive period was
considered superior to a one-month period for several reasons:
. the longer duration increased the probability of capturing appropriate
deposition episodes for evaluation,
. fewer intensives during a year means less stress on the personnel
running the field activities, and
. a longer period between intensives means that the data from the
previous intensive can be more thoroughly analyzed to support the
final planning of the next intensive.
4
-------
The only disadvantages to the longer and fewer intensives were that a
given season may be sampled only once or that some seasons may not be
sampled at all. In any case, seasonal differences in synoptic patterns
must be considered in scheduling intensives. Spring and autumn seasons
offer the greatest opportunity for frontal precipitation and cyclonic
storm systems. Summer affords opportunities to sample convective episodes
and the warmest temperatures. Winter presents the lowest temperatures
and solar irradiances, and a high probability for solid phase
precipitation.
Because of resource limitations, the process module evaluation field
studies may not be funded. Therefore, field study design for the vertical
profiles and subgrid deposition variability studies should be developed
independently of the module evaluation field studies.
1.2 Specific Background for This Project
The daily surface network described 1.n this DQO is essential for the
model evaluation program. The surface network is actually COmposed of
several networks sponsored by different organizations: the EPRI OEN
(Operational Evaluation Network), the OME APIOS Network, the AES CAPMoN
and the EPA ME (Model Evaluation)-35 Network (co-located with the NTN
Dry Deposition Network). Each should follow a compatible Protocol for
measuring deposition, air quality, and meteorology. The combined surface
network should be of national scale. Most of the regional-scale models
focus on the eastern half of the United States and southeastern Canada
because this is where there are known impacts and because deposition
patterns in the northeast are important to an assessment of effects. As
a result, a higher spatial density of sampling sites in this region is
warranted. Subgrid information, obtained at clusters of sites spaced
less than about 80 km, cannot be compared directly with model predictions.
However, such measurements are essential to assessing the representa-
tiveness of standard network sites and is discussed in the DQO for
Proj ect 3.
1.3 MesoSTEM Evaluation
The model evaluation field study program offers the opportunity to evaluate
models other than RADM and ADOM. EPA plans to adapt its efforts to also
field evaluate the Sulfur Transport Eulerian Model with a predictive
mesoscale transport driver; this version is named IIMesoSTEM.1I
All of the projects and DQOs relate to MesoSTEM evaluation, except this
one. That is because the spatial scale of the observations is too coarse.
5
-------
2.
Clients' Statement of the Problem
2.1 General Statement of the Problem
The clients (EPA, EPRI, OME, and AES) desire an operational evaluation of
the acidic deposition models, RADM and ADOM. There are presently no
acceptable monitoring data bases for an operational evaluation of RADM
and ADOM. Also, there is not agreement among evaluators and modelers on
the methods and procedures of comparing model predictions of acidic
deposition to future network monitoring results. Furthermore, there are
no acceptable data bases for diagnostic and mechanistic evaluation of
these models and their process modules (see Section 1.1 and Sarchet 1986
for definitions of the terms operational, integrated, diagnostic and
mechanistic evaluations). To address these problems, the clients sponsored
workshops to develop designs for a field measurement program to provide
the necessary information for evaluation. Workshop participants were
asked to provide answers to the following questions for each of the field
projects:
. What are the model/field variables to be compared?
. What are the spatial and temporal scale averages to be used
comparison?
. How will the comparisons be performed?
. How will model performance be judged?
Specific information on the inputs and outputs of the models was provided
to the participants by the modelers as a basis for developing answers to
these questions. Table 1. summarizes the temporal and spatial character-
istics of the input and output parameters for the various modules of RADM.
in the
There was a consensus among the participants at both the Model Evaluation
Protocols Workshop and the Field Study Design Workshop that the surface
monitoring network is required for an operational evaluation of the RADM
and ADOM. Field data on the chemical species in Table 1 that are involved
in wet and dry deposition must be obtained for a minimum period of two
years to perform operational evaluations of the models.
2.1.1 Sources of Error
The operational evaluation of the models must recognize and manage the
following sources of error:
2.1.1.1.
Representation of Fundamental Processes in the Model.
Those sources of error will be identified through four field study projects
(6-9) identified in Section 1.1 of this document.
The errors due to numerical solutions and computation will not be treated
by any of the nine projects identified in Section 1.1 of this document.
6
-------
TABLE 1.
----------------------------------------------------------------------
INPUT AND OUTPUT VARIABLES OF RADM.
----------------------------------------------------------------------
Transcort/Discersion
Incuts
3-Dimensional wind field
Temperature
Pressure
Pollutant or tracer concentrations
Outcuts
Inter-grid fluxes
Layer-average ccncentrations (15)
Gas Phase Chemistry
Incuts
Pollutant concentrations
Photolysis rates
Meteorology
Outouts
Layer-average concentrations and
chemical conversion rates for:
S02' 03' HOOH, HN03' HCHO. NO.
+ 2-
N02' VOC. PAN. NH3' NH4 . S04 .
and N03-
Wet Decosition
Incuts
Temperature
Precipitation
Pressure
Pollutant concentrations
Outouts
Vertical redistribution, chemical
conversion and wet deposition
fluxes of: S02' HN03' HOOH. 03'
+ + 2- -
NH3' H , NH4 ' S04 . and N03
7
Temcoral
Soatial
1 hour
80 Ian
II
II
II
II
a
II
1 hour
80 Ian
..
..
1 hour
80 Ian
a
..
..
a
1 hour
80 Ian
1 hour
80 km
"
II
..
"
..
"
1 hour
80 Ian
-------
Drv Decosition
Temcoral
Scatial
Incuts
Surface characteristics
Surface meteorology
Pollutant concentrations
1 hour
.
80 km
.
.
.
Outcuts
Surface fluxes of:
S02' HN03' 03' HOOH. NH3. NO. N02'
and partlcles
1 hour
80 km
-------------------------------------------------------------------------
2.1.1.2 Input Data
These sources of error are to be treated through field investigations.
The errors arise in estimating emissions (Project 4) and in estimating
fluxes of pollutants from outside of the modeling domain (Project 5).
Input or.emissions data for NH3' VOCs. H2CO. and organic acids are
particularly unreliable or unavailable. In order to model H+, ion balance
is required; this means accounting for emissions of cations not specified
in Table 1. There are input errors in the meteorological variables.
Many of these factors will be treated through model sensitivity studies.
2.1.1.3 Monitoring Data
. Sources of error in the monitoring data are to be treated by field
investigations (Projects 1 and 2). The errors which arise in network
monitoring are mostly traceable to non-ideal siting, sampler and instrument
performance, sample transport. and analytical laboratory performance.
The errors that occur in the surface network are a concern to this project
and DQO. Errors also arise in the spatial averaging (interpolation)
from a few stations; this is discussed in Section 2.1.2.2. Another
important source of error in the surface network is subgrid variability
(Le.. spatial variability within a grid-cell); it is treated in Project 3.
2.1.2 Model Operational Evaluation Problem
The following discussion is presented to provide information to the
implementor about the type of comparisons that may be made. The evaluation
of the models against the monitoring data base is not part of this project.
At the clients' workshops, no clear decisions were reached on the methods
that should be used in making the comparisons or on the criteria that
should be used in judging model performance. Specification of the
evaluation methods and comparison criteria will be presented in a future
task. However. these methods for comparing modeled and measured variables
applicable to this task are expected to be used in this model evaluation.
8
-------
2.1.2.1 Conventional Point to Grid Comparisons
This approach to model evaluation is based on the suggestions of the AMS
Workshop on Dispersion Model Performance (Fox 1981). This approach has
been applied to the evaluation of regional-scale models 1n the Regional
Air Quality Model Assessment and Evaluation Project sponsored by EPRI
(Ruff et al. 1984) and in the model evaluation work of the MOl (MOl 1982)
In this approach, the difference, di, between an observed variable, Coi'
and. the modeled counterpart, Croi, is the basic Quantity used in asSesslng
the performance of a model.
Assuming that all processes in the atmosphere are completely deterministic,
a perfect model driven by perfect initial and boundary condition data
would yieTd di = 0 at every point of comparison in the model domain. In
reality, however, this level of agreement is impossible. Real models
entail numerous compromises in terms of their spatial and temporal
resolution and in the degree of realism to which the various physical
and chemical processes occurring within the atmosphere can be represented.
Additional errors are introduced by the data used to drive the model.
These are never sufficient to accurately describe initial and boundary
conditions. Given these considerations, both the MOl and EPRI model
evaluation projects used the fOllowing criteria for model .perfection"
(Ruff et al. 1984):
(d> = 0, where d is the bias.
ud' the root mean square error, is small compared to the standard
deviation of Cro, Uc .
m
di is not a function of the model input parameters. (diCroi> = 0 is
a necessary, but not sufficient condition, for this state.
The definitions of these properties are:
(d> = (l/N) ri di
di = Coi - Cmi
ud = [ri (di - (d»2/(N - 1) ]1/2
Uc = [ri (Croi - (Cm»2/(N - 1) ]1/2
m
(di~i> = (l/N) ri di~i
where ~i and Coi are determined for some averaging time of interest.
In addition to these quantities, the EPRI study prepared scatterplots of
Coi versus ~i' showing the perfect fit line, the correlation coefficient
9
-------
(Coi - (Co») (Cmi - (Cm»/{ ri(Coi - (Co»)2
ri (Smi - (Sm»2 }1/2
and the mean bias, (d). Graphical and statistical comparisons can also
be made of the frequency distributions of Coiand Smi using the
Kolmogorov-Smirnov tests for agreement. .
r - r.
- 1
The principal advantage of this approach to model evaluation lies in its
conventionality. Assuming that the differences, di, are normally
distributed, the various statistical tests that can be applied to the
data are well established, and their meanings are fairly clear. The
principal disadvantage of the approach lies in the problem of comparing
grid-averaged model data with observations made at specific points.
Usually, there will be at most only one observation within any grid and
there is no reason, a priori, to expect a single-point measurement to
correspond to the grid-averaged value. One method of dealing with this
problem is to restrict comparisons to those stations that are thought to
be representative regional-scale phenomena. In the EPRI evaluations,
these stations were identified by autocorrelation analysis on hourly
averaged concentration data. The hypothesis was that high correlations
between I-hour or 3-hour ~easurements indicated sites primarily influenced
by large-scale processes; whereas low correlations indicated sites strongly
influenced by local sources.
Another problem with this method of model evaluation is that the
statistical measures of model performance are sensitive to slight spatial
misalignments between the observed and modeled fields. For example, in
areas with large gradients, large differences (dis) can result from slight
discrepancies in the shape and placement of the modeled and observed
fields. These discrepancies could, in reality, be inconsequential either
in terms of the intended use of the model or in terms of the actual spatial
resolution of the observational network. This problem can be addressed
through subjective comparisons of objectively analyzed and model-derived
fields, but this approach is subject to the weaknesses inherent to
subjective judgments.
2.1.2.2 Pattern Comparisons
Some of the problems of paint to grid Comparisons can be overcome by
pattern Comparisons. This method of comparison is also more consistent
with the way models will be applied in policy analysis. As indicated in
Section 1 of this DQO, EPA is especially interested in the ability of
the models to compute mean annual and seasonal deposition to areas on
the order of the size of states, or portions of states, and not necessarily
to specific receptor sites. In order to compare observed and mOdeled
deposition fields or patterns, however, a method must be found for
converting individual point measurements of deposition or ambient
concentration into an estimated field. One Possible technique is kriging
(Barnes 1980). Kriging is a data interpolation technique that uses a
weighted moving average to estimate the value of a function between points
10
-------
where it is actually known. The method can be applied to both model-
generated data and observational data. Kriging is expected to be applied
to both types of data for intercomparison of results.
The basic kriging method, called .simple kriging-, is based on two
assumptions:
. The expected value of the function is constant everywhere in the
domain of interest (i.e.J there are no trends in the data).
. The difference between data values at two points is a function only
of the distance between them (i.e., the variance is isotropic).
The procedure requires estimating the variogram function, which describes
how the variance of the differences between values at two points 'changes
with the distance between them. If the simple kriging assumptions hold,
the variogram is not difficult to estimate, provided that there are
sufficient data.
Kriging has several advantages that make it attractive for analyzing
deposition and concentration fields. One of these is the ability to
estimate the variance or the "kriging error- at each point for which an
estimate is made. The kriging error is defined as the difference between
the estimate and the true, unknown value. It can be used to select optimal
locations for additional measurements, or it can be used to put a kind
of "error band" or .confidence banda on the location of an interpolated
isopleth. This latter ability is particularly attractive for model
evaluation purposes. It enables the data analyst to make an objective
decision as to whether or not differences between observed and modeled
deposition or concentration patterns are significant compared to
uncertainties in the ability of the observations to define these fields.
An additional advantage of kriging is that the system of equations used
to derive the optimal weights for the weighted moving average does not
depend on the data values. Instead, the system depends on the variogram
and the relative geometry of the data points. This lack of dependence
on data values means that single, large values do not overly influence the
estimates. If the data contain information on measurement variance or
uncertainty, this information can be included in the kriging system so
that less reliable data are given less weight.
Finally, kriging enables estimates, as well as the uncertainties in these
estimates, to be averaged over grid blocks of varying sizes and over the
entire field.
Kriging is certainly no panacea. It has limitations. The most serious
of t~ese are obviously the fundamental assumptions of constant expected
value and isotropy. Although these assumptions can be relaxed somewhat,
in practice, it is clear that deposition data from the eastern United
States with their strong spatial gradients, could violate at least one of
them. If the data do not satisfy even the relaxed conditions of simple
kriging, the technique of universal kriging might be applicable. Universal
11
-------
kriging can account for trends in the data as long as these trends are
gentle enough to be represented by a low-order polynomial. Data drift
can then be accounted for by various procedures. As pointed out by Barnes
(1980), it is not valid to remove data drift by fitting a least squares
surface to the data and calculate the variogram on the residua1.s.
Even though the data set may not meet all of the criteria for its proper
application, it can still be used to produce reasonable-looking fields.
The problem appears to be in interpreting the meaning of the estimates
of kriging error. If the kriging assumptions are not met, the validity
of the estimates of interpolation error are questionable. This situation
would call into question the assumption that the estimates of kriging
error can be used as an objective test for agreement between observations
and model results. Work is currently underway to assess the ~ignificance
of this problem: and at the present time, not all individuals who have
investigated the use of kriging to analyze deposition and concentration
fields are convinced that the technique is completely valid.
Kriging does not place a great deal of restrictions on data collection
protocols. The technique can be used to analyze individual events or
long-term averages. Kriging is not affected by data gaps at individual
measurement locations, as long as the gap is not sufficiently long to
affect the sample average. The main demand that the technique places on
data collection is that the number of data points (i.e., the spatial
coverage) must be sufficient to adequately define the variogram (in the
case of simple kriging) or the spatial co-variance structure (for universal
kriging). The spatial density of the data must also be sufficient to
meet the accuracy requirements for a given evaluation objective since
kriging error, in a particular region of the domain of application, will
decrease with increased density of data in that region.
2.1.2.3 Principal Components Analysis
Principal components analysis (PCA) is another technique that has been
suggested for use in evaluating model performance. It is essentially a
technique for multivariate analysis of complex systems that are
characterized by a large number of interdependent variables. It can be
used for both temporal and spatial data analysis. The basic idea of PCA
is to find a linear transformation that will change the original set of
correlated variables into a set of independent, uncorre1ated ones. The
key for finding this transformation is to diagonalize the correlation
matrix formed from the original data set: for if a set of variables are
independent, their correlation matrix will be a diagonal one: namely,
the identity matrix. This transformation is done by finding the eigen
values and eigen vectors of the correlation matrix. The ordering of the
principal components (first, second, third, etc.) is given by the magnitude
ofHthe associated eigen values. It can be shown that the first principal
component is that linear combination of variables that explains the
greatest amount of variability in the original data. The second principal
component explains the next largest amount of variability, and so forth.
Usually, most of the information contained in the original data can be
explained by a small number of principal components.
12
-------
It is the significance of the ordering of the principal components that
makes PCA attractive for model evaluation. In such an evaluation, PCA
would be applied to modeled and observed data at a series of points. If
the model is an accurate representation of the physical and chemical
system, then the principal components, or at least the first two or three
derived from the model results, should be the same as those determined
from the measurements. .
A principal concern about PCA as a model evaluation tool is uncertainty
about the robustness of individual principal components. The technique
is definitely sensitive to outliers. Thus, differences between the model
and the observations that might not have any practical significance,
could completely change the definitions of the principal components. An
investigation of the sensitivity of the technique to these factors is
required before it could be applied in practice. A question related to
robustness is how to interpret the results. It is clear that if the
significant principal. components for a model-produced data set and for
the actual observations are identical, then the model is probably doing
something right. But what does it mean if the definitions of the principal
components are different? What is an objective measure of the amount of
disagreement?
The.demands that PCA places on a data set8are somewhat more severe than
those made by kriging. PCA attempts to model the temporal response of
the key variables in a system at specific points in space. Thus, the
technique requires simultaneous time-resolved measurements of each variable
to be included in the analysis at every point of interest.' Excessive
data gaps or frequent periods in which the measurements are close to the
detection limit of the instrument, a situation which results in little
variation or signal being introduced into the data record, will result
in increased instability in the makeup of the derived components. Such
instability, of course, reduces the confidence that one might have in
the significance of the principal components. In judging the sufficiency
of a given data set for PCA, one rule of thumb states that the number of
degrees of freedom per variable should be more than 30 and, if possible,
equal to 100 or more. In addition, the temporal resolution of the data
must resolve the temporal behavior of the major processes affecting the
response of the system. In terms of data completeness, PCA requires at
least a 90% valid data capture rate.
2.2 Specific Statement of the Problem
In the preceding section, a general discussion of problems that are
expected to be encountered in the regional-scale model evaluation program
were presented. In this section, specific discussions pertaining to the
monitoring network(s), as described in this OQO, are presented. In the
Workshop on Model Evaluation Protocols (Pennell 1986), participants called
for a model evaluation network with 24-hour sampling of precipitation
chemistry and 12-hour sampling of aerometric variables. However, the
Canadian networks (CAPMoN and APIOS) are presently monitoring with a
24-hour average sampling protocol. Although the RFP for EPRI's Operational
13
-------
Evaluation Network (OEN) (RP2434-4) contains an option for 12-hour
sampling, it is likely that EPRI will not exercise it. Also, EPA does
not have the resources to fund 12-hour sampling and still maintain the
spatial coverage of the planned network. To assist in attaining uniform
protocols, the EPA ME-35 network will perform 24-hour average sampling
for model evaluation.
In eastern North America, wet deposition and air concentrations will be
monltored by NAPAp.s National Trends Network (NTN), EPRI.s Operational
Evaluation Network (OEN), and the Canadian networks (CAPMoN and APIOS)
The major difference in the operation of the networks is the sampling
period. The sample averaging period for the NTN is one week, whereas
the other networks will be following a daily sampling protocol, although
presently there are differences. Networks with a common or highly
compatible protocol are required to provide information on the wet
deposition and surface concentrations of key acidifying materials.
The purpose of this project is to monitor and to produce a data base of
24-hour average wet deposition and air concentrations at 35 of the EPA
NTN Dry Deposition sites (including 7 MAP3S Precipitation Chemistry Network
sites) using sampling methods, analyses, and QA/QC that are highly
compatible with those of EPRI, OME, and AES (A workshop will be convened
in September 1986 to negotiate common samplers and protocols or to
establish procedures for intercomparison experiments). The data base
will be used to evaluate the performance of the models. The recommended
variables to be compared in the evaluation were identified in both the
field studies (Barchet 1986) and model evaluation protocols (Pennell
1986) workshops. These 24-hour average variables are specified in
Table 2.
The "second priority" variables indicated in Table 2 reflect the clients'
recognition that dependable, inexpensive samplers and analyzers have not
been demonstrated. If suitable methods become available, those species
will be moved into the "first priority" category. Some+"first priorityU
variables, such as the metal cations, are needed with H for ion balance
calculations.
Those variables in Table 2, and additional ones, will be monitored with
an averaging time of about 3-6 hours at a subset of about 3-6 of the ME-
3~ stations during the intensive periods. However, those experiments are
not included in this project and DQO (e.g., see the DQO for Project 9,
"Evaluate Gas Phase Chemistry Module").
The NTN will consist of 150 wet deposition monitoring sites operated by
the USGS and up to 100 selected air concentration monitoring sites (NTN-
Dry) operated by EPA/EMSL. The latter sites will provide estimates of
weekly wet and dry deposition, respectively. The first 35 sites of the
100-station NTN-Dry that EMSL will install are in the northeastern United
States. The number and location of these sites have been selected, with
consideration of the existing Canadian CAPMoN and APIOS and EPRI OEN
sites, in order to optimize the capability of the combined networks to
14
-------
TABLE 2. VARIABLES SPECIFIED BY THE CLIENT TO BE MONITORED FOR
MODEL EVALUATION
------------------------------------------------------------------------
------------------------------------------------------------------------
Gases (24-hour average):
First priority: S02' HN03' 03' NH3'
Second Priority: NO, PAN and H2CO.
Particles (24-hour average):
N02' and HOOH.
+ + 2- - -
H , NH4 ' ~04 ' N03 ' and C1 .
metals (V, Mn, Fe, As, Se, Sb, Hg, Pb).
metals (Na, Mg, K, Ca),
2- - + - 2-
S04 ' Cl , NH4 ' N03 ' and C03 .
Precipitation (24-hour average):
First priority: H+ (free), conductivity, S042-, N03-' NH4+' Cl-,
Na+, K+, Ca2+, Mg2+, HOOH, and S(IV).
First priority: (2 pm
Second priority: (2 pm
2-10 pm
Second priority:
Thi rd pri ori ty:
H2CO.
metals (V, Mn, Fe, .As, Se, Sb, Hg, Pb).
Meteorology (3-hour average):
Surface wind speed and direction, temperature, pressure,
precipitation, relative humidity, and insolation.
-----------------------------------------------------------------------
resolve significant spatial patterns and to achieve an acceptable level
of uncertainty (e.g., a goal of z30% on the seasonal averages). Kriging
techniques described in Section 2.1.2.2 were used in the optimization
analysis.
This project DQO, for 35 EPA Model Evaluation (ME-35) stations (in addition
to the 35 CAPMoN, APIOS, and OEN stations) relates only to the 35 EPA/EMSL
NTN Dry Deposition (NTN-35 Dry) sites located in the northeastern United
States. EMSL will establish its 35 weekly air concentration monitoring
sites through a separate DQO and contract (see Appendix A and Figure 1
for their approximate locations). The purpose of this task is to establish
at the NTN-35 Dry the capability to obtain the variables identified in
Table 2 without perturbing the I-week NTN wet and dry protocol. The
NTN-35 Dry will measure and record continuously the meterological variables
and 03 and will provide to the ME-35 the quality assured magnetic tape
copies of these variables for each station. However, the ME-35 will
operate separate gas samplers, particle samplers, and precipitation
samplers. Each model evaluation case is expected to be 2-4 days. This
15
-------
+
+
<>
+
".
. - "EPA. Var
o - EPA. I1E-35
o . - EPRI, 081
.1 - HY - DEC
+ - C4nadfan
X - Proposed HTN
o - Proposed Canadi an
I
.~
I
I
I
r
I
i
i
I
I
I
I
I
.
!
.
i
I
I
I
i
i
FIGURE 1.
Model Evaiuation Network Sites
evaluation plan provides for the integration through the diurnal cycle
while preserving the major featur~s of the air concentration and chemical
deposition patterns. .
The clients recognize that research monitoring coordinated among several
networks and funding agencies has not been previously attempted.
Therefore, an intensive QA/QC program is required. The Toronto QA workshop
participants recommended that one site in each of the four networks be
designated as an inter-network comparison site, where samplers from each
16
-------
of the networks using their respective protocols would be collocated
(Olsen 1986). Thus for the EPA, three additional sites are needed.
Duplicate samples must be obtained at each of the inter-network sites,
as well as at 10: of the other sites. Interlaboratory comparisons will
be required also. Revision of the DQO is expected after the review of
the initial data base and sampler performance records. Common start and
stop times for the daily sampling are also needed.
With respect to microscale siting decisions, various site selection
criteria have been developed for selecting deposition and aerometric
monitoring sites that are reasonably free of influence from local sources.
It is presumed that sites chosen according to these criteria will be
representative of grid-scale averages; however, the truth of this assertion
can never be proven a priori. In order to examine the question of how
representative a single measurement might be of a grid-scale average, a
subgrid variability study will be conducted as part of the model evaluation
program. This study will be summarized in a separate DQO (see Project 3,
"Determine Subgrid Variability'I).
3.
Clients' Descriction of the Acclication of the Product
The product of this project is a quality assured regional acid deposition
monitoring data base of 24-hour average wet deposition and air concen-
trations. The EPA, EPRI, OME, and AES evaluators will use that data
base to compute monitoring station time series, deposition patterns, and
principal components. The evaluators will compare those results with
RADM and ADOM predictions.
For the first year1s operation, all data will be released to the model
developers for use in testing and improving model performance as soon as
they have been quality audited and after a blind evaluation of the models
for selected events during that year has been performed. A comparison
of the revised models predictions against the first year1s data will
also be conducted to establish the level of performance improvement, if
any. The second year's data, however, will be quality audited and
sequestered for use in a hands-off model evaluation exercise.
Major milestones and related reports that depend on this project are:
Qili..
Title
03/89
Report on blind operational and diagnostic evaluations of RADM,
ADOM and MesoSTEM against first 6-months FY88 benchmark data
base and results. .
09/89
Report on blind operational and diagnostic evaluations of RADM,
ADOM and MesoSTEM against second 6-months FY88 benchmark data
base and results.
06/90
Report on operational and diagnostic evaluation of revised
RAOM, revised ADOM and revised MesoSTEM against all FY88 data.
17
-------
12/90 .
12/91
Report on the blind operational and diagnostic evaluation of
revised RADM, revised AD OM and revised MesoSTEM against all
FY89 data.
Final report on operational and diagnostic evaluations of RADM,
ADaM and MesoSTEM.
4.
Clients' Constraints of Time and Deliverables
For the clients to deliver the products in Section 3 on schedule, this
project must provide the deliverables according to the following schedule:
Date
01/87
04/87
08/87
10/87
03/89
06/89
09/89
03/90
04/90
06/90
09/90
5.
Titl e
Approved work plan with SOPs and QA/QC plan.
Begin installation of model evaluation (ME-35) monitoring
network.
Pre-monitoring QA assessment report.
OEN and ME-35 Network operational.'
QC report for FY88 data.
Quality assured data base for FY88.
(Released to modelers.)
Quality audit report for FY88 data base.
QC report for FY89 data.
Network dismantled.
Qua 1 i ty assured data base for FY89. (Sequestered from mode 1 ers.)
Final quality audit report.
Constraints on Budget
Planning budgets are confidential. This section will be completed upon
award and negotiation of revisions to this DQO.
6.
Imolementor's Discussion of Alternatives and Selection of Aooroach
6.1 General Approach
None. There are no alternatives that will equal the cost effectiveness
of surface monitoring.
18
-------
6.Z Spatial Alternatives
None. The clients have specified the site locations to be the EPA NTN-35
Dry. Specifications for the OEN, CAPMoN, and APIOS site locations have
also been established. The candidate sites are shown in Figure 1 and
listed in Appendix A.
6.3 Temporal Alternatives
None. The clients are specific (Table Z) in stating the requirements; they
are cost effective for the client's goals. In order to assure uniformity
in the sampling, the clients need to agree on a common start and stop
time (common GMT) for the daily sample collections, e.g., start after
sunrise, but before the photochemical cycle is established.
6.4 First-Priority Gases and Fine Particles (Z pm)
Filter-pack and diffusion-denuder type samplers are the most cost-effective
approach to monitoring the gases and fine particles identified in
Table Z. The status of those samplers is:
6.4.1 Transition-flow reactor sampler (adaptation of Canadian filter
pack)
. Successfully field-demonstrated in 1985 by EMSL and ASRL for HN03'
NH3' SOZ' NOZ' fine particulate N03- and S04Z- (Knapp et al. 1986).'
. Successfully demonstrated in the California Nitric Acid Inter-
comparison Study (9/85) for HN03' NOZ' NH3' fine particulate N03-
+
and NH4 (Ellestad et al. 1986).
. The TFR sampler is highly compatible with the Canadian filter pack;
it consists of a cyclone (050 = Z pm), followed by a tube (the TFR)
lined with films reactive for HN03 and NH3' followed by a sampler
similar to the Canadian filter pack. The TFR permits HN03 and NH3
to be monitored without evaporation biases.
. In TFR, HN03 is collected on nylon film liner and extracted and
analyzed as N03- by IC. NH3 is collected on Nafion film liner and
+
extracted and analyzed as NH4 by IC.
. In filter pack, S04Z-, N03-, NH4+ are collected on Teflon filter,
extract;d and analyzed by IC. HN03 is collected on Nylon filter,
etc. NH3 is collected on oxalic acid, extracted, and analyzed as
NH4+ by IC. SOZ is collected on KZC03 coated filter, extracted,
oxidized, and analyzed as S042- by IC. NOZ is collected by TEA-coated
filter, extracted, and analyzed as N02- by IC.
19
-------
. TFR presently operates at three EPA prototype sites.
6.4.2 Canadian filter pack
. Successfully field-demonstrated in 1985 by EMSL for HN03 plus fine
particulate N03-' NH3 plus fine particulate NH4+, S02' and S042-.
. Successfully demonstrated in the California Nitric Acid
Intercomparison Study (9/85) for HN03 plus fine particulate N03-'
NH3 plus fine particulate NH4+, S02' and S042-. However, estimates
of HN03 were 20-60: greater than spectroscopic measurements.
. Presently operating in Canadian networks and at six EPA prototype
sites. Designated by EPRI for its Operational Evaluation Network.
.6.4.3 Denuder-difference sampler
. Not successfully field-demonstrated by EMSL or ASRL for HN03' NH3'
- 2-
S02' N02' fine particulate N03 and S04 .
. Successfully demonstrated by four independent laboratories in the
California Nitric Acid Intercomparison Study (9/85) for HN03 and fine
particulate N03-.
6.4.4 Annular-denuder sampler
. Not successfully field-demonstrated by EMSl or ASRl for HN03' NH3'
- 2-
S02' N02' fine particulate N03 and S04 .
. Not successfully demonstrated in the California Nitric Acid
Intercomparison Study (9/85) for HN03 and fine particulate N03-.
Inter-laboratory variance was large; average values for HN03 and
particulate N03- were about 20-30% below other methods that exhibited
good agreement with spectroscopic measurements of HN03.
Best sampling approach: either the TFR sampler, the revised TFR, or the
Canadian filter pack. The clients propose to convene a workshop in
September 1986 to adopt common samplers and protocols or intercomparison
experiments. The method of choice of EPA is the TFR sampler. It is
proven for all of the first-priority species, except HT; 03 will be
provided by the NTN-35 Dry. The TFR sampler is derived from the Canadian
filter pack and is highly compatible with the Canadian and EPRI networks.
The TFR sampler is not being commercially produced; however, descriptions
of theory and design are available.
6.5 Second-Priority Gases
20
-------
6.5.1 H2CO
Formaldehyde will be collected by Waters Associates Sep-Pak C18 cartridges
coated with 2,4-DNPH, followed by HPLC of derivatives. (Kuwata et al. 1983).
6.5.2 PAN and HOOH
PAN and HOOH are not planned for routine sampling; no routine cumulative
sampling 1s yet demonstrated. However, HOOH should be implemented at as
many sites as possible.
6.6 Second-Priority Particles
Particles are collected with manual dichotomous sampler on Teflon filters.
. + + 2+ 2+
Partlcles are extracted and analyzed for Na , K , Ca , and Mg by AA
or IC. Other metals are analyzed by Inductively Coupled Plasma Emission
Spectroscopy (ICPES) or x ray fluorescence; other soluble cations and
anions are analyzed by IC.
6.7 Precipitation
Precipitation shall be collected with the Aerochem sampler using the NTN
sampling and analysis protocol. Changes in sampling and analysis protocol
may be anticipated to achieve compatibility with EPRI, OME, and AES (see
the end of Section 6.4). A separate collector is required for each of
the species, S(IV) and HOOH, because of sample preserving techniques.
In areas where more than 20: of the annual precipitation is snow, Nipher
gauges shall also be used. QC diagnostic information (rain gauge readings,
sampler lid position, etc.) must also be recorded. The following analysis
methods apply:
. First priority precipitation species: H+ by pH electrode; acidity
by strong base titration; cations by AA and anions by IC; S(IV)
collected with TCM and analyzed by pararosaniline method.
. Second priority precipitation species:
for preserving sample.
. Third priority precipitation species:
no method yet demonstrated
by ICPES.
Only precipitation samples with volumes greater than 10 ml will be
analyzed.
6.8 Instrument Shelter
The following requirements apply: out-door, ~eather-proof; temperature
controlled (%5 C); volume approximately 2-4 m. Alternative: use the
NTN-35 Dry stations and shelters.
21
-------
6.9 Data Acquisition System
No data acquisition system is specified. Chart recorder of sampler flow
rate.
6.10 Partitioning of Variance
Variance will be partitioned into sampler preparation, sampler operation,
sampler handling, transport, storage, laboratory sample treatment, and
analysis. Techniques will incl~de collocated samples, field replicates,
and field splits. The analytical laboratory will perform laboratory
replicates, laboratory splits, analysis replicates, and analysis splits.
The variance will be managed to load the major contributions into sampler
operation and anal¥sis.
6.11 Partitioning of Bias
Based on recovery, bias will be partitioned into inter-network, matrix
effects associated with sample preservation, shipping, preparation, and
analysis. Techniques will include exchange of references, field matrix
spikes, laboratory matrix spikes, and analysis matrix spikes.
6.12 Data base: ADS.
6.13 Relationship to MesoSTEM
The spatial scale of the ME-35 (and OEN, CAPMoN, and APIOS) network is
about 250 km; this is too coarse to provide an evaluation of MesoSTEM,
which requires a scale finer than about 25 km.
7.
Data Quality Objective Statement for Imclementor1s Selected Accroach
7.1 Precision and Accuracy
7.1.1 Model:
not applicable to this project.
7.1.2
Surface measurements
Standard and research analytical procedures will be used for the
determination of the concentrations of species related to acidity.
Precision and accuracy goals for this task are summarized in Table 3.
7.2 Representativeness
Spatial locations are specified by the client(s). Approximate locations
of the NTN-35 Dry are listed in Appendix A and shown in Figure 1.
Temporal: 24-hour average samples for 2 years. This sampling period
does not capture the diurnal cycle, which modelers have requested.
However, the problem is diminished by constructing synoptic cases of 2-4
days length (by adding daily record) for the comparison with the models.
22
-------
------------------------------------------------------------------------
TABLE 3. PRECISION AND ACCURACY GOALS FOR EPA ME-35
(24-HOUR AVERAGE)
------------------------------------------------------------------------
Estimate
Ana1ytic~1) Ana1~t~ca(b) of Overall
Scecies ~ Accuracv Prec1s10n Precision
Gases (,uq/m3)
S02 1-200 .10: .10: .15%
HN03 1-20 II II II
NH3 1-20 " II u
NOx (NO, N02) 1-20 " " II
H2CO 1-20 " II It
HOOH 1-20 " " It
PAN 0-10 " II "
VOC ? " It "
Ozone 1-250 It " II
Fine Particles (JLO 1m3)
SO 2- 1-50 .10: .10: .15%
4
N03 - 1-20 u " u
NH4+ 1-20 " " "
C1- 1-20 " " "
H+ (will not be measured)
Coarse Particles
SO 2-
4
N03-
NH +
4
Cl-
CO 2-
3
Metals
(uq/m3)
1-50
1-20
1-20
1-20
1-20
10-3_10
Prec;citation
+
H (as pH)
conductivity
SO 2-
4
N03-
NH +
4
(uf'1/L) (c)
(Pi' em)
2-8
1-200
0.2-10
0.2-10
0.2-10
.10: .10: .15%
" " U
It " "
" " "
II II II
" " II
.5% .0.1 unit d5:
.5: .5: .15%
.10: .10: .15:
" " "
" " "
23
-------
Scecies
Precicitation (~/l)(c)
~
Ana1ytictl)' Ana1~t~catb)
Accuracy Preclslon
Estimate
of Overall
Precision
S(IV)
C1-
Na+
K+
Ca2+
Mg2+
Other Metals
0.2-10 .10% .10: .15%
0.1-10 " " II
0.1-10 " " "
0.1-10 " .. ..
0.2-20 " .. "
. .
0.2-20 " " ..
10-3_10 " " "
(a) The difference as a percentage of the reference or true va1ue~ or
the percent recovery of a spike.
(b) Expressed as a percent relative standard deviation of replicates of
ambient samples (or laboratory referen~es if insufficient ambient
samples).
(c) For sample volumes greater than 10 ml.
------------------------------------------------------------------------
7.3 Completeness
Completeness is defined as the number of valid data points acquired divided
by the total number planned. Required data capture rate (after quality
auditing) at each ME-35 station is 90% for each variable specified in
Table 3. Since the sampling frequency is once per day, at least 330
precipitation and air concentration filter samples must be obtained per
year to meet this completeness requirement.
REFERENCES:
Barnes, M. G. 1980. "The use of Kriging for Estimating the Spatial
Distribution of Radionuclides and Other Spatial Phenomena" TRAN-STAT:
Statistics for Environmental Studies, No. 13, PNL-SA-9051, Pacific
Northwest laboratory, P.O. Box 999, Richland, Washington
Ell estad, T. E.
Fox, D. G. .1981. "Judging Air Quality Model Performance", Bull. Amer.
Meteorol. Soc., 62, No.5, pp 599 - 609.
Knapp, K. T. 1986.
Kuwata, K. M., M. Uebori, H. Yamasaki, Y. Kuge, and Y~ Kiso. 1983.
"Determination of Aliphatic Aldehydes in Air by liquid Chromatography."
Anal. Chern., 55:2013.
24
-------
Ruff, R. E., K. C. Nitz, F. L. Ludwig and C. M. Bhumralkar. 1984.
Recional Air Oualitv Model Assessment and Evaluation. EA-3671, Electric
Power Research Institute, Palo Alto, California.
United States-Canada Memorandum of Intent (MOl) Work Group 2. 1982.
Final Report on Atmospheric Sciences and Analysis. Washington~ D.C. and
Toronto, Canada.
8. Summary
This DQO, with exceptions clearly indicated in Sections 6 and 7, meets
the goals expressed by the clients for the agreed upon approach.
.
9.
DOO Recommended bv
10.
DOO Accroved bv
25
-------
APPENDIX A. CLIENTS' NETWORK SITE LOCATIONS
Lat. Long. 1.0.
EPA NTN-35 Ory Network Sites
1. 35.900 -78.867 A $ 101 2
2. 35.950 -84.283 A $ 102 5
3. 41.350 -74.033 A $ 103 27
4. 44.383 -73.850 A $ 104 29
5. 40~783 -77.933 A S 105 34
6. 37.300 -78.000 CIS 106 10
7. 39.083 -79.567 81$ 107 14
8. 43.800 -72.000 83S 109 26
9. 42.733 -76.650 A $ 110 31
10. 41.100 -80.000 83$ 112 35
11. 40.500 -83.500 0 $ 114 39
12. 42.050 -84.033 C2S 115 40
13. 39.000 -76.900 0 $ 116 9
14. 40.300 -79.700 0 $ 117 12
15. 38.033 -78.533 82S 118 13
16. 38.750 -81.000 0 $ 119 15
17. 37.038 -81. 033 C2S 120 16
18. 37.067 -82.983 81S 121 18
19. 39.517 -84.717 A $ 122 19
20. 41. 000 -82.100 CIS 123 37
21. 43.000 -83.000 0 $ 124 38
22. 36.000 -81.000 Cl$ 126 3
23. 36.300 -86.500 0 S 127 6
24. 40.100 -76.800 CIS 128 8
25. 37.667 -84.967 81$ 129 20
26. 40.050 -88.367 A $ 130 23
27. 39.800 -86.500 0 $ 133 44
28. 44.200 -89.900 0 $ 134 36
29. 45.483 -69.650 CIS 135 47
30. 35.050 -83.417 81S 137 4
31. 38.733 -87.483 81$ 140 21
32. 40.000 -75.000 CIS 144 30
33. 41.700 -87.983 A $ 146 43
34. 40.067 -81.133 C2S 113 36
35. 43.000 -85.500 0 S 149 *
26
-------
Lat. Long. 1.0.
EPRI OEN Sites
36. 44.617 -68.967 U $ U01 *
37. 44.517 -72.867 U $ U02 *
38. 42.583 -72.533 U $ U03 *
39. 43.817 -74.900 U $ U04 *
40. 41.567 -75.983 U $ U05 *
41. 39.983 -82.017 U $ U06 *
42. 39.233 -82.467 U S U07 *
43. 41. 033 -85.317 U S U08 *
44. 37.867 -87.117 U $ U09 *
45. 44.933 -84.633 U $ U10 *
46. 38.133 -83.450 U $ U11 *
47. 35.783 -89.133 U $ U12 *
48. 35.717 -78.667 U $ U13 *
49. 32.050 -82.467 U $ U14 *
SO. 32.467 -87.083 U $ U15 *
51. 32.350 -90.283 U S U16 *
52. 44.708 -88.624 U S U24 *
53. 46.233 -91.936 U S U25 *
New York Department of Environmental Conservation Sites (if supported)
54. 43.300 -74.100 N $ N01 *
55. 42.200 -74.800 N $ N02 *
56. 44.600 -75.400 N $ N03 *
57. 42.100 -77.000 N S N04 *
58. 42.100 -79.400 N S N05 *
Canadian Sites (OME Recommended)
59. 44.1' -77.8 0 $ 002 *
60. 44.2 -81.0 0 $ 005 *
61. 42.8 -81.5 0 S 001 *
62. 44.2 -65.8 0 $ 007 *
63. 46.8 -71.6 0 $ 008 *
64. 45.2 -72.5 0 S 006 *
65. 49.0 -74.6 0 $ 011 *
66. 45.8 -77.3 0 S 004 *
67. 45.3 -79.5 0 S 003 *
68. 49.5 -82.6 0 $ 010 *
69. 46.7 -84.2 0 $ 009 *
27
-------
Lat.
Long.
I.D.
Top Priority Sites for Any Addit10nal NTN Sites in FY87 (Network=100)
70.
71.
72.
73.
34.500
38.500
41.500
39.900
-78.500 F1S
-76.000 F2S
-72.200 F3S
-78.800' F4$
136 1
F02 *
F03 *
F04 *
Suggested Priority Sites for Additional Canadian Sites
74.
75.
76.
77.
46.300
44.400
43.500
43.100
-74.800
-79.000
-81.400
-79.300
FC1S
FC2S
FC3S
FC4$
FC1 *
FC2 *
FC3 *
FC4 *
28
------- |