vvEPA
United States
Environmental Protection
Agency
Office of Research and
Development
Washington, DC 20460
EPA/600/R-92/097
February 1992
Fourth Annual
Ecological Quality
Assurance Workshop
-------
EPA/600/R-92/097
February 1992
National Water Research Institute
Burlington, Ontario, Canada
Sponsored by:
U.S. Environmental Protection Agency
Environmental Monitoring Systems Laboratory
Office of Research and Development
Cincinnati, Ohio, U.S.A.
Environment Canada
Conservation and Protection
National Water Research Institute
Burlington, Ontario, Canada
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
ANDREW W. BREIDENBACH ENVIRONMENTAL RESEARCH CENTER
CINCINNATI, OHIO
O/) Printed on Recycled Papei
-------
EXECUTIVE SUMMARY
The first annual Ecological Quality Assurance Workshop sponsored by the
Environmental Research Laboratory in Corvallis was held in Denver, Colorado in 1988 for
the purpose of providing a forum for exchange of information among scientists from
government, academia, and the private sector in the United States and Canada on the
use of quality assurance principles and quality control techniques in the design and
conduct of ecological studies. Since then, each year the Organizing Committee has
endeavored to improve the workshop format and attract a larger and more diverse
audience. Consequently, attendance at the workshop has grown from 40 participants in
1988 to nearly 80 at this fourth workshop.
This year, the Ecological Quality Assurance Workshop was held at the U. S.
Environmental Protection Agency's Andrew W. Breidenbach Environmental Research
Center in Cincinnati, Ohio on February 26-28, 1991. The Workshop provided a forum for
scientists to discuss applications of statistical tools, data quality objectives, and other
quality assurance concerns in the context of ecological research. The Workshop agenda
included plenary sessions, in which invited papers were presented, and workgroup
sessions, which provided an opportunity for participants to discuss their experiences with
specific quality assurance applications in the areas of aquatic, terrestrial, and atmospheric
monitoring programs.
The Organizing Committee for this fourth workshop expected to advance ecological
research from a quality assurance standpoint by creating an environment in which
scientists would be encouraged to share their experiences and guidance in a format that
can be passed on to others. Speaking for the Organizing Committee, USEPA's Dr.
James Lazorchak, stated that "the principal goal for this fourth workshop was to develop
guidelines for application of statistical tools and data quality objectives in ecological
research for use by scientists in the U.S. and Canada." Dr. Lazorchak noted that this
goal was an ambitious one largely because the concepts involved are so new. He
emphasized that the broad goals for the workshop were secondary in importance to the
experience gained by workshop participants as they worked through the processes of
statistical study designs and setting data quality objectives.
IV
-------
Plenary Sessions
Ten invited papers were presented during two plenary sessions. Nancy
Wentworth, newly appointed Director of USEPA's Quality Assurance Management Staff,
provided the keynote address on the principles of total quality management in ecological
research. Other papers presented by authors from both the U.S. and Canada addressed
specific applications for data quality objectives, statistical tools in ecological monitoring
and comprehensive quality assurance programs for environmental monitoring. One
noteworthy paper addressed applications of quality assurance concepts developed at the
Third Ecological Quality Assurance Workshop to an aquatic monitoring program. Finally,
a paper entitled "Ecological Survey of Land and Water in Britain" by John Peters of the
United Kingdom Department of the Environment, Rural Affairs Directorate, was presented
as a poster display because world events prevented Mr. Peters from attending the
workshop. Mr. Peters' paper is included in its entirety in these proceedings.
Workgroup Sessions
Workgroup sessions on applications of statistical tools and the data quality
objectives development process were offered in three areas: aquatic monitoring, terrestrial
monitoring, and atmospheric monitoring. All workshop participants attended one session
each on statistical tools and data quality objectives, for the area of their choice. Each
session was led by a scientist with special expertise in the subject area. Although the
specific formats for the workgroup sessions varied at the discretion of the individual
leaders, each generally provided for examining the topic through specific case studies and
allowed the participants to draw from their own experiences in solving problems
encountered by others. Ample opportunities were provided for participants to bring
problems that they have encountered in their own research before their peers to obtain
guidance and assistance. At the conclusion of the workshop, each workgroup leader
presented conclusions and recommended guidelines for the topic derived from the
discussions.
Although each workgroup session examined different monitoring problems and
applications, some common themes were evident in the conclusions and recommended
guidelines offered by the workgroup leaders. These included:
The advantages of making a statistician an integral pan of the
study design team. Statistical tools were recognized by the
workshop as a fundamental building block of sound
monitoring program design. Participants agreed that,
irrespective of the type of program or its purpose, ensuring
appropriate use of statistical tools by consulting very early
with a qualified statistician, experienced in environmental
survey design, substantially increases the likelihood that the
eventual design will successfully and reliably answer the
research question or questions.
v
-------
The importance of conducting pilot studies to assist in
designing full-scale monitoring programs. Pilot studies were
cited in all workgroup sessions as a valuable tool for
developing stratified sampling schemes, ensuring
representative sampling, and refining the investigator's
conceptual model for the ecosystem. Although most
recognized that historical data can be used for similar
purposes, there was general agreement that the reliability of
historical databases is generally not known. Only through a
well-designed pilot study can a researcher be assured of
obtaining initial information regarding temporal and spatial
variability needed for full-scale design.
The limitations of existing ecological data. Workshop
participants cited a universal lack of quality control information
as an important limitation in the existing body of ecological
data and recognized that this condition has generally resulted
from the fact that ecologists are not usually trained in quality
assurance and statistics beyond an introductory level.
Moreover, workshop participants recognized that budgetary
and schedule constraints have acted to limit the utility of data
collected in the past and continue to restrict study designs
today. Consequently, the workgroup leaders agreed that, in
conducting ecological research, scientists should attempt to
collect as much useful information as is practical while in the
field.
With regard to use of the data quality objectives process in ecological monitoring,
workgroup participants generally agreed that the process provides a framework for
negotiating a study design, particularly where many interested parties and perspectives
must be satisfied. The workgroup leaders recommended that effective communications
and a willingness to accept input from all perspectives on the part of scientists are critical
to the success of the data quality objectives process. They also recognized that scientists
should not expect to anticipate the full range of concerns and confounding factors that
must be accounted for at the start of a study. Consequently, the data quality objectives
process should be viewed as a dynamic process that begins in the initial phases of study
design and continues throughout the study, as the initial questions to be answered are
re-examined and refined based on experience and the investigator's growing
understanding of the ecosystem.
VI
-------
The Fifth Annual Ecological Quality Assurance Workshop is planned for the winter
of 1992 and will be held in Canada. The organizing committee plans to develop an
agenda that builds on the results of this year's workshop and continues to develop
practical guidelines for application of quality assurance concepts in ecological research.
For information concerning the Fifth Workshop, contact Mr. Robert Graves, U.S.
Environmental Protection Agency, Environmental Monitoring Systems Laboratory -
Cincinnati, 26 W. Martin Luther King Drive, Cincinnati, Ohio 45268; or Mr. Robert Bisson,
Environment Canada, 867 Lakeshore Road, P.O. Box 5050, Burlington, Ontario L7R 4A6,
Canada.
vii
-------
ABSTRACT
The Fourth Annual Ecological Quality Assurance Workshop was held at the U.S.
Environmental Protection Agency's Andrew W. Breidenbach Environmental Research
Center in Cincinnati, Ohio, February 26 - 28, 1991. Manuscripts of invited papers and
reports of workgroup discussions, conclusions and recommendations are presented in
this proceedings document.
The purpose of this workshop was to provide a forum for interdisciplinary exchange
of ideas and resolution of issues associated with quality assurance (QA) in ecological
studies. The workshop served to bring together international representatives from both
government, academia and the private sector to discuss ways of improving the quality of
ecological studies. The papers and workgroup sessions were designed to meet the
following goals:
1. To foster the development of ecological QA concepts.
2. To foster the exchange of ecological quality management concepts and
techniques, along with their integration into ecological studies.
3. To assess the effectiveness of these activities in improving the quality of
ecological studies.
Workgroup topics centered on (1) the development of guidelines for use of data
quality objectives (DQOs) in field monitoring programs, and (2) the development of
guidelines for use of statistical tools in field monitoring programs. Participants in the
workgroups (workgroups were organized by ecological disciplines: terrestrial, aquatic and
atmospheric) were invited by their respective group leader to identify and develop
consensus views on important elements in the application of DQOs and statistical tools
to environmental field monitoring. Each workgroup presented a brief summary of its
discussions in a plenary session held on the last day.
Workgroup presentations covered a broad spectrum of QA topics; ranging from
total quality management, DQOs and statistical concepts to particular QA problems
encountered by specific programs.
VIII
-------
ACKNOWLEDGMENTS
The Organizing Committee for this Fourth Ecological Quality Assurance Workshop
included Dr. James Lazorchak and Mr. Robert Graves, of the U.S. Environmental
Protection Agency's Environmental Monitoring Systems Laboratory, Mr. Robert Bisson
and Mr. John Lawrence of the Canadian National Water Research Institute, Mr. Craig
Palmer of the University of Las Vegas, and Ms. Lora Johnson of Technology Applications,
Inc. The Committee wishes to acknowledge the efforts of Ms. Johnson, who supervised
the workshop logistics, Mr. Dennis M. McMullen of Technology Applications Inc., who
assisted with planning the workshop, and Ms. Allison Cook, Mr. John Willauer, Mr.
Michael Guill, and Mr. Michael Piehler, all of AScI Corporation, who served as rapporteurs
for the workgroup sessions. These proceedings were prepared by Ms. Jan Edwards, of
Edwards Associates, Falls Church, Virginia, with assistance from Ms. Johnson and AScI
Corporation of McLean, Virginia.
IX
-------
CONTENTS
DISCLAIMER NOTICE jj
FOREWORD Hi
EXECUTIVE SUMMARY iv
ABSTRACT viii
ACKNOWLEDGMENT ix
PLENARY SESSION A 1
Overview of Total Quality Management 2
Nancy W. Wentworth, P.E.
U.S. Environmental Protection Agency
Washington, D.C.
Data Quality Objectives for the National 10
Pesticides Survey: Evaluation and Results
Paul Black
Decision Sciences Consortium/ICF
Reston, Virginia
Lora Johnson
Technology Applications, Inc.
c/o USEPA
Cincinnati, Ohio
Harold Lester
ICF International, Inc.
Fairfax, Virginia
Planning Approaches in Canadian Field 24
Monitoring Studies
Ann J. Neary
Ontario Ministry of the Environment
Dorset, Ontario
-------
Statistical Thinking and Tools for Improving 44
Excellence in Ecological Projects
John A. Flueck
University of Nevada
Las Vegas, Nevada
Statistical Quality Control in Environmental 62
Impact Assessment: Roles and Objectives
A. H. El-Shaarawi
Environment Canada
Burlington, Ontario
Comparison of Alternate Approaches for 76
Establishing Quality Objectives for
Ecological Studies
C. J. Palmer, Ph.D.
B. L Conkling, Ph.D.
University of Nevada
Las Vegas, Nevada
PLENARY SESSION B 86
NOAA National Status and Trends Program - 87
Quality Assurance Program
Adriana Cantillo, Ph.D.
National Oceanic and Atmospheric Administration
Rockville, Maryland
Application of Quality Assurance Concepts from 93
the Third Ecological Quality Assurance Workshop
to a Long-Term Aquatic Field Assessment
Pilot Study
James B. Stribling
Michael T. Barbour
EA Engineering, Science, & Technology
Sparks, Maryland
xi
-------
Comparison of Laboratory and Field Good 121
Laboratory Practices (GLP) and Quality
Control Viewpoints in Terrestrial
Ecological Monitoring
A. F. Maciorowski
S. R. Brown
B. W. Cornaby
R. A. Mayer
Battelle Laboratories
Columbus, Ohio
Management and Results of a Quality Assurance 146
Program for a Large Canada-United States
Atmospheric Field Study
Robert J. Vet
Atmospheric Environment Services
Downsview, Ontario
POSTER SESSION 167
Ecological Survey of Land and Water in Britain 168
John C. Peters
United Kingdom Department of the Environment
Rural Affairs Directorate
Bristol, England
WORKGROUP SESSIONS 197
Development of Guidelines for Use of Data 198
Quality Objectives in Aquatic Monitoring
Programs
Leader:
Mr. Dennis M. McMullen
Technology Applications, Inc.
Cincinnati, Ohio
Development of Guidelines for Use of 204
Statistical Tools in Aquatic Monitoring
Programs
Leader:
: Dr. Peter MacDonald
McMaster University
Hamilton, Ontario
XII
-------
Development of Guidelines for Use of Data 209
Quality Objectives in Terrestrial Monitoring
Programs
Leader:
Dr. Elizabeth Leovey
USEPA
Washington, D.C.
Development of Guidelines for Use of 217
Statistical Tools in Terrestrial Monitoring
Programs
Leader:
Dr. James Gore
Austin Peay State University
Clarksville, Tennessee
Development of Guidelines for Use of Data 221
Quality Objectives and Statistical Tools
in Atmospheric Monitoring Programs
Leader:
Dr. Neville Reid
Ontario Ministry of the Environment
Toronto, Ontario
APPENDIX A: LIST OF PARTICIPANTS
XIII
-------
FOURTH ANNUAL ECOLOGICAL QUALITY ASSURANCE WORKSHOP
PLENARY SESSION A
Wednesday, February 26, 1991
-------
OVERVIEW OF TOTAL QUALITY MANAGEMENT
IN ECOLOGICAL MONITORING
Nancy W. Wentworth, P.E., Director
Quality Assurance Management Staff
U.S. Environmental Protection Agency
ABSTRACT
The mission of the U. S. Environmental Protection Agency
(USEPA) is to make decisions that protect or enhance the quality of the
Nation's environment. Most of these decisions are based on
environmental data. The USEPA quality program uses the principles of
Total Quality Management (TQM) to improve the efficiency and
effectiveness of the Agency's environmental data operations.
The essential principles of a Total Quality Management program
include:
Customer focus
Focus on PROCESS as well as results
Prevention versus inspection
Mobilization of workforce expertise
Senior management commitment and
involvement
Feedback
This presentation will outline the principles of TQM in detail and
will show how they can be applied to ecological monitoring.
INTRODUCTION
Over the years, the USEPA has been challenged to quantify the effect of pollution
on the environment. These efforts have largely been directed at determining the impact
of a single pollution source (or group of sources) on a limited geographical area, or at
estimating the net change in discharge resulting from the imposition of regulatory controls
across the country. Due to these limitations in scope, the studies have not generated
data that could be used to assess the overall condition of the environment, or measure
the changes in environmental quality over long periods of time.
-------
In his speech at the National Press Club in September 1990, USEPA Administrator
William Reilly noted that the historical guiding principle for developing environmental policy
has been the "ready - fire - aim" approach. That is, with each new environmental concern
came a new regulatory program, usually requiring expensive control measures, that were
not coordinated with the control measures developed in response to some previous crisis.
To quote Administrator Reilly:
"Rarely did we evaluate the relative importance of individual chemicals or individual
environmental media. We didn't assess the combined effects in ecosystems and
human health from the total loadings of pollutants deposited in different media,
through separate routes of exposure, and at various locations. We have never
been directed by law to seek out the best opportunities to reduce environmental
risks, in toto. not to employ the most cost efficient, cost effective ways of
proceeding." (Aiming Before We Shoot, The Quiet Revolution in Environmental
Policy)
The Administrator has recognized that there is a continuum in the environment that
cannot be managed piecemeal. Also, without understanding how the environment has
been affected by pollutants, the Agency cannot make the most efficient and cost effective
decisions. The principles of Total Quality Management (TQM) can assist in improving
these decision making processes.
THE EVOLUTION OF TOTAL QUALITY MANAGEMENT
After World War II, the Japanese saw a clear need to improve the quality of their
products to gain acceptance in world markets. With help from W. Edwards Deming and
others, they recognized that process analysis and understanding were keys to improving
product quality. "Made in Japan" in 1950
was a joke; "Made in Japan" in the 1990's
has a totally different meaning. The
outstanding success enjoyed by the
Japanese, beginning in the 1950's, was
the first major milestone in the modern
quality movement.
Customer/
Supp I ier
Commun i cat i on/
Understanding
Organization and Human
Resources DeveIopment
Once quality practitioners became
adept at integrating process analysis into
their operations, the next step in the
evolution of quality programs was the
focus on customers -- more specifically,
the realization that customers had explicit
needs, that could be defined, accepted in
advance, and then met. This focus on
-------
4
customer-supplier understanding moved away from the historical practice of designing
a product and then persuading the market that the product met its need. This new
philosophy required the customer and supplier to achieve mutual understanding by
collaborating in defining exact specifications for the product (or service), and developing
measures designed to assure that the product/service would meet the defined
specifications.
More recently, organizations which have achieved success through process
analysis and customer-supplier communication have come to realize that there is a third
vital dimension to TQM implementation. Cutting-edge practitioners now place great
emphasis on human resources and organizational development. Recognizing that all
levels of the workforce have much to contribute to the process, they have created tools
and techniques to channel that expertise and enthusiasm. The workforce represents a
tremendous wealth of knowledge and opportunity to improve the way business is done.
Who knows better how a process operates than someone who has ten years experience
"on the floor?" Involving the workforce in decision making can remove barriers to
implementation of new procedures, because the workforce may well have suggested
them!
Associated with workforce empowerment is a clearer focus on the organization
itself. By analyzing the organization as a culture undergoing change, TQM practitioners
can dramatically increase the likelihood that the TQM philosophy will take hold and bring
solid and lasting improvements.
THE PRINCIPLES OF TOTAL QUALITY MANAGEMENT
Each of the Quality masters (Deming, Juran, Crosby, Ishikawa, etc.) has a slightly
different approach to defining and achieving quality. There are, however, a number of
recurrent principles in the literature. These essential principles of a Total Quality
Management program include:
Customer focus
Focus on PROCESS as well as results
Prevention versus inspection
Mobilization of workforce expertise
Senior management commitment and involvement
Fact-based decision making
Feedback
-------
Customer Focus
The customer for a product, be it a widget or environmental data, must be defined,
and queried about the proposed use of the product. Unless there is clear communication
and understanding of needs and expectations, there is significant risk that the completed
product will not meet the customer's need, and may require rework, at additional expense
in dollars, time, and (in the case of environmental data) possible political concern about
exposure to potentially dangerous conditions.
Focus on Process as Well as Results
Process understanding is critical to the efficient and effective use of resources. In
environmental data collection, a focus only on results may lead to inefficiency in design,
superfluous data, data of an inappropriate quality, etc. The focus on process leads to an
application that provides the right product, at the right time, for the right price. An
example of a process that ensures that the right data are collected will be presented later
in this document.
Prevention Versus Inspection
Quality cannot be "inspected in." Quality must be considered from the very
beginning of any process design. If the customer and supplier have agreed on the
product and the performance measures for the product, then appropriate processes can
be created to ensure that those performance measures are achieved. This allows the
quality to be designed, not inspected in. Inspection discovers defects and deficiencies
after the resources have been expended, and then requires rework, at additional expense
in both time and money.
Mobilization of Workforce Expertise
The workforce has an immense knowledge of their areas of responsibility. This
knowledge is available "for the asking" by managers who realize that the answers to many
of the questions regarding quality and production rest with the workers. The workers
appreciate recognition of their knowledge and value to the organization, and can provide
very important insights into any process evaluation.
-------
Senior Management Commitment and Involvement
Change cannot occur within a system unless the highest management supports
the change. The implementation of TQM is, in many organizations, a significant change
that requires funding (for training, etc.), time (to define processes and establish feedback
mechanisms), and a continual push to assure that there is not a return to the "old ways"
of doing business. Only senior managers can assure that the money, time, and attention
are devoted to understanding and improving their organizations.
Fact-Based Decision Making
Having the right information available for decision making is not always easy. It
requires understanding the process, understanding the cause of problems, having the
skills to obtain the information needed to define and correct the problems (e.g., team-
building, communications, etc.), and having the opportunity to use facts, not innuendo,
in decision making. This focus on facts, not individuals, moves TQM further from the
historical "blame-based" operation to one in which everyone is seeking solutions, not
assigning blame.
Feedback
Feedback provides the foundation for TQM - it is the single TQM principle that
allows the others to work. Feedback is the means by which organizations learn whether
they have accomplished their goals. It provides information on the success of planning
and implementation, whether the client's needs were met in the agreed upon manner, etc.
TQM IN ECOLOGICAL MONITORING
Perhaps the most difficult step in any environmental monitoring program is defining
the "customer." Environmental monitoring is not a "production process" in the sense of
manufacturing; it is a process whose product is information for use in decision making.
Therefore, the "customer" may more correctly be called a data user or decision maker.
The decision maker may be an official of the USEPA or another Federal agency, a state
program manager, the Congress, etc. For purposes of this presentation, it is assumed
that USEPA is the decision maker for the ecological monitoring.
For ecological monitoring, the most urgent quality issue is the establishment of
effective communication between the decision maker and the "suppliers" (or data
collectors) to define the purpose of the data collection and the measures of its success.
In the absence of this dialogue, data collectors are likely to aim for the "best" data that
a laboratory analytical procedure can provide within budget, even when non-laboratory
-------
sources of error are more important. They are also highly likely to collect data which will
not be responsive to the user's true needs, and which are not ultimately suitable for
decision making.
Total Quality Management principles point to the solution to this dilemma. A TQM-
based approach would lead to a consideration of the following questions during the
planning stage of an ecological monitoring program:
Why do we want to collect data?
What data do we need?
To what uses do we expect to put the data/what decisions will be made?
How much uncertainty are we willing to accept in decisions based upon
these data?
While these questions seem straightforward, they are often difficult to answer, especially
for a broad monitoring effort which is not directly linked to a specific regulatory program.
It is certain that they cannot be addressed intuitively or haphazardly. What is required is
a structured TQM planning process which assures effective communication and thorough
consideration of the issues before data collection commences.
Initial input must come from the customer/decision maker, who must describe the
problem, the specific questions to be answered with data, and anticipated boundaries on
the data collection program. It is reasonable to expect that the decision maker will be
able to articulate some of these points only in a general sense. In most cases, he/she
requires help in specifying the problem, the desired decision, and the data needs. The
data collector should work with the decision maker by presenting descriptions of the
problem and possible approaches.
Effective communication at the planning stage will assure that the customer states
and knows what he/she needs, and that the supplier understands specifically what the
customer needs. A false start at this stage will waste valuable time and money. Without
this communication, the supplier will likely go off in a direction of limited ultimate value to
the customer.
As planning continues, the principal focus now shifts to the supplier. The supplier
works with the decision maker's input to establish qualitatively how comfortable the
decision maker is with arriving at wrong decisions (these could be called discomfort
levels). Next, the supplier develops a formula for decision-making that defines how the
different elements of the decision are to be combined. Finally, the supplier converts the
decision making formula and the discomfort levels into performance measures that
-------
8
DATA QUALITY OBJECTIVES PROCESS
State Problem
1
| Identify decisions that address problem
| 5eIect elements/factors affecting decision
Develop logic statement
1 Establish constraints on uncertainty
Optimize design for data collection
Figure 2. The DQO process includes a
comprehensive set of steps in a logical
sequence.
establish the constraints on the
design for producing the
required product.
The preceding discussion
is a brief overview of the Data
Quality Objectives (DQO)
process, a TQM tool devised by
the Quality Assurance
Management Staff for guiding
the planning stage of
environmental data operations.
The DQO process is a tool for
handling complex issues related
to ecological monitoring in a
structured and effective fashion.
It embodies the basic principles
of TQM as follows:
CUSTOMER FOCUS - The DQO process brings the customer's needs to the
forefront, assures constructive dialogue between customer and supplier, and
establishes specific performance measures for gauging whether the customer's
needs have been met.
FOCUS ON PROCESS AS WELL AS RESULTS - Too often, the decision maker's
message to data collectors amounts to: "Just get me good data!" This is an
invitation to confusion, inefficiency, and rework. The DQO tool compels a focus
on the planning process, thereby enhancing program effectiveness and credibility.
PREVENTION VERSUS INSPECTION - The aim of the DQO process is to do the
right thing, the right way, the first time. Data quality assurance programs which
emphasize review over planning simply detect mistakes instead of preventing them.
MOBILIZATION OF WORKFORCE EXPERTISE - The traditional approach to data
collection does not take sufficient advantage of the expertise of technical staff.
Lacking an adequate understanding of management's perspectives and needs,
they are compelled to operate by standard procedure or best professional
judgment, not on the basis of specific programmatic needs. The DQO process,
by contrast, is designed to create maximum involvement and understanding on the
part of technical staff.
-------
SENIOR MANAGEMENT COMMITMENT AND INVOLVEMENT - Often, senior
managers do not play a meaningful role in planning data collection programs,
because their management and policy perspectives are not readily translated into
technical terms. The DQO process does exactly that, leading to refinement and
quantification of the manager's qualitative input.
FACT-BASED DECISION MAKING - The Data Quality Objectives process is a
highly effective decision management tool. It helps managers to analyze decisions
and bring to bear exactly those data needed to address them. Thus it clarifies and
enhances the role of environmental data in Agency decision making.
FEEDBACK - One of the strengths of the DQO process is that it facilitates
feedback among all those involved in planning -- managers, technical staff,
statisticians, etc. The process is designed to be iterative, to incorporate feedback
as planning proceeds, thereby producing a much stronger and more thoughtful
plan.
CONCLUSION
This paper has provided an overview of the principles of Total Quality Management,
and has described how these principles apply to one TQM tool highly relevant to
ecological monitoring ~ the Data Quality Objectives process. The Quality Assurance
Management Staff has also developed other TQM tools which can benefit ecological
monitoring program managers. For instance, process flow modelling has proved highly
effective in pinpointing opportunities for process improvement.
Total Quality Management is a proven winner which can bring continuous
improvement to ecological monitoring programs. We on the Quality Assurance
Management Staff will be happy to provide additional information and support to anyone
interested in further exploration of these tools.
-------
10
NATIONAL PESTICIDE SURVEY DATA QUALITY OBJECTIVES:
EVALUATION AND RESULTS
Paul Black
Senior Research Statistician
ICF International, Inc.
Fairfax, VA 22031
Lora Johnson
Staff Environmental Scientist
TAI c/o USEPA
Cincinnati, OH 45219
Harold Lester
Vice-President
ICF International, Inc.
Fairfax, VA 22031
ABSTRACT
EPA's National Survey of Pesticides in Drinking Water Wells (NFS)
is the Agency's most extensive monitoring survey for pesticides, pesticide
degradates, and nitrate in the United States. Results of the Survey, recently
released in the Phase I Report, complete several years of planning and
implementation. The Survey specified qualitative and numerical data quality
objectives (DQOs) in its design phase. This paper reviews these objectives,
explains their development and testing, evaluates the Survey's success in
attaining its DQOs, and provides recommendations for future data collection
efforts. Specific topics discussed are: (1) explaining the development of
numeric precision objectives for survey estimates; (2) identifying data
requirements and data elements; (3) analyzing alternative survey designs; (4)
comparing objectives and statistical performance of the NPS; and (5)
evaluating the role of costs, detection limits, and number of samples in
meeting DQOs.
-------
11
INTRODUCTION
"Did the study answer the question?" "How confident are we in the data?" "Are
the data useful for directing policy decisions?" -- all typical questions asked during the
final phases of environmental studies. If the answers are unsatisfactory, both scientists
and policy makers may find themselves in the unfortunate position of wishing they had
designed the study somewhat differently. To prevent this situation from occurring, the
U.S. Environmental Protection Agency in 1984 formalized a planning process by which
Agency decision makers and scientists interact to reach a consensus on design and
implementation issues taking into consideration the limitations of both measurement
technology and finite resources (monetary, personnel, physical plant, etc.). As described
by Nees, et al. (1988), this process known as the Data Quality Objective (DQO) process,
was applied to the National Pesticide Survey (NPS), a complex survey sponsored by the
EPA's Office of Water and Office of Pesticides and Toxic Substances to provide
statistically valid national estimates for the occurrence of 126 pesticides and pesticide
degradates, and nitrate/nitrite in private and community drinking water wells. The
purpose of this paper is to review the original DQOs and evaluate the Survey's
performance in relation to them.
SURVEY DESIGN AND RESULTS
A detailed description of the Survey can be found in the NPS Phase I Report (EPA,
1990). Only those basic aspects of the design necessary for discussing the Survey DQOs
will be presented here. The Survey, which spanned the years 1984-1991, used a three
stage stratified design from which more than 1300 wells were selected for sampling using
probability based sample selection techniques. First stage stratification variables were
ground-water vulnerability measured using a modified DRASTIC scoring method
(Alexander, et al, 1986), and pesticide use determined from Doane marketing data and
other sources. Exhibit 1 shows the 12 first stage strata which were used to classify all
counties in the U.S.
The NPS was conducted as two separate surveys, one for community water
system (CWS) wells, and one for rural domestic wells. For the rural domestic well survey
only, ninety counties were selected at the first stage and wells were further stratified at the
second stage using an index that combined information from subcounty DRASTIC scores
and information on cropping intensities. Areas within each county were defined by this
index as "cropped and vulnerable".
-------
12
The goal of a stratified design is to increase sampling efficiency over simple
random sampling by grouping sampling units that are expected to have similar
characteristics. Sampling rates can then be adjusted to account for differences in
variability between the groups, i.e., a group in which more variability is anticipated would
require a larger sample size in order to achieve the same level of precision in the data.
The NFS chose a stratified design believing that differences in hydrogeologic conditions
and pesticide use would affect the number of wells with detectable levels of pesticides
and to provide control of sample sizes within strata.
Results of the Survey indicate that about 10 percent of the approximately 95,000
CWS wells in the country can be expected to contain at least one of the 126 pesticides
and pesticide degradates included in the NFS. For the approximately 10.5 million rural
domestic wells, at least one detectable pesticide can be expected in about 4 percent of
the wells. Initial results of the NFS are available in the NFS Phase I Report.
Exhibit 1: FIRST STAGE OR COUNTY-LEVEL STRATIFICATION CELLS
Vulnerability
High
Moderate
Low
Pesticide Use
High
1
2
3
Moderate
4
5
6
Low
7
8
9
Uncommon
10
11
12
DISCUSSION OF DQOs
One of the goals of the DQO process is to develop quantitative statements about
the level of confidence needed to answer the question(s) prompting the study. The two
major goals of the NFS were:
• To determine the frequency and concentration of the presence of
pesticides and nitrate in drinking water wells nationally; and
-------
13
• To examine the relationships of the presence of pesticides and
nitrate in drinking water wells to patterns of pesticide use and
ground-water vulnerability.
During the Survey development, EPA analyzed numerous alternative designs,
sample sizes, and data collection activities to meet these goals. The Agency evaluated
acceptable alternative precision estimates for the study domains and estimated costs of
the Survey. Survey costs were largely a function of the number of wells to be sampled.
Exhibit 2 shows the final DQOs established for the Survey.
For the NPS, DQOs were stated as precision requirements to determine the
proportion, or equivalently, the frequency, of the presence of pesticides in drinking water
wells nationally. DQOs were not specified for the presence of nitrate, in terms of pesticide
or nitrate concentrations, nor in terms of relationships between pesticide or nitrate
presence and patterns of pesticide use and ground-water vulnerability. The Survey did
not explicitly include final reporting limits for the chemical analysis of the water samples
in determining these objectives, but were implicitly included through the assumed
detection rates. These topics and their implications will be discussed at greater length
in a later paper.
The NPS precision requirements were stated for several different well categories,
called domains, as shown in Exhibit 2. For the CWS well survey two domains were
defined: all wells nationally, and wells in counties with high ground-water vulnerability. For
the rural domestic well survey, five domains were defined: all wells nationally, wells from
high pesticide use counties, wells located in counties with high ground-water vulnerability,
wells located in "cropped and vulnerable" areas within counties, and wells located in
counties with both high pesticide use and high ground-water vulnerability.
-------
14
Exhibit 2: PRECISION REQUIREMENTS FOR THE NPS
DOMAIN
CWS:
National
CWS:
High Ground-Water
Vulnerability
Domestic:
National
Domestic:
High Pesticide Use
Domestic:
High Ground-Water
Vulnerability
Domestic:
"Cropped & Vulnerable"
Domestic:
High Pesticide Use &
High Vulnerability
Detection
Rate
P
0.005
0.005
0.01
0.01
0.01
0.01
0.01
Relative
Standard
Error
RSE(p)
0.658
1.040
1.00
0.85
0.85
0.525
1.25
Detection
Probability
DP(p)
0.90
0.60
0.63
0.75
0.75
0.97
0.47
-------
15
In essence the precision requirements for each domain correspond to the a priori
expected proportion of contaminated wells (detection rate), together with the associated
variance of that proportion. Variances were equivalently stated in terms of relative
standard errors, detection probabilities, and confidence intervals. The detection rates and
their variances were then used to calculate the necessary minimum sample size that
would achieve that level of precision in the NPS estimates. In practice the precision
requirements were stated conservatively to provide further assurance of an adequate
sample size.
The Survey used disproportionate sampling (i.e., oversampling) in the high
vulnerability strata, cells 1, 4, 7, and 10 in Exhibit 1, and the "cropped and vulnerable"
strata, to increase the expected number of drinking water wells containing pesticides and
to allow separate estimates for the specified domains. Proportionate stratification, with
stratum sample sizes made proportionate to stratum population sizes, was expected to
yield an insufficient number of detections. EPA was particularly interested in estimating
pesticide occurrence in areas of agricultural pesticide use and areas vulnerable to ground-
water contamination.
As an example of how to interpret the DQOs or precision requirements, consider
the CWS national domain in Exhibit 2. Survey planners assumed that a conservative
estimate of the proportion of wells nationally that would have at least one pesticide
detection was 0.005, and at least a 90 percent chance of detecting at least one pesticide
or pesticide degradate in at least one sampled well was required. In the underlying
statistical model the detection rate, 0.005, corresponds to a Binomial proportion, and the
detection probability, 90 percent, is a function of the detection rate and its variance. The
relative standard error (0.658 in this case) is also a function of the detection rate and its
variance and provides an alternative, though equivalent, characterization of the underlying
model. Confidence intervals were also calculated as part of the original DQOs but are not
presented here1. The mathematical relationships between these quantities are presented
at the end of this paper.
Three major points of interest concerning the NPS precision requirements are
shown in Exhibit 2:
• The constant assumed rate of detection across CWS well
domains (0.005) and across rural domestic well domains
(0.01);
• The relatively high detection probability (0.97) for the "cropped
and vulnerable" domain in the rural domestic well survey; and
Confidence intervals presented as part of the original DQOs were based on a normal distribution approximation to the
Binomial distribution. This approximation is not valid for the extremely small proportions (detection rates) assumed for the NPS.
-------
16
• The higher assumed rate of detection for rural domestic well
domains (0.01) than for CWS well domains (0.005).
The first point on its own seems contrary to the need for oversampling, i.e., detection
rates in each survey were not specified to be greater in some strata (e.g., counties with
high vulnerability) than others. The overriding consideration, however, was that each
detection rate was stated conservatively to further ensure adequate sample sizes. The
second point translates into a larger sample size requirement for the "cropped and
vulnerable" domain which corresponds to oversampling of that domain. This reflects
EPA's desire to ensure the most precise estimates for the "cropped and vulnerable"
domain.
The third point concerns the quality of the information on which the precision
requirements were based. The precision requirements state that the proportion of
detections expected in the rural domestic well survey was double the proportion expected
in the CWS well survey. The survey results demonstrate the opposite effect.
Furthermore, Survey results do not indicate a greater proportion of detections in the
subdomains than in the national domains. This brings into question the benefits of the
stratification. Effective stratification resulting from high quality information reduces the
variance of sample estimates, but ineffective stratification results in loss of performance
and possible confounding of results.
The sample allocation procedure used to compute the sample sizes presented in
the first column of Exhibit 3 involved a complex optimization procedure that
simultaneously calculated sample sizes for the domains. This procedure was performed
separately for the two surveys. The optimization procedure accounted for the precision
requirements and the cost estimates for each stage of the sampling process. Although
the optimization procedure accounted for all domains simultaneously, the sample
allocation results were driven mainly by the cost constraints and the domains for which
the lowest variance, or highest detection probability, was specified (i.e., the "cropped and
vulnerable" domain in the rural domestic well survey and the national domain in the CWS
well survey).
Referring to Exhibit 3, the comparison between achieved and specified sample
sizes can be made by examining their ratio, termed completeness. With the exception
of the "cropped and vulnerable" domain of the domestic well survey, all values for
completeness were greater than 95 percent which, considering its complexity, suggests
that the Survey was implemented in accord with the design specifications. The low
achieved sample size for the "cropped and vulnerable" domain in the rural domestic well
survey was largely due to a reduction in the oversampling rate that occurred between the
initial sample design and the start of the implementation of the Survey. The initial
oversampling rate was reduced to avoid problems associated with wells having high
survey weights which could have dominated analysis of the survey data.
-------
17
EVALUATION OF DQOs
Several approaches can be used to evaluate the DQO requirements. The most
transparent measure of the success of the Survey in meeting the requirements of the
DQOs is to compare achieved domain variances with the initial variance specifications.
As can be seen through the detection probabilities or relative standard errors in Exhibit
4, these indicators of survey performance show that the Survey exceeded the variance
specifications in all domains. However, the driving force behind the performance of the
Survey, measured in these terms, is the higher than expected detection rates.
Exhibit 3: COMPARISON OF DESIGN AND ACHIEVED SAMPLE SIZES
DOMAIN
CWS:
National
CWS:
High Ground-Water
Vulnerability
Domestic:
National
Domestic:
High Pesticide
Use
Domestic:
Highly Ground-Water
Vulnerability
Domestic:
"Cropped &
Vulnerable"
Domestic:
High Pesticide Use &
High Vulnerability
SAMPLE
ALLOCATION
564
203
734
200
254
463
79
ACHIEVED
SAMPLE SIZE
540
197
752
217
264
343
83
COMPLETENESS
96
97
102
108
104
74
105
-------
18
Exhibit 4: COMPARING DESIGN SPECIFICATIONS AND SURVEY RESULTS
DOMAIN
CWS:
National
CWS:
High Ground-Water
Vulnerability
Domestic:
National
Domestic:
High Pesticide
Use
Domestic:
High Ground-Water
Vulnerability
Domestic:
"Cropped &
Vulnerable"
Domestic:
High Pesticide Use &
High Vulnerability
DESIGN
SPECIFICATIONS
P
0.005
0.005
0.01
0.01
0.01
0.01
0.01
RSE(p)
0.658
1.040
1.00
0.85
0.85
0.525
1.25
DP(p)
0.90
0.60
0.63
0.75
0.75
0.97
0.47
ACHIEVED RESULTS
*
P
0.104
0.093
0.042
0.036
0.028
0.055
0.014
RSE(p)
0.14
0.24
0.23
0.47
0.50
0.32
1.10
DP(p>
1.00
1.00
1.00
0.99
0.98
1.00
0.56
Detection probabilities are reported to two decimal places. Values reported as
1.00 correspond to values greater than 0.995, but less than 1.00
-------
19
The design effect offers a simple post hoc summarization of the complex modeling
process. The design effect can be used to calculate an effective sample size that
corresponds to the sample size that would be required to produce equivalent survey
precision under simple random sampling (assuming the same detection rates). For a
simple random sample the weights are the same, hence the design effect is equal to one.
Desin Effect, d:
d =
n
y" Wj)2
2=1
Effective sample size, n7:
n/ - ^.
where n is the sample size achieved by the survey, and the W, are the survey weights
associated with each sampled well. Exhibit 5 provides the design effect, achieved sample
size, and the effective sample size for each domain of the Survey.
Another way to evaluate survey performance is to compare the achieved effective
sample size to the sample size that would have been allocated to each domain separately
under simple random sampling given the precision requirements. A comparison between
columns 2. and 3 of Exhibit 6 indicates that for most domains the sample allocation was
adequate using this method. The exception is the "cropped and vulnerable" domain in
the rural domestic well survey where approximately twice the number of observations
would be required under simple random sampling to satisfy the precision requirements.
Assuming a design effect of 2.06 (see Exhibit 5) for the "cropped and vulnerable" domain,
the number of observations required to satisfy the precision requirements for detection
rates and variance is 740. This calculation does not account for cost constraints.
The "cropped and vulnerable" domain is a subset of the national domain. A
requirement of 740 observations for the "cropped and vulnerable" domain results in a
requirement for substantially more observations for the rural domestic well national
domain. In terms of population characteristics the "cropped and vulnerable" areas
comprise approximately 35 percent of rural domestic wells nationally (see the NPS Phase
I Report). Allowing for the revised rate of oversampling of "cropped and vulnerable" areas
-------
20
(3:1), this implies a sample size nationally of about 1,200 wells. By this measure the
sample allocation procedure underestimated the number of samples required to achieve
the precision requirements, but as the observed detection rates were substantially greater
than those specified in the precision requirements, the Survey appeared to perform well.
This procedure mixes aspects of the sample allocation procedure and survey
results to evaluate survey performance. Using this approach it is clear that all NFS
domains fell within precision requirements because the proportion of wells that had a
detectable pesticide was much higher than assumed during the design stage, but the
specified detection rates were stated conservatively to encourage an adequate sample
size.
Exhibit 5: DESIGN EFFECT AND EFFECTIVE SAMPLE SIZE
DOMAIN
CWS:
National
CWS:
High Ground-Water
Vulnerability
Domestic:
National
Domestic:
High Pesticide
Use
Domestic:
High Ground-Water
Vulnerability
Domestic:
"Cropped &
Vulnerable"
Domestic:
High Pesticide Use &
High Vulnerability
Design ~
Effect
(<*)
1.27
1.16
1.84
1.77
1.91
2.06
1.44
Sample
Size
(n)
540
197
752
217
264
343
83
Effective
Sample
Size
(n/)
425
170
409
123
138
167
58
-------
21
Exhibit 6: COMPARISON OF ALTERNATIVE SAMPLING METHODS
DOMAIN
CWS:
National
CWS:
High Ground-Water
Vulnerability
Domestic:
National
Domestic:
High Pesticide
Use
Domestic:
High Ground-Water
Vulnerability
Domestic:
"Cropped &
Vulnerable"
Domestic:
High Pesticide Use &
High Vulnerability
Achieved
Sample
Size
540
197
752
217
264
343
83
Effective
Sample
Size
425
170
409
123
138
167
58
Sample
Allocation
Under SRS
459
183
99
137
137
359
63
SRS Sample
Allocation
Adjusted
for Design
Effect
583
212
182
242
262
740
91
IN RETROSPECT
The Agency is preparing an extensive analysis of Survey results (NPS Phase II).
Final conclusions about the Survey design await release of the study. Preliminary
conclusions, however, concerning the final design and the DQO process are appropriate.
First, the data quality objectives were satisfied, but largely because the Survey used
conservative assumptions (i.e., detection rates). Second, DQOs did not address other
analytical and modeling questions that Survey data will be used to answer. Third, the
benefits of stratification for the Survey are not obvious. Detection rates among the
domains are not very different.
-------
22
The Survey goals, to provide baseline and modeling data over a broad range of
compounds, required satisfying multiple objectives and precluded specifying quantitative
data quality objectives for every aspect of the Survey. The NFS reporting limits, while low
enough to protect the public health and simultaneously analyze for 126 compounds, were
not low enough to generate sufficient data to develop concentration occurrence curves,
which would have greatly increased the utility of the study for environmental assessment
purposes.
In addition, using less extensive stratification and proportionate sampling,
considering the actual detection rates within each domain, would have provided the ability
to allocate Survey funds for more samples which could have reduced the variance of the
estimates and provided a richer database for analysis. Environmental surveys should
critically evaluate the desirability of complex stratified designs using a variety of planning
assumptions. Sample costs, stratification data quality, and reporting limits are key areas
for intensive scrutiny in the planning and pilot test development stage.
References
Alexander, W.J, S.K Liddle, R.E. Mason, and W. B. Yeager, 1986. Ground Water
Vulnerability Assessment in Support of the First Stage of the National Pesticide Survey.
Research Triangle Institute, unnumbered report. 168 pp.
EPA, 1990. National Survey of Pesticides in Drinking Water Wells - Phase I Report. PB91-
125765
Nees, Monica and Cynthia Salmons, 1987. National Survey of Pesticides in Drinking
Water Wells. A Review of the Planning Process and the Data Quality Objectives.
RTI/7801/08/01F
-------
23
STATISTICAL ASSUMPTIONS AND FORMULAS
1 . Detections follow a Binomial Distribution denoted Bin(n,p)
where:
n represents the sample size; and
p represents the detection rate or proportion of detections
_ nationally. _
2. The variance of p is defined as:
n
3. Relative Standard Error (RSE)
4. Detection Probability fDP)
DP(p) =l-[l-p]e
where 6 is a surrogate for the sample size:
e_p(i-p)
VAR (p)
5. Binomial Confidence Intervals
Lower bound is the value pL that satisfies:
a/2
Upper bound is the value pu that satisfies:
(6)p^(l-p[7)(e-i) i a/2
\ 1I
If 6 and p are large enough, the Normal distribution approximation to
the Binomial distribution may be used. This is not the case for the
NPS generally (because p is most often too small).
-------
24
PLANNING APPROACHES IN CANADIAN TERRESTRIAL FIELD
MONITORING STUDIES
Anne J. Neary
Laboratory Services Branch
Ontario Ministry of the Environment
Dorset, Ontario.
ABSTRACT
Over the last ten years, many terrestrial field monitoring studies were done
in Canada to determine the effects of acidic deposition on the environment.
Quality assurance related activities, in the early studies, were incorporated
into the work and were done on an individual project basis. Only recently
has QA evolved into a formalized, separately Identified effort. As a result, it
has only been in the last five years that Canadian terrestrial field monitoring
studies have adopted a formalized QA approach to planning. This, however,
does not preclude the success of the earlier studies. Indeed, early Canadian
monitoring efforts were usually well planned and incorporated the necessary
QA/QC criteria to provide a valuable data base. Recent terrestrial studies,
however, done by the Ontario Ministry of the Environment, nave benefited by
greater incorporation of QA into the planning stages of the study. A formal
QA plan and well documented QC protocols can strengthen a study and
ensure the value of a data base for future users.
INTRODUCTION
In the late 1970's, Canada recognized the need to examine the environmental
effects of the long range transport of airborne pollutants (LRTAP). Task oriented federal
and provincial programmes were established to address this issue. During the next
decade, many terrestrial monitoring studies began within these programmes. Most of the
studies were done in Ontario, Quebec and the Atlantic provinces, where acidic deposition
rates are highest.
Much of the early work was similar in design to earlier phytotoxicological studies.
Previously defined and tested field and laboratory procedures were used. Other early
monitoring efforts formed the terrestrial part of calibrated watershed studies which also
included aquatic and depositional components. As a result, quality assurance was usually
addressed on an individual project basis. The QA/QC, which often evolved from earlier
studies, was incorporated into the work. The QA/QC needs and practices were reviewed
and refined as the project progressed.
-------
25
Formalized QA planning in environmental studies is a relatively recent
phenomenon. This meant that, initially, QA was not a separately organized effort.
Instead, QA was addressed as part of "good science", since many of these studies were
destined for eventual publication, with associated peer review. Much of the QA/QC is
available only in draft form or as internal reports generated during the studies. Most of
the analytical support for these programmes was through government laboratories which
often also supported several other programmes. Consequently, the laboratory QA/QC
is often found in a more formalized form.
Increased emphasis on formalized QA/QC for monitoring studies began in the late
1980's for several reasons.
1. Over the last decade there was an increased awareness that
a formal QA plan is helpful in ensuring the integrity of any
data collected. This was especially important given the long
term nature of most of the studies.
2. As studies progressed, it became clear that data quality
objectives should be more accurately defined at the beginning
of the study.
3. The political and economic implications of the acid rain-related
effects on the environment had gained an international profile.
This increased the need for data which were comparable
among different agencies, provinces and countries.
4. Small projects carried out internally by two or three people,
increased in size and number. As a result, parts of the field
measurements, sample collection and data review were
contracted to universities and the private sector.
By 1986, QA/QC initiatives began to be put in place before rather than during the study.
OVERVIEW OF TERRESTRIAL PROGRAMS IN CANADA
Table 1 shows the variety of terrestrial studies that were carried out. Many of these
studies began without a QA plan or manual. Quality assurance manuals were often
produced during the study. Three main types of studies can be identified:
biogeochemical monitoring (intensive studies in a small area), baseline monitoring and
forest assessment and decline studies.
-------
26
Biogeochemical studies were located at calibrated catchments located in
Kejimkujik, Nova Scotia, Montmorency, Quebec (Lac Laflamme), and in Ontario in
Muskoka/Haliburton (Plastic and Harp Lakes), the Algoma area (Turkey Lakes), and the
Kenora Experimental Lakes Area (ELA) (Figure 1). Smaller terrestrial monitoring studies
were also undertaken near Thunder Bay, Ontario (Hawkeye Lake) and Sudbury, Ontario
(High Falls). The early terrestrial work at many of these watersheds consisted of soil and
vegetation characterization, and stemflow, throughfall and soil lysimeter leachate
monitoring. Only the Turkey Lakes and ELA biogeochemical studies, had formal QA
plans and QA manuals were developed during the studies. The Kejimkujik, Plastic, Harp
and Hawkeye Lake sites did not have formal QA plans, however, for the latter three sites,
QA documentation can be found in internal reports (Lozano et al. 1987, Lozano and
Parton 1987, Lozano 1986, Senes Consultants 1986, Senes Consultants 1987a,b,c,d,e)
and in Neary (1986). Over the years, terrestrial work at these sites shifted from
monitoring to process oriented studies.
In 1980, The Ontario Ministry of the Environment began baseline monitoring. This
was designed to benchmark the terrestrial system by providing information on the
chemical status of soils, vegetation and lichens in the province. Quality assurance and
procedural manuals for soil and vegetation monitoring were developed during the study
(Neary 1986). Quality assurance for the lichen monitoring project is briefly addressed in
the final report for the project (Case Biomanagement 1990). A soil variability study was
also initiated as a separate pilot project to provide an estimate of the variability at specific
baseline sites. This provided information on seasonal variability, sampling times and
sample representativeness. Forestry Canada also conducted some baseline foliage
chemical monitoring. Quality assurance activities related to foliar chemistry variability are
given in journal publications (Morrison 1972, 1974, 1985). Soil sensitivity mapping for
Canada was conducted by Environment Canada and for Ontario by Environment Canada
and the Ontario Ministry of the Environment (Cowell 1986). These mapping efforts cannot
be classified as monitoring studies, but the latter study used baseline soils data from the
monitoring studies. The mapping criteria and methodology and the conceptual model
used for site evaluation are listed in Cowell (1986).
Studies related to forest health in Canada began in the mid to late 1980's. Forest
studies undertaken in Quebec, by the Ministere de I'Environnment and the Ministere de
I'Energie et des Ressources, as well as the Canadian Forestry Service's Fundy Birch
Assessment and Maple Decline study were designed with formal QA plans, use of field
methods manuals and attention to error definition in the field and laboratory. (RMCC
Quality Assurance Subgroup 1990). Ontario forest assessment and decline studies,
including the Acid Rain National Early Warning System (ARNEWS) which began in 1985
on a regional scale, began without formal QA plans. During the study, however, field
tested procedures were used, determinations were made of accuracy, comparability and
sources of error, measurements and site selection criteria were standardized and pilot
studies were used (McLlveen et. al. 1989, McLaughlin et al. 1988, RMCC Quality
Assurance Subgroup 1990, Beak Consultants 1990). ARNEWS tested many of the
-------
27
methodologies used during the routine Forest Insect and Disease Surveys (Magasi 1988).
The North American Maple Project (NAMP), which began in 1987, is carried out by a
number of different agencies in Quebec, Ontario and the USA. Special attention was
given to the production of a QA plan, objective setting, testing, detailed documentation
of methods and training of field crews (Miller, 1988).
Other studies shown in Table 1 but not covered in this overview are smaller
research oriented studies. The absence of a formal QA plan, however, does not preclude
the success of a study. This is seen by looking more closely at some of the terrestrial
studies done as part of the Ontario Ministry of the Environment's Acidic Precipitation in
Ontario Study (APIOS).
-------
Table 1. QA Considerations in Canadian terrestrial LRTAP studies. (Adapted from: RMCC Quality Assurance Subgroup 1990).
Study Type and Name
BIOGEOCHEMICAL
Calibrated Watersheds
E.L.A.
Boa Acificalion
V
Turkey Lakes
Kejimkujik
Plastic Lake
Harp Lake
•— f - —
.Qiki
CFS Quebec Balsam Fir
CFS Quebec Sugar Maple
CFS Maritime Nil. Chcro.
Halifax Urban Watershed
Nashwaak Watershed
OME Canopy Effects
OME Forest Soils
BASELINE MONITORING
Seotilivily
OME Soil Baseline
Foliage Chemistry
Ontario Lichens
TRACE METALS
New Brunswick (DMAE)
In Arctic Lichens
OTHER
CFS Maritimes - Air
Pollution and Plant
Reproduction
Y - Yes B - Before
N - No D » During
Formal
QA
QA Plan Manual
Y
N
Y
N
N
N
Y
Y
Y
Lab Only
N
N
N
N
Y
N
N
N
N
N
1 • Informally
D
B
B
-
1
1
.
B
N
N
D
D
.
D
O
B
-
•
Considered
Study Type and Name
FOREST ASSESSMENT
DECLINE STUDIES
ARNEWS
Prairies
Maritime*
British Columbia
North American Maple
Project fNAMPl
Environment Ontario
Hardwood Survey
Dendrochronology
Maple Decline
Hardwood Nutrition
Early Diagnosis
Insect Defoliation
Canadian Forestry
Service Maritime
Cuticle Effects
Pollution Effects
Maple Decline
Mitigation
Birch Assessment
Other
Quebec Forest Health
ARA (NS) Maple
Product Quality
Formal
QA Plan
N
N
N
Y
N
N
N
N
N
N
N
N
N
Y
Y
Y
Y
QA
Manual
D
B
B
B
B
B
D
B
N
-
n/r
N
B *
B
B
N
no
00
-------
Figure 1.
Biogeochemistry Study Sites
Lac Laflam
Kejimkujik
-------
30
ONTARIO MINISTRY OF THE ENVIRONMENT - APIOS PROGRAMME
Organization/Goals and Objective Setting
The APIOS program began in 1979 with a series of defined tasks. Investigations
were made into 1) the atmospheric deposition of acidic substances, 2) the effects of
acidic deposition on the aquatic systems in Ontario, and 3) the effects on the terrestrial
systems of Ontario and 4) the economic effects of acid rain. In 1980, planning of the
terrestrial studies began. The organizational and reporting structure of the program was
well-defined (Figure 2). A small group of scientists responsible for the project design,
implementation and data interpretation were responsible to the Terrestrial Effects Working
Group. This group included members from the laboratory supporting the programme
(Ontario Ministry of the Environment Laboratory), APIOS Programme Coordination Office,
senior management, invited members of the Atmospheric and Aquatic Working Groups,
and representatives of other Ministries and industry.
A Technical Subcommittee was formed which included the project scientists,
laboratory and APIOS Coordination office representatives, and sometimes members of
other work groups. At this level the study planning details were resolved. The specific
expectations, goals, objectives and study strategy were defined. Study design, siting
criteria, number of sampling locations, field measurements, number of samples, field
procedures, sample laboratory analysis requests, data precision and accuracy
requirements, completeness, error sources, and data handling all received approval from
this group before the project began. The Technical Subcommittee reported, through the
Terrestrial Effects Working Group to the Science Committee. The Science Committee
provided scientific peer review, and technical and budgetary direction to the proposed
projects. The Science Committee reported to the Acid Rain Committee, a group
responsible for policy statements for the programme. This structure allowed for
considerable input into the project design and required that a certain amount of QA-
related activities be built in to the studies themselves.
In 1985, a Quality Management Plan for the APIOS program was written and
endorsed by Science Committee. The plan outlined the organizational structure and gave
a broad policy statement for all APIOS tasks to carry out in their respective programmes.
A QA/QC representative was assigned to help provide direction in formalizing QA/QC
activities and identify shortcomings. Types of planning approaches used in the terrestrial
studies are discussed in the following case studies.
-------
31
Case 1 - Biogeochemistry Studies
Field Studies
Biogeochemistry studies were set up in the early 1980's in small subwatersheds
within the Plastic, Harp and Hawkeye Lake catchments. The High Falls site near Sudbury
was also monitored for a short period in the mid 1980's. Planning of the work was done
in stages.
Goals and objectives were first defined. These were presented to the Terrestrial
Effects Working Group and then to the Science Committee. Long range goals included
measuring the effects of acidic deposition on selected forest ecosystems of Ontario and
assessing the role of these ecosystems in regulating lake and stream chemistry. Short
term objectives included: 1) documenting element and nutrient distribution and cycling
in forested ecosystems receiving similar and different acid loadings; 2) defining
interactions between precipitation, vegetation and soil, and their ability to alter the water
chemistry of discharge to aquatic systems; and 3) modelling the effects of acidic loading
on selected watershed systems (Technical Subcommittee Terrestrial Effects Working
Group, 1983).
Once the proposed work was approved, the Technical Subcommittee decided
which processes to monitor. A conceptual model was defined (Figure 3). Decisions
were made on monitoring methods and what type of additional site information was
required. Sampling strategies for soil, vegetation, litterfall, biomass estimates,
precipitation, throughfall, stemflow and soil water were determined. The data quality
objectives were discussed in light of their ability to detect change.
Siting criteria were then chosen. Many criteria were listed which were divided into
requirements and desirable attributes. All sites had to meet the requirements, but not all
sites had to include all the desirable attributes. Criteria included accessibility, land tenure,
subwatershed size, drainage characteristics, soil and vegetation type and fire and
harvesting histories. Two of the sites (Plastic and Harp Lake) were also chosen, to
coincide with intensive aquatic studies on these lakes and the presence of a
meteorological site in the area. The Hawkeye Lake site, located in an area of low acidic
deposition, was chosen after a preliminary soils analysis and forest inventory. The High
Falls site provided a site in the Sudbury area. This site was later dropped due to the
confounding effects of sulphur emissions from Sudbury.
-------
Figure 2.
APIOS Program - Terrestrial Effects
32
ORGANIZATION CHART
ACID RAIN COMMITTEE
Responsible for environmental
policy statement
SCIENCE COMMITTEE
Provides direction on program planning
and budget for all APIOS tasks
TERRESTRIAL EFFECTS WORKING GROUP
Responsible for Terrestrial Effects
Program Planning
TERRESTRIAL EFFECTS TECHNICAL SUBCOMMITTEE
Responsible for technical details regarding
project planning, methodologies, priorities
data handling, QA/QC concerns
OUTSIDE CONTRACTORS
Responsible for field
sampling programs «-
PROJECT SCIENTISTS
Responsible for
study undertaking
-------
33
In order to help organize the activities and better determine the workload, sample
timing was examined and the samples were categorized as follows:
Frequency Measurement
once only watershed area, forest cover, vegetation biomass, soil
characterization and mass event incident precipitation,
throughfall, stemflow, lysimeter leachate
fixed interval precipitation, dry deposition, stream water, litterfall, litter
decomposition, groundwater, evaporation rate
continuous stream flow, SO2,
O3, NOX, windspeed
Throughout these planning stages, quality assurance issues were addressed. Much
research went into the type of sampling equipment and its construction. Comparisons
of collector efficiency were made. For event sampling, quality control protocols were
agreed to and documented by the Technical Subcommittee. Equipment cleaning, sample
container cleaning and maintenance procedures helped to increase the accuracy of
measurements (Gizyn 1983). Standard field forms meant the collection of all necessary
field information at each site visit (Gizyn 1983, Senes Consultants 1987e). Field replicates
provided precision data on the variability of stemflow, throughfall and soil water samples
between similar trees and soil types. Sampling (aliquoting) techniques for throughfall and
stemflow samples were compared to determine sample representativeness and data
comparability with other studies (Senes Consultants 1986).
Other aspects of accuracy were addressed through contamination studies.
Sample bags were leached with distilled water and acidified solutions. The solutions were
analyzed for signs of contamination. The possibility of metals in the sample adsorbing
on the walls of the bag was examined, and an aqueous perishability study for throughfall
and stemflow samples was conducted. All results were reported to the Technical
Subcommittee and the sampling protocol reviewed. Unfortunately, most of the QC
activities were presented as memos to the Subcommittee or informal reports. Lysimeter
contamination for the collection of soil waters was addressed in a slightly more formal way
and a recommended procedure for washing the lysimeters before use was outlined
(Neary 1985).
Detailed documentation of sample collector construction, sampling times, sample
collection and processing procedures is available for "once only" samples (Neary 1986,
Lozano et al. 1986, Lozano and Parton, 1986, Lozano 1986, Senes Consultants
1987a,b,c,d,e). Sample representativeness was considered and included checking soil,
foliar and litterfall chemical variability. The number of soil pits and litter plots required to
-------
34
estimate the elemental concentration with a 5 percent and 10 percent level of precision
was calculated (Tables 2 and 3). Similarly, estimated error in the forest inventory was
kept to a minimum. The number of sample plots necessary to maintain error below 15
percent was calculated (Table 4). Potential error sources for each input into a biomass
model were identified and the size of the error estimated (Senes Consultants 1987e).
These inputs included Bitterlich basal area estimates and diameter/height relationships.
Replicate sampling of soil and vegetation was done routinely with every terrestrial
project to provide a measure of precision. Replicate soil sampling protocols (side by
side duplicates, across pit sampling, sample compositing, etc.) were also discussed at
length before sample collection. Contamination from mortar and pestle sample grinding
was minimized by the use of agate mortar and pestles. Recently, the effects of sample
grinding on extractable iron and aluminum in soils was also investigated (Barnes and
Neary 1990).
Laboratory Analysis
All of the chemical analysis for precipitation, stemflow, throughfall, lysimeter
leachate, biomass, lichens and soil were performed by the main Environment Ontario
laboratory. A formal QA plan was in effect in the laboratory long before the inception of
the APIOS programme. The key QC elements for all the analyses done for the Terrestrial
Effects Program included:
a) Documented, referenced and tested, standard analytical
procedures, and proper technician training.
b) QC protocols for instrumental analysis. These included
calibration standards (matrix matched, if necessary), sensitivity
checks, baseline drift checks, longterm blanks and recovery
checks.
c) QC protocols for sample preparation (extraction, digestion,
etc.) and instrumental analysis. These included within-run
duplicates, method blanks, spiked samples, and between-run
duplicates (standard soils/vegetation run with each batch).
-------
35
Evopotronspirotion'
Evaporation
Transpiration
1
1
o
Seepage
Slreamflow
Output
Figure 3: Conceptual Model of Biogeochemistry Study
-------
36
Table 2. Number of soil pits required to estimate within 10% error at P = 0.05 the mean
of a given soil property for Orthic Humo-Ferric Podzols at Plastic 1. (Source:
Lozano et al. 1987)
Actual
_CH
Exchangeable
No. of
Pits
LFH
Ahe
Ac
Bfl
Bf2
6
2
8
8
5
H2O
3
5
2
2
0
CaCl2
4
8
4
1
0
ORGC
11
65
157
32
95
Sand
„
17
18
13
22
Sill
— —
78
96
61
88
Clay
„
37
86
45
44
N
3
44
66
28
66
Ca
15
39
290
501
81
Ng
9
77
111
51
108
K
9
32
35
18
90
Al
119
4
70
194
41
CEC
8
10
27
78
41
LFH
Ahe
Ae
Bfl
Bf2
Actual
No. of
Pits
6
2
8
8
5
PvroohosEhate3
sol
33
28
38
39
41
Fe
758
17
213
147
124
Al
489
25
25
52
67
Dithionite4
Fe
183
2
124
44
33
A]
401
12
30
78
38
A]-Ca3
101
20
88
270
150
Cu
28
16
122
46
12
Acid Extractable
Zn
15
3
65
27
11
Pb
15
201
89
28
56
Ni
106
1
161
35
25
1 NaCl Extraction
2 Water Extraction
3 Sodium-pyrophosphate Extraction
4 Citrate-bicarbonate-dithionite Extraction
5 CaCl2 Extraction
-------
37
Table 3. Number of litter plots required to estimate weights of litter fall/nutrients to
within ± 5 and 10 percent of mean. P » 0.05 and 0.1 (1985-86). (Source:
Lozano 1986)
Plastic Lake
TouJ
Prob. Prec. Leaf Twit Seed Litter £PJ£.£t.M*.3.iiS.Es Al MJJ Cu .Zn
95 5 23 44 1974 29 61 61 90 44 86 38 82 82 174 447 46 300
95 10 6 11 494 7 15 15 22 11 22 9 20 21 43 112 12 75
901037303 4 991471361313 27 69 746
Harp Lake
Total
Prob. Pree. Leaf Twig Seed Litter N^JS^aV^JNalS A] Mn Cu Zn
95 5 113 435 1466 88 101 88 75 44 59 67 80 1597 1534 130 49 138
95 10 28 109 366 22 25 22 19 11 15 17 20 399 384 32 12 35
90 10 17 67 225 K 15 14 11 7 9 10 12 245 236 20 8 21
-------
38
Table 4. Actual number of sample plots taken during the inventory at Plastic 1 and
calculated number of sample plots required to meet 15% allowable error of
estimates in a given stand at Plastic 1. (Source: Lozano and Parton 1986)
Rock Outcrops
Brush and Alder Swales
Total
Number of Sample Plots
Forest
Pws
Msj
Ce? i
Pw3
He4
Stand Types
Hej Ofj M$i Oj
Pw2 Cfrj Bwj Orj Hej C
Sb: Swj O:
M&2 Or2 Bwj O2
Ofj Pwj Msj Poj Oj
-(A)
>i ' (B)
-(C)
-(D)
-(E)
Actual
46
49
35
28
26
Calculated1
17
12
94
16
20
-(G)
184
159
1 Number of sampling points required to provide a 15,^) standard error around the mean stand basal area.
P\v = white pine
He = hemlock
Or = red oak
Ms = sugar maple
0 = other
Sb = black spruce
Ce = cedar
Bw = white birch
Po - popular
-------
39
d) QC protocols involving regular participation in interlaboratory
comparison and analysis of standard reference samples.
Case 2 - Forest Assessment Studies
Forest decline was recognized as a serious problem in parts of North America and
other parts of the world. In 1984, the decision was made to initiate hardwood decline
studies in Ontario. The planning approach used for these studies benefitted from the fact
that the decline problem identified had a regional scope. Project leaders, therefore, had
to ensure that their work could be compared to work in other parts of North America, or
that their methods could be easily mimicked elsewhere and provide comparable data.
The subjective nature of decline assessment presented the challenge of good QA
planning.
Once again the goals and objectives were identified and the siting criteria were
defined. A total of 110 plots across the province were established. Details on plot
selection, size, markings etc. are provided in Beak Consultants 1990. To check the
quality of tree assessments by each crew, several plots had assessments done by more
than one crew. Seven plots were chosen randomly for overlapping assessments. Each
plot overlapped on more than one occasion and the overlaps occurred randomly
throughout the province and study duration (Beak Consultants 1990). A statistical
evaluation of the results gave a "quality of crew assessment". Several plots were co-
located with plots for the ARNEWS and NAMP programmes.
For site visits, each crew was trained in the standard operating procedures and
given a field procedural manual. The manual contained the names of contact
personnel, detailed descriptions of the field personnels' tasks, contingency plans, "TO DO"
and equipment lists, the Tree Assessment Methodology Manual, tree identification
package, and a hardwood disease and insect identification package (Beak Consultants
1990). Within 24 hours of visiting a site, all field notes, plot descriptions, location maps
and topographic maps were mailed to the head office. A separate file was set up for
each plot.
The primary QA effort in the decline project was an improved rating scheme for
plot assessment. Tree rating schemes have been very subjective in the past. Assessment
parameters are broad, qualitative, non-descriptive and have poor resolution (McLauglin
et al. 1988). This makes them of limited value in detecting trends. Instead, a high
resolution, quantitative rating system was developed which was reproducible and had a
narrow confidence interval within a large gradient.
-------
40
This decline index was based on the most often noted symptoms in hardwood
decline: dieback of the fine branch structure, pale green or chloritic foliage and
undersized leaves (Mclaughlin et al. 1988). The three crown parameters were individually
assessed and then combined in a numerical weighting scheme. The improved objectivity
in the assessment resulted in improved data precision, accuracy and comparability.
Precision and comparability were also increased by decreasing the variability
between technicians. Laminated field assessment templates were prepared which
illustrate a series of deciduous tree crown silhouettes in decline gradients of 10 percent
(McLaughlin et al. 1988). On the reverse side of the template, three series of colour chips
each containing six chips illustrate the range of foliar colour seen in sugar maple in
southern Ontario. With the aid of the template, the technician estimates percent decline,
chlorosis and undersized leaves. To increase the accuracy (or sensitivity) of the rating
scheme, the foliar parameters are weighted proportional to the live healthy crown. A low
amount of dieback may still give an elevated decline index if a large percentage of the
crown is chloritic. This is important because foliar abnormalities usually occur earlier than
dieback (McLaughlin et al. 1988).
Finally, field trials were performed. Ten people were trained in the use of the
decline index and identification. Ten trees were chosen in plots in Ontario and randomly
marked. All trees were rated by each person and then all the tree numbers were
changed and the trees rated again. This was repeated five times with the tree numbers
changed between ratings. The field trial was again repeated using paired evaluators
discussing the evaluation and coming up with one rating. Favourable results in the field
trial led to the adoption of this rating scheme for all hardwood decline studies in Ontario.
The improved decline index provided reduced subjectivity, faster training of field
crews, better precision and comparability and more accurate measurements. These
subjective types of studies benefit considerably from a QA/QC conscience approach to
planning. The projects can now be carried out over the long term and provide a very
valuable data set.
SUMMARY
Canadian LRTAP-related terrestrial field monitoring studies were performed as part
of well coordinated large federal and provincial programmes. Planning approaches used
over the last decade, however, were neither formalized nor standardized. Nevertheless,
with few exceptions, the studies themselves were well planned and QA/QC activities were
incorporated into the planning to some degree. Significant work, both in the field and
laboratory, was devoted to the production of good quality data, but the draft form and
informal presentation of much of this work is unfortunate. Everyone knows the frustration
that occurs when the "perfect" historical data base is found to be of limited use because
-------
41
QA/QC information is unavailable. Without proper documentation, a good data set which
is useful today may be of little use twenty years from now when it is most needed. There
is little doubt that the earlier studies would probably be done differently today using a
more rigorous QA/QC approach to planning.
Over the last decade, there has been an evolution of QA activities into a more
structured format. The advantages of a QA approach to planning became apparent in
the Ontario Ministry of the Environment's hardwood decline surveys. It is clear that
Canadian terrestrial studies started in the last five years, benefitted from an organized QA
effort. While formal QA plans help strengthen a study and ensure the integrity of the data,
it is important to emphasize that they do not, in themselves, guarantee the success of a
study. Similarly, the absence of a formal QA plan does not mean the failure of a study.
Efforts of the Federal/Provincial Research Monitoring Coordination Committee, over the
last ten years, have resulted in increased documentation of QA-related activities, thereby
improving the quality of data produced in Canadian terrestrial monitoring work. These
improvements can only help strengthen future monitoring work in Canada.
REFERENCES
Barnes, S. and Neary, A. (1989). The Effect of Sample Grinding on Extractable Iron and
Aluminum in Soils. Draft Report, Laboratory Services Branch,
Ontario Ministry of the Environment.
Beak Consultants Limited (1990). Results of the 1989 Hardwood Decline Survey for
Ontario. A final report for Air Resources Branch, Ontario
Ministry of the Environment, Toronto, Ontario.
Case Biomanagement (1990). An Investigation of the Use of Lichens and Mosses as
Biomonitors of Acidic Precipitation in Ontario. Report prepared
for the Ontario Ministry of the Environment. Report No. ARB-
180-89-PHYTO/APIOS 014-89.
Cowell, D.W. (1986). Assessment of Aquatic and Terrestrial Acid Precipitation Sensitivities
for Ontario. Environment Canada, Lands Directorate/Ministry
of the Environment, Air Resources Branch. APIOS Report
009/86, ISBN 0-7729-2113-x
Gizyn, W. (1983). Acidic Precipitation in Ontario Study, Biogeochemistry Study, Dorset,
Ontario: Sample Collection and Processing Procedures.
Internal report.
-------
42
Lozano, F.C. (1986). Total Amount and Elemental Content of Litter Fall on the Harp 4
and Plastic 1 Biogeochemical Sites. Faculty of Forestry,
University of Toronto. Presented to the Ontario Ministry of the
Environment.
Lozano, F.C. and W.J. Parton (1987). Forest Cover Characteristics of the Harp #4 and
Plastic #1 Subcatchments of the Southern Biogeochemical
Study. Final report prepared for the Ontario Ministry of the
Environment.
Lozano, F.C., W.J. Parton, J.K.H. Lau and L. Vanderstar (1987). Physical and Chemical
Properties of the Soils at the Southern Biogeochemical Study
Site. Faculty of Forestry, University of Toronto Report.
Presented to the Ontario Ministry of the Environment.
Magasi, LP. (1988). Acid Rain National Early Warning System Manual on Plot
Establishment and Monitoring. Canadian Forestry Service,
Information Report DPC-X-25, Ottawa, Ontario.
Mcllveen, W., D.L McLaughlin and R.W. Arnup (1989). A Survey to Document the
Decline Status of the Sugar Maple Forest of Ontario: 1986.
Ontario Ministry of the Environment report, Air Resources
Branch.
McLaughlin, D.L, W.I. Gizyn, D.E. Corrigan, W.D. Mcllveen and R.G. Pearson (1988). A
Numerical Decline Index Rating System to Monitor Changes
in Tree Condition of Forest Hardwood Species. Proceedings
of the Technology Transfer Conference, Toronto, Ontario,
Nov. 28, 29.
Miller, Imants (1988). North American Sugar Maple Decline Project Cooperative Work
Plan. Canadian Forestry Service and U.S. Department of
Agriculture, Forest Service.
Morrison, I. (1972). Variation with crown position and leaf age in content of seven
elements in leaves of Pinus banksiana Lamb. Canadian
Journal of Forest Research. 2:89-94.
Morrison, I. (1974). Within-tree variation in mineral content of leaves of young balsam fir.
Forest Science 20:276-278.
Morrison, I. (1985). Effect of crown position on foliar concentrations of 11 elements in
Acer saccharum and Betula alleghaniensis trees on a till soil. Canadian
Journal of Forest Research 15: 179-182.
-------
43
Neary, A. (1985). Preparation of alundum/ceramic plate tension lysimeters for soil water
collection. Canadian Journal of Soil Science. 65:169-177.
Neary, A. (ed.) (1986). Procedures Manual Terrestrial Effects APIOS. Ontario Ministry of
the Environment. Report ARB-93-86-PHYTO.
RMCC Quality Assurance Subgroup (1990). 1990 Assessment of the Quality Assurance
in Studies Monitoring the Effects of Acid Deposition in the Canadian
Environment. Draft Report November 24, 1990.
Senes Consultants Limited (1986). Acidic Precipitation in Ontario Study: Terrestrial Effects
Programme Northwestern Region. Comparison of Equal and
Proportional Volume Sampling Techniques. Report Prepared
for the Ontario Ministry of the Environment.
Senes Consultants Limited (1987a). Acidic Precipitation in Ontario Study: Hawkeye Lake
Biogeochemical Study Site. Soil Classification and
Nutrient/Element Studies. Report Prepared for the Ontario
Ministry of the Environment.
Senes Consultants Limited (1987b). Acidic Precipitation in Ontario Study: Hawkeye Lake
Biogeochemical Study Site Grove Throughfall and Stemflow
Studies July 1983 to March 1986. Report Prepared for the
Ontario Ministry of the Environment.
Senes Consultants Limited (1987c). Acidic Precipitation in Ontario Study: Hawkeye Lake
Biogeochemical Study Site Litterfall Studies July 1983 to
October 1985. Report Prepared for the Ontario Ministry of the
Environment.
Senes Consultants Limited (1987d). Acidic Precipitation in Ontario Study: Hawkeye Lake
Biogeochemical Study Site Litter Decomposition Studies
November 1984 to March 1986. Report Prepared for the
Ontario Ministry of the Environment.
Senes Consultants Limited (1987e). Acidic Precipitation in Ontario Study: Hawkeye Lake
Biogeochemical Study Site. Status Report and
Recommendations, Forest Inventory and Biomass/Bioelement
Studies. Report Prepared for the Ontario Ministry of the
Environment.
Technical Subcommittee Terrestrial Effects Working Group (1983). Overview
Biogeochemical Studies Acidic Precipitation in Ontario Study.
Internal Report.
-------
44
STATISTICAL THINKING AND TOOLS FOR IMPROVING EXCELLENCE IN
ECOLOGICAL PROJECTS
John A. Flueck
Environmental Research Center
University of Nevada
Las Vegas, NV
ABSTRACT
The topics of quality assessment, quality control, quality improvement
and overall excellence have been with us since the dawn of civilization. Over
time, the relative emphasis on these topics has changed, and today the focus
in industry is on continual product quality improvement and excellence.
Ecological projects and products have definite stages of activity, i.e., a "life-
cycle," and quality improvement is possible at all of the life stages. The
presentation will review the components for pursuing excellence, the role of
statistical thinking, and the methods and tools appropriate for continual
quality improvement. Statistical thinking will be seen to be important
throughout the stages of an ecological field project. Furthermore, many of the
present "Quality Assurance" methods and tools will be seen to be of direct
use in the pursuit of excellence within ecological projects. The presentation
will conclude with some additional tools and thoughts for pursuing excellence
in future ecological field projects.
INTRODUCTION
The attempt to control product quality is a topic that has been with us for centuries,
and it presently is receiving expanding attention and effort. However, even a brief review
of the history of the pursuit of quality indicates that attention to the quality of products has
been rather variable (e.g., Duncan, 1974; Flueck and McKenzie, 1990). It also is
interesting to note that during periods of mortal competition (e.g., war) interest in the
quality of products used in the conflict typically increases. Examples include the
shipbuilding operation of the Arsenal of Venice in the 1500's (e.g., Skrabec,1990), the
manufacture of muskets in France in the 1700's (e.g.Durfee, 1984), the extensive training
in quality control for ordnance manufacturers during WWII (e.g., Wallis, 1980), and as
noted in the newspapers even the present "Desert Storm" action has increased the
emphasis on the quality of current military products (Figure 1).
-------
45
The historical record of the pursuit of quality indicates considerable fluctuation in
the methods used to secure quality (Figure 2). During the Eqyptian pyramid building
period (ca. 2000 B.C.), the emphasis was on application of uniform methods and
procedures through strict adherence to fixed standards. The craft guilds of the middle
ages (ca. 1300 A.D.) relied upon strict training and continual oversight by the guild-master
to produce products of suitable quality. The industrial revolution of the 1800's brought
the implementation of inspection after the production event with the attendant problem of
what to do with the nonconforming units.
The 1920's brought a rapidly increasing demand for telephones, and this lead to
the idea of inspection of sampled units during the production event; the initiation of
statistical quality control (e.g., Shewhart, 1931). The 1950's gave birth to the idea of
building-in quality in all activities within a company; total quality control (e.g., Feigenbaum,
1956) and later termed total quality management. The 1960's saw the ideas and methods
of continuous quality improvement being shaped (e.g., Deming, 1982). Finally, the 1980's
brought forth an emphasis on cost-minimizing "robust" product designs (e.g.,
Taguchi,1979).
The recent decades (i.e., the 1960's and beyond) have placed the emphasis on
methods that plan for quality and build-in quality "up-front". In fact, in many industrial and
service companies, quality is now looked upon as an extended process reaching from the
quality of Design (i.e., determining the product that really will satisfy the client or
customer), through quality of Conformance (i.e., the degree to which the producer and
his suppliers are able to surpass the design specifications generated by the customers),
to the quality of Performance (i.e., the evaluation of how well the product performs in the
marketplace, Gitlow et al., 1989).
-------
46
FIGURE 1
WAR-TIME EXAMPLES OF THE IMPORTANCE OF PRODUCT QUALITY
Time Period Event Action
1500's The Arsenal of Venice Built armed galleys to
protect maritime trade
1700's Jefferson's trip Manufacture of
muskets in to
France a French
factory
1940's World War II and A major national
program in statistical
quality control for
military ordnance
suppliers
1990's Desert Storm Federal investiga-
tions of military
suppliers for failure
to properly test their
products
-------
47
FIGURE 2
A HISTORY OF METHODS USED IN PURSUIT OF QUALITY
Time Period
Event
Method
2000's B.C.
1300's
1800's
1920's
1950's
1960's
1980's
Egyptian Pyramids
Craft Guilds
Industrial Revolution
Statistical Control
Total Quality Control
Continuous Improvement
Robust Products
Compliance to fixed
standards
Training and Monitoring of
the complete project
Inspection after production
Process Inspection during
production
Build-in quality in all
activities
Expanded Shewhart-
Deming Improvement
Cycle
Taguchi "quality by
design"
In ecological monitoring and research, we need to investigate the more promising
methods and practices of others who already have tread the path of quality improvement.
Many successful stories and techniques are available in other disciplines (e.g., Harrington,
1987; Townsend, 1990; Stratton, 1990), and we need to examine them and determine
how they might serve us in our quest for excellence. In particular, we need to focus on
the opportunities for building in excellence in the planning and implementation stages of
an ecological study. We also need to move beyond the viewpoint that a "quality job" will
occur just because the "right people" are involved in the planning of the project and a
voluminous quality assurance plan is created with many stated check-points. Aside from
the question of who are the "right people" and will they fully participate, there remains the
point, as Shewhart so aptly remark many years ago, that one cannot inspect quality into
a product.
-------
48
PRINCIPLES FOR EXCELLENCE
There have been numerous discussions in the quality control literature concerning
what really constitutes quality and the methodology of Total Quality Control (e.g.,
Feigenbaum, 1961; Ishikawa, 1985). From a simple dictionary translation, quality can be
defined as "degree of excellence", and Total Quality Control can be defined as:
"the complete and unified power to manage the degree of excellence"
Thus Total Quality Control focuses on creating excellence in products and services
through teamwork. Needless to say, the above statement immediately raises a number
of issues including the questions of how is excellence of a product to be defined, how is
it to be measured, and how should it be pursued?
One approach to the question of how is excellence to be defined is to treat the
concept of "excellence" of a product as a vector with the elements of the vector
representing the various characteristics of the product relevant to its utilization (e.g., the
final product is an information one such as a newspaper and the elements of interest are
size, topics, depth of coverage, readability, accuracy, and timeliness). To measure the
amount or degree of excellence (y), one can form a weighted linear combination of the
elements (Xj) using a set of weights (a,) derived for the particular customer,
y= a^ + 83X2 + a3x3 + + akxk.
Then assuming that the x/s can be standardized and measured, one is able to compute
the excellence score for each competing product in the given market for the particular
customer. One also can compare the size of the a.'s to assess the relative importance
of each of the k characteristics of excellence.
To create excellence in a product, there are some fundamental principles that must
be mastered if a Total Quality or Quality Improvement approach is to be attempted. They
are:
1. Understanding that there are Suppliers, Products, and
Demanders (customers) in all transactions and that each has
an important role to play,
2. Realization that product excellence must be earned by
satisfying and/or delighting the customers,
3. Commitment to the concept that management leadership,
teamwork, training, and rewards are needed to secure
excellence in the jointly produced products, and finally
-------
49
4. Recognition that "statistical thinking", statistical tools, and
computerized technology are the work-horses for improving
the excellence of products.
The presentation and implementation of these principles are illustrated in numerous
writings (e.g., Deming, 1986; Harrington, 1987; Oakland, 1989; andTownsend, 1990), and
it should be noted that a Total Quality approach results in the shifting of the quality
improvement task to a broader array of people (e.g., Quality Improvement Teams) than
just the quality assurance staff.
STATISTICAL THINKING
The method of statistical thinking has been present in quality control for some time,
and Walter Shewhart (1931), W. Edwards Deming (1986), Brian Joiner (1987), and Ron
Snee (1990) are examples of people who have advocated and applied it in the quest for
quality improvement. In essence, statistical thinking is the view that virtually everything
can be seen as a process or a group of processes (i.e., a system) whose inputs and
outputs have variability, and the process can be identified, characterized, quantified,
sampled, analyzed, and understood such that it's variability can be reduced and the
process and products can be improved. A generic diagram of the steps in statistical
thinking is presented in Figure 3 (Snee, 1990), and we see that these steps actually are
an expansion of the traditional Shewhart-Deming cycle of Plan, Do, Check, and Act
(Deming, 1986). Thus statistical thinking encompasses both the art and science of
scientific problem solving.
In ecological monitoring and research studies, the idea is to apply statistical
thinking to problems in each stage of the life of a project from conception through design,
feasibility, implementation, analysis, reporting, and expost studies (Figure 4; Flueck, 1986)
to continually improve the final information products. In particular, emphasis should be
placed on building in excellence up-front in the design stage, sustaining it throughout the
life-cycle of an ecological project, and reducing the need for ex-post checking. In the
Environmental Monitoring and Assessment Program (EMAP, an EPA program to assess
the conditions of the nation's ecological resources), examples of the use of statistical
thinking and building in excellence up-front already have occurred in the design and
implementation stages of the program.
In the design stage, the problem of adequate prior information has been resolved
by the use of a two-stage sampling design with the first stage focusing on obtaining more
general information and the second stage using this information to sharpen the estimates
of ecological conditions and trends. The initial problem of providing suitable spatial
coverage has been addressed by imposing a regular triangular point grid over the entire
conterminous USA, with approximate spacing of 27km, yielding about 12,600 potential
sampling points. In addition, the grid was given a random start in order to provide a
-------
50
probability basis for spatial estimates of ecological populations (Overton et al.,1990).
These characteristics of the general EMAP design are unique and path-breaking in the
design of ecological field studies.
In the implementation stage, examples of statistical thinking include the EMAP-Near
Coastal group's use of "bar-coding" for many of the sample types with considerable
reported success. Also, the EMAP-Forest group utilized portable data recorders (PDR's)
and lap-top computers for some data taking, screening, checking, and transmission to
the home-base with a reported 90 percent reduction in data entry errors such as incorrect
code, out-of-range values, etc. (C. Liff, internal communication, September, 1990).
Clearly, many more opportunities for building in excellence up-front in each of the seven
stages of a typical ecological project await their discovery by EMAP participants and
EMAP "Improvement Teams".
TOOLS AND TECHNOLOGY
A number of tools and techniques are available for assisting in the task of
improvement of quality, and a growing number of them are routinely used in the industrial
and service industries both here in the USA and abroad. A list of these tools or methods
are presented in Figure 5 and classified by elementary, intermediate, and advanced as
presented by Ishikawa (1985). In Japan, the seven elementary or so-called "Seven
Indispensable Tools" of quality control (i.e., Pareto charts, Cause-and-Effect diagrams,
Stratification, Check Sheets, Histograms, Scatter Diagrams, and Control Charts) are
reportedly used by everyone from company directors and presidents on down to research
and development staff, foremen, and workers on the production line. The intermediate
tools largely are used by engineers and the QA staff, and the advanced tools apparently
are utilized by a very limited set of engineers and staff to solve complicated process and
quality analysis problems (Ishikawa, 1985). It appears that all of these tools have some
use in ecological monitoring and research studies, but one wonders how often they
actually get used.
In keeping with the spirit of "continuous quality improvement", four additional tools
are presented which have enjoyed considerable use in numerous disciplines including
atmospheric, climate, and medical research. These tools (i.e., flowcharts, stem-and-leaf
plots, box-and-whisker plots, and median smoothers) are briefly defined in Figure 6.
Flowcharts (Figure 7) already are being used by many quality improvement teams,
and a number of quality professionals have made them one of the elementary seven tools
(stratification typically is dropped). The remaining three tools have been extensively
utilized in exploratory data analysis activities (e.g. Tukey, 1977), and they appear to be
very applicable to ecological studies and their quality improvement efforts. It should be
noted that both the stem-and-leaf plot (Figure 8) and the box-and-whisker plot (Figure 9)
provide greatly enhanced distributional information when compared to the typical
-------
51
summary parameters of mean and variance or a histogram. In fact as shown in Figure
9, the time sequence of yearly box-and-whisker plots for krypton-85 gives both level and
variability information within the same diagram. As such, it is a distributional "control
chart", and it could become an attractive alternative for displaying grouped ecological time
series data.
The median smoother is appropriate when attempting to find the "central path" of
a time series, and because it uses a sequence of running medians (e.g., Tukey, 1977) it's
pathway is uninfluenced by outlying observations. All four of these graphical tools appear
to be suitable for tasks within quality improvement and reporting activities in ecological
monitoring and research studies.
With regard to technology, all of the tools presented in Figure 5 have available
software for their application. Furthermore, the four additional tools presented in Figure
6 also have computer code available (e.g., Velleman and Hoaglin, 1981), and a number
of statistical packages (e.g., Minitab, Stat-Graphics, S-Plus, etc.) already have
incorporated these tools into their packages. Clearly, the use of computerized routines
has an important place in both on-line and off-line quality improvement studies.
CONCLUDING COMMENTS
Numerous methods and techniques have been successful in improving the quality
of products in the industrial and service industries. Many of these methods and
techniques appear to have promise for improving the quality of ecological projects at all
stages of activity.
Ecological studies produce information as their main product, and these products
can take many forms (e.g., data, tables, visual displays, estimation results, reports, etc.).
Consequently, the goal of quality improvement activities in ecological projects should be
to utilize a Total Quality method to assist in the improvement of the content, form, and
delivery of these information products. In this respect, QA should be termed Quality
Assistance as suggested by Dan Heggem (Papp et al., 1989).
Finally, assuming that most information in ecology, and in science in general, is
fragile and perishable, the early recognition of the need for statistical thinking, appropriate
tools, and quality improvement is important. In ecological studies, as in automobile
production, it is greatly preferred to build-in excellence in the design stage rather than
attempt to retrofit while on the road.
-------
52
rigure 3. Statistical Thinking in Quality Improvement.
-------
53
START
Meet
Requirements
Fail
Requirements
The Process
of Interest
Surpass
Requirements
Not Feasible
Return to
Design
Substantial
Problems
STOP!
Figure A, The Typical Stages, or "Life Cycle." of a Research Study.
-------
54
FIGURE 5
A LIST OF ELEMENTARY, INTERMEDIATE, AND ADVANCED METHODS AND
TOOLS USED IN QUALITY IMPROVEMENT
Elementary Methods or Tools
1. Pareto chart
2. Cause and effect diagram
3. Stratification
4. Check sheet
5. Histogram
6. Scatter diagram
7. Shewhart control charts
Intermediate Methods or Tools
1. Sampling surveys
2. Inspection Sampling
3. Statistical testing and estimation
4. Sensory testing
5. Statistical design of experiments
Advanced Methods or Tools
1. Advanced design of experiments
2. Multivariate analysis
3. Operations Research methods
-------
55
FIGURE 6
A LIST OF ADDITIONAL TOOLS APPLICABLE TO QUALITY IMPROVEMENT
TOOL DESCRIPTION
1. Flowchart A formal diagram for depicting the steps and
pathways in a process or system
2. Stem-and-Leaf Plot A display of each observation in a sample
organized by leading integers (stems) with the
leaves carrying the remaining integers
3. Box-and-Whisker Plot A summary of the empirical distribution of the
sample data using quartiles (i.e. 1,2,3,& 4th) and
highlighting outliers
4. Median Smoother Running (moving) medians of varying lengths
and repeats
-------
Figure 7. Flowchart for Software Release
56
Acctpubl* foi
rtltiM
Dtftnt m»im«nwc«
0(|M\IUtlOfl IAd
rttporuibdiiMi
•CUV*
; Adipuv*
<
| Prrf*
• Uwf communication
-------
57
Figure 3. Stem and Leaf Display of Heights of Highest Points in Each State
units are in 100 ft. and rounded to nearest hundred (1 digit leaves)
stat Leaves
o**
1,3**
2 ~~
3
4
S,0*«
6 ~
7
8
9
10,0**
11 ~
12
13
14
15, 0"
16 ~
17
18
19
20,0**
3,4,5,8,8
2,3,6,7,8,8
0,0,3,4,4,8
2,4,5,5,6
0,1,4,8,9
0,3,3,4,7
3,6,7
2
8
2
6,7,8
1,2,5,8,8
4,4,5
3
Fla., D
Penna.
S. Dak.
Texas .
Oregon
Col., V*
Alaska
(5)
(6)
(6)
(5)
(S)
(5)
(3)
(D
(D
(D
(3)
(S)
(3)
Source: Statistical Abstract of the United States, 1970 U.S. Bureau
of the Census, Washington, B.C. pp. 169.
-------
Figure 9. Ueekly Kr-8S Data: Rachel, NV
32 -
28 -
e
o
528+
«.
4*
12
i
1
I
s
Source: NKI), KMSI.-KI'A
Las Vegas, NV
-t-
4-
-H
+
-\ H
-f
•*-
79 88 81 82 83 84 85 86 87 88 89
39 49 43 58 45 48 45 58 43 43 48
Sample wu«ks
01
00
-------
TEST SUBSTANCE
Location
Parameter
Receipt
Labelling
Characterization
Inventory
Custody
Contamination
Storage
Environmental Controls
Transport
Handling
Safaty Issuas
BID*
s
8
9
S
S
D
D
D
D
D
D
Laboratory
Sam* controls regardless of location
Lass change dua to controlled
environment
Faculties established
Wall monitored dua to establlshad
facility
Low chanca of acddants dua to mainly
shorter dlstancas
Not as complicated dua to close
mixing
Usually known and controlled
Field
Graater chanca dua to Interaction
with amblant surroundings
FacJUttas variable
Lass control dua to mobile nature
Greater chanca for acddants dua
to longar dlstancas
Variable formulating scenarios dua
to Incraasad distances
Accidental relaasa* more Hkely dua
to ambient anvlronment
*S=Similar, O=Different
01
CD
-------
60
REFERENCES
Deming, W.E., 1986. Out of the Crisis. MIT Press, Cambridge, MA.
Duncan, A.J., 1974. Quality Control and Industrial Statistics. Irwin, Homewood, IL
Durfee, W.F., 1984. "The History and Modern Development of the Art of Interchangeable
Constructions In Mechanisms", J. Franklin Inst.. Philadelphia, PA.
Feigenbaum, A.V., 1961, 1983 (3rd ed.). Total Quality Control. McGraw-Hill, New York,
NY. 1956. "Total Quality Control". Harvard Bus. Rev., Nov-Dec, 93-102.
Flueck, J.A., 1986. "Principles and Prescriptions for Improved Experimentation in
Precipitation Research", Ch. 16 in Precipitation Enhancement - A Scientific Challenge. R.R.
Braham, Jr. (Editor), Monograph Vol. 21, No. 43, Amer. Meteor. Soc., Boston, MA.
Flueck, J.A. and D. McKenzie, 1990. "A Total Quality Approach to Environmental
Monitoring and Assessment", submitted for publication.
Gitlow, H. and S. Gitlow, A. Oppenheim, and R. Oppenheim, 1989. Tools and Methods
for the Improvement of Quality, Irwin, Homewood, IL
Harrington, H.J., 1987. The Improvement Process, McGraw-Hill, New York, NY.
Ishikawa, K., 1985. What is Total Quality Control? The Japanese Way. Prentice-Hall,
Englewood, NJ.
Joiner, B.L., 1985. "The Key Role of Statisticians in the Transformation on North American
Industry", Amer. Stat.. 39, 224-34.
Oakland, J.S., 1989. Total Quality Management, Heinemann, Oxford, England.
Overton, W.S., D.White,and D.L. Stevens, 1990. Design Report for EMAP. Environmental
Monitoring and Assessment Program. Part I. Dept. of Statistics, OSU, Corvallis, OR.
Papp, M.L, D.T. Heggem, R.D. Van Remortel, R.W. Gerlach, J.M. Pollard, and LD.
Stetzenbach, 1989. Alaskan Oil Spill Bioremediation Plan and Standard Operation
Procedures. Draft, U.S. EPA, Las Vegas, NV.
Shewhart, W.A., 1931. Economic Control of Quality of Manufactured Product. D. Van
Nostrand, New York, NY.
-------
61
Skrabec, Q.R., 1990. "Ancient Process Control and Its Modern Implications", Quality
Progress. 23, no. 11, 49-52.
Snee, R.D., 1990. "Statistical Thinking and Its Contribution to Total Quality", Amer. Stat.
44, 116-121.
Stratton, B., 1990. "Four to Receive 1990 Baldrige Awards", Quality Progress. 23,no.12,
19-21.
Taguchi, G., 1979. Introduction to Off-line Quality Control. Central Japan Quality Control
Assoc., Magaya, Japan.
Townsend, P.L with J.E. Gebhardt, 1990. Commit to Quality. J. Wiley & Sons, New York,
NY.
Tukey, J.W., 1977. Exploratory Data Analysis. Addison-Wesley, Reading, MA.
Velleman, P.P. and D.C. Hoaglin, 1981. Applications. Basics, and Computing of
Exploratory Data Analysis. Duxbury Press, Boston, MA.
Wallis, W.A., 1980. "The Statistical Research Group, 1942-1945". Amer. Stat. Assoc.. 75,
320-333.
-------
62
STATISTICAL QUALITY CONTROL IN
ENVIRONMENTAL IMPACT ASSESSMENT:
ROLES AND OBJECTIVES
A.M. El-Shaarawi
Rivers Research Branch
National Water Research Institute
Canada Centre for Inland Waters
Burlington, Ontario, L7R 4A6
ABSTRACT
Decisions and policies for achieving specific environmental objectives are
ideally based on measurements and observations, describing the response
of the ecosystem to natural and anthropogenic perturbations. Actions based
on data of questionable quality are likely to be inappropriate for achieving the
intended goals. To avoid this situation, it is necessary to have in place,
within any environmental monitoring program, a statistical quality control and
quality assurance component which is capable not only of ensuring that the
data generated have the required quality but also of communicating the
results to decision makers.
Several approaches from the analysis of variability are described in this
paper. The aims are to be able to identify, locate, estimate and control the
sources of variation. The tools described for doing this include both
descriptive graphical methods and statistical models. Some aspects related
to the design of data quality studies will also be discussed.
INTRODUCTION
The assessment of the impact of anthropogenic activities on the health of
ecosystems is a major issue in environmental research. Besides testing various
ecological hypotheses, the detection and estimation of temporal and spatial changes in
the structure and the function of the ecosystem community are the basic objectives of
much of the research and monitoring activities. By relating observed changes to various
levels of intervention such as Load's (nutrients and contaminants) control, it is possible
to identify the most effective means of achieving management objectives. For example,
the identification of phosphorous and nitrogen as the main factors that stimulate aquatic
primary production and lead to accelerated eutrophication of lakes has resulted in the use
of phosphorous load reduction as a management tool to control the occurrence of algal
blooms in the lower Great Lakes.
-------
63
Approaches commonly employed in the assessment range from a controlled
experiment (eg. bioassays, experimental lakes, etc.) to regular and routine monitoring of
the ecosystem. The ability to extrapolate the findings from these approaches to the
system of concern is limited by not only the scope and design of the investigation but
by also the quality of the generated data. The use of data with questionable quality
carries the risk of making wrong decisions with serious consequences. This could be
avoided by including a data quality control component as an integral part of the
investigation. The objectives of such a component are to locate, estimate and control
major sources of variation. Two sources of variations can be distinguished: natural or
uncontrolled variation and controlled variation. Large uncontrolled variations are common
in the environmental studies due to the natural variability of the material under
investigation. Estimating the contributions of natural variation to overall variability should
be an essential aim of the quality assurance program. The impact of this variation on the
results can be reduced at the stage of study design through the use of replications.
The aim of this paper is to present techniques for assessing data quality. Some of
these techniques are informal (mainly graphical) while the others are formal (estimation
and hypothesis testing).
INFORMAL TECHNIQUES FOR ASSESSING DATA QUALITY
Graphs are very important informal tools for checking data values for the presence
of outliers and for identifying violations of certain assumptions. The following are
examples of graphical techniques which are commonly employed to look for small groups
of discrepant values.
Frequency distributions, histograms and boxplots
These provide: (1) a data summary (measures of locations and spread), (2)
information about possible probability models for representing the data, and (3)
identification of illogically inconsistent values.
Control Charts
The ideal for an analytical laboratory is to maintain, at appropriate levels, the
reproducibility of all the data produced at the required levels of quality. This means that
the main statistical characteristics of data (mean, variance, etc.) remain the same for all
the samples analyzed. Control charts are useful instruments for tracing causes of
variation especially when the process is not well established and all the conditions that
affect the data quality may not be known or may not have been brought under control.
A second use of control charts is for the routine control of the quality of the analytical
process and for providing evidence that the laboratory is producing results with the
required quality.
-------
64
Cumulative Sum Charts (CUSUM)
The CUSUM charts are more effective than the control charts in detecting whether a
change in the mean level has occurred and to estimate the magnitude of the change.
FORMAL ANALYSIS OF VARIATION
This section describes some methods for estimating the magnitude of variations due
to various sources. These are usually called the analysis of the components of variance.
Estimation of Two Variance Components
Suppose that n independent water samples were taken from a fixed location in a lake
and suppose that each water sample is well mixed prior to dividing it to m equal volume
subsamples. Each subsample is then analyzed in the laboratory to determine the
concentration of a specific substance. Let x.^ be the random variable corresponding to
the measured concentration in the jth subsample of the ith sample (j= 1,2,...,m, i =
1,2,....,n). The x,'s are subject to two types of errors, error due to sampling and error
due to subsamplmg and analytical work. As a model for this setup, we take
Xij = n + ei
Where /i is the mean concentration at the sampling station, e-t is the error due to
sampling and o>j is the error due to subsampling and analytical work. Under the
assumption that the errors are mutually independent with zero means. Let ae2 and aw2
be the variances of e and o respectively. The variance of x^ is then the sum of a€2 and
aw2. The object of variance component analysis is to estimate af2 and aw2 and to test
different hypotheses regarding their values. Let X; and s2 be the mean and the variance
for the ith sample, where
m
-------
65
and
m
s2 is an unbiased estimate for the subsampling variance aw2. Hence as a pooled estimate
aw2 based on all samples we have
n
Further let s2 be the between samples variance which is given by
n
~ - If)2 I (n - 1)
and
n
x = */ n
Then it can be shown that
.2 _2 -2 , __
ae = s - <3W I m
is an unbiased estimate for af .
-------
66
The ratio of the between sampling variance ae2 to the within sampling variance aw2
provides an estimate for the impact of two sources of errors on the precision of the
results generated by this set up. As an estimate of this ratio we use
X =
aw
As an example, consider the hypothetical data of Table 1. The rows in the Table
correspond to the subsamples while the columns represent the samples. The estimates
of the means x, and the variances s2 are given as the two bottom rows of the Table. The
estimates of a 2 and a 2 are respectively. Hence
A, = -1 = 7.519
.2
°w
This shows that the between samples variation is more than seven times of that due to
subsampling.
Design Consideration
How can the above information be used to set a routine monitoring program? The
answer depends on the objectives of the program. For illustration we consider that the
program is designed to either estimate: (1) the mean at the sampling station, or (2) the
components of variance. Also the case of estimating both the mean and the components
of variance will be discussed. The approach considered here is to minimize the variance
of the quantity that we intend to estimate. In all cases the assumption is made that
c = mn is fixed (e.g., the number of subsamples to be analyzed is a priori fixed in
advance).
-------
67
Estimation of the Mean
The variance of the overall mean x is
2
m
2
n
(1)
=The variance within sample + Variance between samples
no. of subsamples no. of samples
Since c = mn is fixed, then the optimum design determines m or n so that a2 is
maximized. The optimum value of m is
m =
\
which is estimated by
m =
\
and
Since A is 7.519 and c = 54, then
Wl
m =
\
—.519 = 2.68 » 3
Hence n, the estimate of n, is 18.
-------
68
Estimation of the Variance Components
To estimate n or m, the variances of aw2 and oe2 as well as the Covariance
between them are needed. For simplicity it is assumed the e, and Wj are normally
distributed. It is easy to show that
4
. _ o. bOu/
Var (O = TTITTT-
2
2
2*1
and
<*» a
.
Minimization of the determinant of the variance-covariance matrix is used to
determine the optimum value of m and n subject to
the condition c=mn. This determinant is
The value m = (2Ac + c + 1)/(Ac+A + c) maximizes the function f(m). Note that for A-> INF,
the we have m= 2c/(c+1) and when A.=0, m= c+1/2. Also this shows that for A,-> INF
and c-» INF we have m-»2, which is the minimum possible value of m for the estimation
of aw2. Indeed the optimum value of m is a decreasing function of X. This is expected
due to the relative increase in the sampling errors as compared to the subsampling
errors.
For the data in Table 1, the value m=2.09 is the estimate of optimum number of
subsamples required in future monitoring. The values of the second column of Table 2
are proportional to f(m). These indicate a serious loss of efficiency occurs if m is larger
than 4.
-------
69
Estimation of the Mean and the Components of Variance
Because of the independence of the mean x and the estimates of the variance
components, the determinant of the variance-covariance matrix of all parameters is
obtained directly by multiplying expressions (1) and (2). so the quantity which we need
to minimize for the variation of m is
/ (m) = f (m)o2
(C
The value of m that maximizes l(m) is one of the roots of the equation
2A. (mJi + 1)~1 + 2A/77 (c + Jim)'2 + (c-m)-1 - (m-1)-1 - m'1 = 0
This equation has to be solved numerically. In Table 2, Column 3, the values of l(m)
c2/Aaw10 are given as a function of m, which show that the minimum value of the
determinant occurs between m=2 and m=3.
Estimation of Three Variance Components
The previous analysis can be extended easily to the case of estimating more than
two variance components. To show how this is done, the case of three variance
components is outlined in this section. Let xijk be the measured concentration of the kth
subsample of the ith sample which was determined in the jth laboratory where i = 1,2,...,n;
j = 1,2,...,l and k = 1,2,...,m. Three types of errors are expected to influence the
determination of xijk. These are errors due to sampling, subsampling and the analytical
work in the laboratory. Assuming additivity, the model of xijk can be written as
*Hk = V- + ei + Y/ + <•>*
The meanings of M, ef, wk, are given before while y= refers to the errors due to the jth lab.
It is also assumed that y. is independent of ei and o>k and has zero mean and variance
a,2. The estimates of au , aY2 and ae2 are respectively
-------
and
where
70
n I m
EEC**- ~*f Inl
-2
-2 2
fi - ^
/ m/
t = E E
1=1 7=1
n
4-
/77
= E
-------
71
and
n
x,.. / n
In Table 3, the data of Table 1 are modified for the purpose of illustrating the
process of estimating three variance components. In Table 3, where three samples were
taken from a single sampling location, and each sample is divided into three subsamples
with each subsample being given for analysis to one of three different labs at random.
Within each lab the subsample is further subdivided to 6 subsamples for the within lab
replicate analysis. Applying the formula's given above yields
o* =0.01896, o* =0.00816
and
6?=0.813
These estimates show that most of the variation is caused by sampling.
Design Considerations
Under the condition that the total number N = nml of laboratory determinations is
fixed, a sampling design for estimating the mean M, minimizes the variance of x... which
is
222
n I m
The minimum occurs when
and
-------
72
2 2
and
m =
No,
2 2
Using these formulas for the data on Table 3 yields
n =
54(0.813)2 ]j
(.00816 x .813)]
=1.43 - 2
/ = 0.62 - 1
and
54 - 2 = 27
In this example it is not realistic to regard I as a variable to be estimated. If this was the
case then optimum design will be exactly the same as in the case of estimating two
components of variance.
-------
73
Table I. Hypothetical Data for the Examples
1
s
u 2
b
s 3
a
m 4
P
1 5
e
s 6
y ]
1
1
1
1
1
1
1.59
.80
.72
.69
.71
.83
L.723
S2 .00727
2
1.72
1.40 1
2.02 I
1.75 1
1.95 1
1.61 1
1.742
.05110
3
1.59
.50 2
.50 2
.49 2
.47 2
.63 2
1.530
.00412
SAMPLE
4 5
2.44 2
.11 2.
.41 2.
.48 2.
.36 2.
.36 2.
2.36
.01716
.27
70
36
36
16
04
6
2
2.
2.
2.
2.
2.
2.315
.05079
.46
21
50
37
24
24
2.
7
1.36
1.43
1.48
1.55
1.53
1.39
338 1
8
1.73
1.74
1.65
1.58
1.49
1.70
.457
.01518 .00583 .
9
1.53
1.41
1.64
1.51
1.52
1.36
1.648
0095 .
Sft2 = .01896
SB2 = -.14256
o\= Si-— = 0.14256
n
-2
= -^ = 7.519
n = 2.086
-------
74
Table 2. The dependence of various variances on the number of subsamples
Values are proportional to
subsamples the mean
No. of
m
2
3
4
5
6
10
15
20
25
30
35
40
45
50
53
Variance of
0.77
0.75
0.81
0.91
1.00
1.49
2.16
. 2.83
3.52
4.21
4.90
5.95
6.29
6.98
7.40
f(m)
4.95
5.44
6.44
7.60
8.86
14.66
23.71
35.47
51.31
73.76
108.02
166.77
290.81
724.96
3069.34
l(m)
3.85
4.09
5.20
6.81
8.88
21.88
51.11
100.56
180.66
310.55
529.53
933.04
1828.65
5061.67
22708.93
-------
75
Table 3. Data for the three variance components
SAMPLE
LAB
1
LAB LAB
2 3
LAB LAB LAB
456
LAB LAB
7 8
LAB
9
1
S
u 2
b
s 3
a
m 4
P
1 5
e
s 6
1.59 1.72 1.59
1.80 1.40 1.50
1.72 2.02 1.50
1.69 1.75 1.49
1.71 1.95 1.47
1.83 1.61 1.63
2.44 2.27 2.46
2.11 2.70 2.21
2.41 2.36 2.50
2.48 2.36 2.37
2.36 2.16 2.24
2.36 2.04 2.24
1.36 1.73 1.53
1.43 1.74 1.41
1.48 1.65 1.64
1.55 1.58 1.51
1.53 1.49 1.52
1.39 1.70 1.36
-------
76
Comparison of Alternative Approaches for Establishing
Measurement Quality Objectives for Ecological Studies
C.J. Palmer and B.L Conkling
Environmental Research Center
University of Nevada
Las Vegas, Nevada
ABSTRACT
The planning approach used in the establishment of measurement
quality objectives for ecological studies influences not only the allocation of
resources but also the overall success of the program. One approach is to
assume that natural population variability is the limiting factor to be
addressed in statistical sampling designs. A second approach assumes that,
in addition, measurement error will be a limiting factor. As a result, significant
resources may be allocated in these studies to reduce measurement error.
A third approach is to set a goal of making measurement error statistically
insignificant relative to population variability. This has the advantage of
identifying when resources allocated to continued improvements in reducing
measurement error will no longer be needed in studies. A requirement of this
approach is the identification of populations of interest and the ability to
determine overall measurement error through replicate routine samples and
reference samples.
INTRODUCTION
Experimental data all have an overall variance associated with them. The overall
variance (a^) is composed of the population variance (a*) and the measurement system
variance (a~,) or measurement error (Van Ee et al., 1990). The a^ includes both spatial
and temporal variation. The a£ consists of the sampling method, preparation, and
analytical error associated with data collection. It has been proposed that a measured
value can be considered as essentially error-less for most uses if the uncertainty in that
value is one-third or less of the permissible tolerance for its use (Taylor, 1987).
Measurement quality objectives (MQOs) are defined in terms of precision,
accuracy, detectability, representativeness, comparability, and completeness. These
characteristics were formerly defined as data quality objectives (DQOs). Usually,
however, DQOs are now applied to overall program objectives while MQOs are defined
for specific measurement parameters.
-------
77
When establishing measurement quality objectives for environmental studies, three
basic approaches have been identified to balance the acceptable variance, the number
of samples, and the cost. These three approaches will be compared. The model used
assumes that unbiased samples are collected so that sampling fluctuation is a function
of population variance.
THEORY
Approach 1 may be expressed as follows:
For: o20 = o2p + a£,
Assume: o* » o^,
Then: a* ~ a*.
In this approach the population variance is assumed to be much greater than the
measurement variance, or the measurement error of currently available methods is
assumed to be negligible. Therefore, the overall variance must be approximately equal
to the population variance. Since the measurement error is assumed to be negligible,
there is no need to improve the measurement system. Project resources are best spent
on taking more samples, either at more sites or more often, to address the natural spatial
or temporal variability in the population of interest.
The second approach may be expressed as follows:
For: o20 = al + a*,
Assume: a* > / a*,
Then: a* * a*.
In this approach the measurement error is not assumed to be negligible. The population
and measurement errors then become approximately equal in importance to the
explanation of overall variance. Reducing measurement error is the major focus.
Procedures are chosen which will result in the highest precision, lowest detection limit,
and least bias. Therefore, significant resources are spent on obtaining the highest quality
data.
Approach 3 (Taylor, 1987) may be expressed as follows:
For: al = o2p + a*,
Set as goal: 10 = 9 + 1,
or: 10 = 32 + 12,
where: ap = 3 and am = 1.
-------
78
This approach focuses on the balance between the population and measurement error
components. The goal is to make the measurement error statistically insignificant.
However, once that goal is achieved, continuing to reduce the measurement error has
little benefit in addressing overall variance, as shown in the following example.
Example:
When:
,-r2 - ,-r2
ao - CTp
10 = 9 + 1
If a£ is reduced by \, then
9.5 = 9 + 0.5.
This approach provides the ability to identify the focus for quality assurance resources.
Pilot studies are needed, however, to estimate the values of al and a£. Table 1 lists
some advantages and disadvantages for the three approaches discussed above.
Table 1. Advantages and disadvantages of the three approaches.
Advantages
Disadvantages
#1 Minimizes effects of ol
#2 Minimizes effects of
#3 Provides proper balance
a£ could be problem
Op could be problem
Requires estimates of am and ap
METHODS
To examine different approaches for establishing MQOs, soils data were examined
from the Direct/Delayed Response Project (DDRP). The DDRP was conducted as part
of the Environmental Protection Agency's (EPA) Aquatic Effects Research Program
(AERP) under the congressionally-mandated National Acid Precipitation Assessment
Program (NAPAP) as described in Church et al. (1989). The populations of interest were
soil classes across a region. The example data, from one of the 148 sampling
class/horizon combinations used in the Mid-Appalachian region (Byers et al., 1990), are
from A horizon soil samples collected from deep, well-drained ultisols and alfisols with no
fragipan (sampling class TWO).
-------
79
Two measurement parameters are used in the following example. The first is
exchangeable magnesium (MG_CL) determined with an unbuffered 1 mole/liter
ammonium chloride solution and soil to solution ratios of approximately 1:13 for mineral
soils, such as these, or 1:52 for organic soils. Atomic absorption or inductively-coupled
plasma atomic emission spectrometry was specified. The second parameter is
extractable silicon (SI_AO) determined by an ammonium oxalate-oxalic acid extraction
using a 1:100 soil to solution ration. Inductively-coupled plasma atomic emission
spectrometry was specified. The class data values, mean, standard deviation, and
variance are listed in Table 2.
Table 2. Data for MG_CL and SI_AO from the DDRP Mid-Appalachian Soil Survey.
X
Sn
MGCL
cmole/kg
0.222
0.433
0.102
0.104
0.105
0.414
1.801
= 0.4544
= 0.5655
= 0.3198
;i AC
wt%
'0.024
0.033
0.025
'0.008
0.060
0.056
0.023
0.0257
0.0286
8.185X10'4
x =
so =
Measurement error in the Mid-Appalachian DDRP samples was estimated using the
sampling scheme shown in Figure 1. Duplicate samples were taken in the field and at the
sample preparation laboratory. Reference samples were also used at the field,
preparation laboratory, and analytical laboratory levels to allow the estimation of bias. The
overall variance (a*) was calculated by pooling estimates over all of the 148 sampling
class/horizon groups. Since the variance was concentration dependent, the data
collection errors were partitioned into concentration intervals to evaluate the data
uncertainty associated with the routine data (a£). The error terms from each interval were
pooled and weighted by the proportion of routine samples within the corresponding
intervals (Byers et al., 1990). This approach adjusted for differences in the distribution of
the routine and QA samples across the observed concentration range. The population
coefficients of variance (CV) averaged from 100-150% over a region, which is not
unexpected for soils data.
-------
80
RESULTS
The DDRP data (Table 2) can be used to illustrate the use of approach 3. The
MG_CL data are an example of low measurement error. The population is all soil class
x horizon combinations in a region,
where: S0 = 0.474 meq/100g
Sm = 0.031 meq/100g
and: S2 = S2 + S2
(0.474)2 = S2 + (0.031 )2
Set: S2 = 1,
Then: 234 = 233 + 1.
Figure 2 (Byers et al., 1990) shows the range and frequency distribution of audit samples
and of the routine MG_CL data partitioned into concentration intervals at uniform variability
and their relation to pooled precision estimates. It can be seen that the standard
deviations are near zero for the audit and duplicate samples, indicating low measurement
error. It should be noted that this parameter was measured during all three
of the DDRP soil surveys. Since the Mid-Appalachian survey was the third survey, the
measurement protocol for MG_CL had been refined during the previous two surveys.
The SI_AO data are an example of high measurement error. The population of
interest is again all soil class x horizon combinations in a region,
where: S0 = 0.029 wt%
Sm = 0.013 wt%
and: S2 = S2 + S2
(0.029)2 = S2 + (0.013)2
Set: S2 = 1,
Then: 5 = 4 + 1.
Figure 3 (Byers et al., 1990) shows the range and frequency distribution of the SI_AO
data. The standard deviations of the audit and duplicate samples are all very different
from zero, indicating high measurement error. Indeed, the audit and duplicate sample
standard deviations are higher than some of the routine sample standard deviations for
certain soil sampling class/horizon combinations. The Mid-Appalachian survey was the
only DDRP soil survey which included determination of SI_AO. Therefore, the protocol
did not have the refinement seen in the MG CL determination.
-------
Estimating Measurement Error - oft
Field Collection
Routine
Field Duplicate
Sample Preparation
Routine
Prep
Duplicate
Field Duplicate
Prep
Duplicate
Laboratory Analysis
Routine
Prep
Duplicate
Field Duplicate
Prep
Duplicate
Figure I. Sampling scheme used for estimation of measurement error.
00
-------
82
CONCLUSIONS
A comparison of three approaches for establishing measurement quality objectives
reveals the importance of balancing the population and measurement components of the
overall variance. Approach 3 allows these components to be balanced using estimates
of overall variance and measurement variance. These estimates may be obtained from
pilot studies, which are a vital part of environmental studies. Approach 3 also allows a
focus on the client; it provides a clear method for defining high quality data.
A key factor in the implementation of the third approach is the proper identification
of the population of interest and the overall measurement system. The population of
interest is generally the lowest aggregation of data for which estimates are made. In the
case of the Mid-Appalachian soil survey, aggregations of data could be made for soil
horizons within sampling classes, overall pedons within sampling classes, across different
sampling classes with similar characteristics, or for the overall region as a whole. In this
case the lowest aggregation is the soil horizon/sampling class combination. In a similar
manner, the components of the overall measurement error need to be identified. This
needs to include all possible measurement error from the field to the laboratory.
With monitoring programs the population of interest may be the changes at a given
site over time. In this case the measurement error needs to include all measurement
errors from the field to the laboratory at any point in time plus changes in the
measurement system over time. The use of reference materials that span the monitoring
intervals is critical to the evaluation of this component of measurement error.
REFERENCES
Byers, G.E., R.D. Van Remortel, M.J. Miah, J.E. Teberg, M.L Papp, B.A.
Schumacher, B.L Conkling, D.L. Cassell, and P.W. Shaffer. 1990. Direct/Delayed
Response Project: Quality assurance report for physical and chemical analysis of soils
from the Mid-Appalachian region of the United States. EPA 600/4-90/001, U.S.
Environmental Protection Agency, Washington, D.C.
Church, M.R, K.W. Thornton, P.W. Shaffer, D.L Stevens, B.P. Rochelle, G.R.
Holdren, M.G. Johnson, J.J. Lee, R.S. Turner, D.L Cassell, D.A. Lammers, W.G.
Campbell, C.I. Lift, C.C. Brandt, LH. Liegel, G.D. Bishop, D.C. Mortenson, S.M. Pierson,
and D.D. Schmoyer. 1989. Future effects of long-term sulfur deposition on surface water
chemistry in the northeast and southern Blue Ridge province: Results of the
Direct/Delayed Response Project. EPA/600/3-89/061, U.S. Environmental Protection
Agency, Washington, D.C., 887 pp.
Taylor, J.K. 1987. Quality assurance of chemical measurements. Lewis
Publishers. Chelsea, Michigan. 328 pp.
-------
83
Van Ee, J.J., LJ. Blume, and T.H. Starks. 1990. A rationale for the assessment
of errors in the sampling of soils. EPA/600/X-90/203, U.S. Environmental Protection
Agency, Washington, D.C.
NOTICE
The information in this document has been funded wholly or in part by the United States
Environmental Protection Agency under cooperative agreement (CR81470) with the
Environmental Research Center of the University of Nevada at Las Vegas. It has been
subjected to Agency review and approved for publication.
-------
MG_CL
Exchangeable Magnesium in Ammonium Chloride
Routine Samples
o>
o
o
0)
c
o
"^
CO
(0
•o
c
CO
35
o
1
Mean (meq/100g)
Mineral Routine S/H Groups
Organic Routine S/H Groups
Field Dups. Lab Audits (Within-Batch)
Preparation Dups. Lab Audits (Between-Batch)
Note: One mineral, one organic sampling class/horizon (S/H) groups exceed plot boundaries.
Figure 2. Range and frequency distribution of MG_CL for sampling class/horizon routine sample data partitioned into
concentration intervals or uniform variability and their relation to pooled precision estimates. (Bycrs ct ul. 1990)
oo
-------
0.06
0.04
-------
86
FOURTH ANNUAL ECOLOGICAL QUALITY ASSURANCE WORKSHOP
PLENARY SESSION B
Thursday, February 27, 1991
-------
87
NOAA National Status and Trends Program Quality Assurance Program
Dr. Adriana Cantillo
Quality Assurance Manager
NOAA/NOS/OOMA
N/OMA3
Rockville, MD 20852
ABSTRACT
NOAA's National Status and Trends (NS&T) Program for Marine
Environmental Quality determines the current status of, and any changes over
time in the environmental health of the estuarine and coastal waters of the
United States. The Quality Assurance (QA) Program is one of NS&T's four
major components. The QA Program is designed to document sampling and
analysis procedures, and to reduce intralaboratory and interlaboratory
variations. To document laboratory expertise, the QA Program requires all
NS&T laboratories to participate in a continuing series of intercomparison
exercises utilizing a variety of materials. Some non-NS&T laboratories
voluntarily participate in the QA Program, and additional monitoring
laboratories are welcome. Selected results of the intercomparison exercises
will be described.
NOAA'S NATIONAL STATUS AND TRENDS PROGRAM
NOAA's National Status and Trends (NS&T) Program for Marine Environmental
Quality determines the current status of, and any changes over time in the environmental
health of the estuarine and coastal waters of the United States, including Alaska and
Hawaii. The NS&T Program consists of four major components: the Benthic Surveillance
Project, the Mussel Watch Project, Bioeffects Surveys, and the Quality Assurance (QA)
Program.
The Benthic Surveillance Project determines concentrations of contaminants in
sediments and bottom-dwelling fish taken in the same area at sites located around the
nation. The frequency of external disease conditions and internal lesions (liver tumors) in
the bottomfish are also being documented. Currently, there are about 75 Benthic
Surveillance sites in estuaries and coastal waters, including both urban and rural areas.
Samples are generally collected biennially at these sites. Sample collection and analysis
for the Benthic Surveillance Project is done by the NOAA National Marine Fisheries
Service laboratories at Seattle, WA, and Beaufort, NC.
-------
88
The Mussel Watch Project determines the same contaminants as in the Benthic
Surveillance Project in sediments and mussels or oysters, instead of fish. The bivalves are
collected on a yearly basis from approximately 220 sites in the United States, while
sediments are collected at the same sites on a less-than-yearly basis. Sample collection
and analysis for the Mussel Watch Projects is done by the Texas A&M University
Geochemical and Environmental Research Group, College Station, TX, and the Battelle
laboratories at Duxbury, MA, and Sequim, WA.
Over 77 contaminants are determined by the NS&T Program, including organic
chemicals such as DDT and its metabolites, other chlorinated pesticides, polychlorinated
biphenyls (PCBs), polycyclic aromatic hydrocarbons, and butyltins, as well as trace and
major elements, such as Pb, Zn, Cd, Ag, As, and Hg (Table 1).
QUALITY ASSURANCE PROGRAM
The quality of the analytical data generated by the NS&T Program is overseen by
the QA Program component, which is designed to document sampling protocols and
analytical procedures, and to reduce intralaboratory and interlaboratory variation. The QA
Program documentation will also allow eventual comparison between different monitoring
programs with similar QA activities and thus will extend the temporal and spatial scale of
such programs. To document laboratory expertise, the QA Program requires all NS&T
laboratories to participate in a continuing series of intercomparison exercises utilizing a
variety of materials. The organic analytical intercomparison exercises are coordinated by
the National Institute of Standards and Technology (NIST), and the inorganic analytical
intercomparison exercises by National Research Council (NRC) of Canada.
Every year, a set of calibration solutions and unknown samples are sent to each
laboratory participating in the intercomparison exercises. The type and matrix of the
samples changes yearly. Sample types have included freeze-dried sediments, extracted
freeze-dried tissues, and frozen tissues. Matrices have included mussel, oyster, fish
tissue, and sediments from pristine and contaminated areas. The initial samples of a given
exercise are sent to the laboratories early in the spring, with complete handling
instructions and data reporting format. Only when a laboratory successfully finishes the
analysis of the first set of samples of a given exercise will the next set be sent. If problems
are encountered during any of the phases of the intercomparison exercises, the
laboratories can contact NIST or NRC for assistance. The results of the intercomparison
exercises are discussed among NIST, NRC, and the participating laboratories during the
yearly QA Workshop held in late fall or winter. During this meeting, a consensus is
reached between NIST, NRC, NOAA, and the laboratories as to the type of materials that
will be used for the following year's intercomparison exercise. Some of the materials used
for intercomparison exercises have become standard reference materials based on the
results of the exercises.
-------
89
METHODOLOGY
NS&T does not specify analytical methodology. Laboratories can use any analytical
procedure as long as the results of the intercomparison exercises are within certain
specified limits of the consensus values. This allows the use of new or improved analytical
methodology or instrumentation without compromising the quality of the data sets. It also
frees the contractor laboratories to use the most cost-effective methodology while
generating data of documented quality. The analysis of reference materials, such as the
NIST Standard Reference Materials (SRMs) and NRC Certified Reference Materials
(CRMs), or of control materials generated for use by NS&T labs as part of the sample
stream, is required. All analytical methodology and sampling protocols used are fully
documented for future reference. This will provide a record of the continuity of expertise
as personnel and instrumentation changes take place. The results of the routine analysis
of reference and control materials, and of the intercomparison exercises, are stored
electronically as part of the NS&T database.
RESULTS
Intercomparison exercises have been taking place since 1985. Overall, the
performance of "core" laboratories, those that have participated since the beginning of the
QA Program intercomparisons exercises, has improved with time. It is not possible,
however, to document this statistically since different types of materials are used each
year and the difficulty of the analyses has increased with time as the level of expertise of
the participating laboratories has improved. Thus, possible analytical errors due to matrix
interference, analyte level, and other variables change from year to year.
Table 1. Chemicals determined as part of the NOAA National Status and Trends
Program
Polyaromatic hydrocarbons
Biphenyl Fluoanthene
Naphthalene Pyrene
1 -Methylnaphthalene Benz(a)anthracene
2-Methylnaphthalene Chrysene
2,6-Dimethylnaphthalene Benzo[a]pyrene
Acenaphthene Benzofejpyrene
Acenaphthylene Perylene
2,3,5-Trimethylnaphthalene Dibenz[a,h]anthracene
Fluorene Benzo[b]fluoranthene
Phenanthrene Benzo [kjfluoranthene
1 -Methylphenanthrene lndeno[1,2,3-cd]pyrene
Anthracene Benzo[g,h,i]perylene
-------
90
DDT and metabolites
0,p'-DDD
o,p'-DDE
0,p'-DDT
p,p'-DDD
p,p'-DDE
p.p'-DDT
Chlorinated pesticides other than DDT
Aldrin Dieldrin
a-Chlordane trans-Nonachlor
Heptachlor Lindane
Heptachlor epoxide Mirex
Hexachlorobenzene
Polychlorinated
congeners
biphenyls
PCB-8
PCB-44
PCB-101
PCB-128
PCB-179
PCB-195
PCB-18
PCB-52
PCB-105
PCB-138
PCB-180
PCB-206
PCB-28
PCB-66
PCB-118
PCB-153
PCB-187
PCB-209
Elements
Al
Cd
Se
Fe
Cr
As
Cu
Sn
Pb
Zn
Hg
The results of the fourth round of intercomparison exercises for trace metals in
sediments and tissues were made available in the fall 1990 (Berman, 1990). Two samples
of sediment and oyster tissue were sent to each of the participating laboratories. The
samples were sediment from the Beaufort Sea, a calcareous sediment collected in San
Antonio Bay, TX, and a composite oyster tissue of specimens collected over the entire
Gulf of Mexico. The oyster tissue was freeze dried and homogenized. The participants
were asked to perform five replicate analyses of each sample and of two NRC Certified
Reference Materials. The sediments were analyzed for Al, Cr, Fe, Cu, Zn, As, Se, Cd, Sn,
Hg, and Pb; analysis of Si, Mn, Sb, and Tl were optional. The oyster tissue was analyzed
for the same suite with the addition of Ag. Ten laboratories reported results.
-------
91
Typical results of the 1990 trace metal intercomparison exercise are shown in
Figure 1. The Zn and As analyses results of one of the sediment samples and the oyster
tissue sample are shown in Figure 1. In all four examples, results from a "core" lab, a
"new" lab, and NRC are shown. The performance of the "core" lab is comparable to that
of the NRC, while that of the "new" lab needs improvement. Major advances were noted
in the determination of As, Se, Ag, Sn, Sb, Hg, and Pb. No apparent improvement was
noted in the 1990 exercise in the determination of Al, Cr, Fe, and Ni which were already
being competently determined. There are still problems in the analysis of Se, Sn, Sb, and
Hg in sediments. Also improvement in the variance of Al and Fe determinations is needed.
The analysis of Al, Si, Cr, Cu, As, Se, Sn, Sb, and Hg in tissues still pose difficulties.
The results of the fifth round of intercomparison exercises for trace organics in
sediments and tissues were made available in the fall 1990. Sixteen laboratories
participated, although some only reported partial results. An enriched bivalve tissue
extract in methylene chloride, and a solution of 6 aromatic hydrocarbons, 6 chlorinated
pesticides and 6 PCB congeners in 2,2,4-trimethylpentane, were sent to the participating
laboratories. The tissue extract was enriched for the same 18 compounds present in the
2,2,4-trimethylpentane solution. The control materials used were a sediment collected in
Baltimore Harbor, a composite Boston Harbor mussel tissue material and NIST SRM
1974. Typical results of the organic intercalibration exercise are shown in Figure 2. The
mean absolute percent errors are much better overall for the "core" lab than for the "new"
lab, although the performance of the NIST is better than that of the "core" lab. The results
of the same "new" and "core" laboratories were used in Figures 1 and 2. Note, however,
that different chemists performed the analyses. Further evaluation of the results of the
exercise are still underway.
FUTURE DEVELOPMENTS
The intercomparison exercises will continue on a yearly basis, following the
established time sequences and final workshop.
The Environmental Protection Agency is implementing a new program, the
Environmental Monitoring and Assessment Program (EMAP), to provide the public,
scientists and Congress with information that can be used to evaluate the overall health*
of the Nation's ecological resources. EMAP will focus on indicators of ecosystem health,
while NS&T focuses on chemical pollutants. The near coastal component of EMAP,
EMAP-NC, is working closely with NS&T in monitoring coastal ecosystems. To assure
data compatibility, the EMAP-NC laboratories are participating in NS&T's QA Program and
financing part of the work.
Some non-NS&T or non-EMAP laboratories currently participate voluntarily in the
QA Program and additional monitoring laboratories are welcome as the QA Program
expands during 1991. The 1991 organic intercomparison exercise materials include a fish
tissue, an extract of mussel tissue, and calibration solutions. A contaminated sediment,
-------
92
a bivalve tissue, and calibration solutions will be used for the inorganic intercomparison
exercises. The expansion effort for the QA Program will continue during 1991. For further
information regarding participation in the intercomparison exercises, please contact the
author, the NS&T Quality Assurance Coordinator.
REFERENCES
German, S. (1990) Fourth round intercomparison for trace metals in marine sediments and
biological tissues NOAA/BT4. Unpublished report, NOAA/NOS/OAD, 50pp.
-------
93
APPLICATION OF QA CONCEPTS FROM THE THIRD ECOLOGICAL
WORKSHOP TO A LONG-TERM AQUATIC FIELD ASSESSMENT PILOT STUDY
James B. Stribling and Michael T. Barbour
EA Engineering, Science, and Technology, Inc.
15 Loveton Circle
Sparks, Maryland 21152
ABSTRACT
The U.S. EPA Region III Regional Applied Research Effort
(RARE) project was initiated in July 1990 and is intended tp aid
in addressing questions relating to the effects of coal surface-
mining on the biological integrity of stream ecosystems. Results
of this joint project between the U.S. EPA and the State of West
Virginia will also be used to help assess the effectiveness of
pollution abatement requirements. In this paper, QA/QC factors
from the Third Annual Ecological QA Workshop are used to
evaluate the quality assurance status of the RARE pilot study of
31 July and 1 August 1990. Also, a broad project overview is
presented with a focus on the proposed 5-year project duration.
INTRODUCTION
The development of an effective Quality Assurance Plan at the onset of conducting
an ecological study is necessary to provide guidance throughout the complex program,
delineate lines of responsibility, and to establish accountability. The efficacy and validity
of the ecological study and resultant conclusions are dependent upon the quality
assurance plan. For ecological studies the major classes of activity of a QA plan are:
Quality Management
Sampling Design
Field Operations
-------
94
Laboratory Activities
Data Analysis
Reporting
Each have various quality control (QC) elements which control for potential sources
of error. Technical interaction of the implemented QC elements with these potential error
sources were presented in matrix form in the document produced by Workgroup 2 in the
Third Ecological QA Workshop held in Burlington, Ontario in April 1990 (Barbour and
Thornley 1990).
The determination of data quality is accomplished through the development of data
quality objectives (DQOs). DQOs are qualitative and quantitative statements developed
by data users to specify the quality of data needed to support specific decisions (Plafkin
et al. 1989). The data quality objective (DQO) logic process resulting from the third
workshop consists of several separate but linked thought processes which include (A)
statement of the problem to be resolved, (B) identification of potentially pertinent variables
and selection of those to be measured, (C) development of a logic statement, (D)
specification of an acceptable level of uncertainty, and (E) optimization of research design
(Figure 1).
Statement of the problem is the central issue, which relates to the overall objective
of the study and the ultimate decision once a judgment is made. Characterization of the
problem includes stating the specific question or questions to be addressed which help
partition the problem into appropriate components and may pertain to such things as use
designation or biocriteria. If historical data exist, they should be acquired and evaluated
for utility in designing appropriate questions.
The next step in the DQO logic process is the identification of all variables
potentially affecting the problem. These are considered in either abiotic or biotic terms
and are related to the central question. From this set, those which will be measured and
evaluated in the project are selected. The selection process may be driven by resource
limitations and/or available expertise in measuring certain variables. Logic statements are
developed for each and include the variable itself, the measured value, the judgment
domain or criterion, and guide for the decision-making process.
-------
State
Problem to be resolved
I
Identify
Possible Variables to be Measured
I
Select
Variables to be Measured
I
Develop
Logic Statement
i
Specify
Acceptable Uncertainty
1
Optimize
Research Design
Figure 1. Data Quality Objective logic flow (Barbour and
Thornley, 1990).
-------
96
Given available resources and/or cost investment, the level of acceptable
uncertainty must be considered and documented. The level of confidence associated
with decisions on all measured variables will ultimately affect the level of confidence in
addressing the central question. This level of confidence is related to error analyses and
the consequences of that error. Two types of error are recognized-false negatives and
false positives. False negatives are when conditions exist and remain undetected.
Consequences of this error could be unabated degradation. The latter is the detection
of a condition when, in fact, none exists. False positives could lead to the requirement
of unnecessary mitigation and the potential loss of credibility for the investigator.
In this paper, an overview is presented of the Quality Assurance plan and the data
quality objective logic flow that was incorporated into the initial project work plan
developed by U.S. EPA Environmental Monitoring Systems Laboratory (EMSL-Cincinnati)
for the West Virginia Regional Applied Research Effort (RARE project) and into the pilot
study conducted in 31 July and 1 August 1990. The framework for this analysis resulted
from the Third Ecological QA Workshop (Hart 1990) and includes the data quality
objective logic process and quality control elements from those proceedings.
THE EPA REGION III RARE PROJECT
Objective
The purpose of this project is to investigate the effects of surface-mining activities
on the biological integrity of the aquatic community. Results of this study will be used by
EPA (Region Ill-Wheeling and EMSL-Cincinnati) and the West Virginia Divisions of Energy
and Natural Resources (WVDOE, WVDNR) to determine the efficacy of pollution
abatement requirements. They will also enable them to address the potential necessity
of additional regulations. The data gathered during each sampling event will be of
sufficient quality to address the central problem: Is there an effect of the combined
surface-mining activities on the biological integrity of streams adjacent to the mining
areas? More specific questions relating to the central problem can be developed which
may allow evaluation of causative factors. Specific questions to be addressed in this
RARE project are:
1. Are the mining activities effecting degradation of stream
habitat and/or water quality?
2. If so, is this degradation adversely affecting benthic
macroinvertebrate community vigor? and,
3. Likewise, is it adversely affecting fish community vigor?
-------
97
General Overview
Surface-mining involves several stages including clear-cutting of timber, grubbing
(term describing removal of stumps and debris), materials extraction, and reclamation.
Logging and blasting of bedrock began on August 01 as field teams were completing
samples at the final sites. During the clear-cutting and grubbing portion of this process,
much bedrock and all groundcover are removed and nearly all surface soils are
mechanically disrupted. This produces a situation ideal for erosive activity during rain
events and consequent heavy sediment loading into streams.
The study site is located in Mingo County near Wilsondale in western West Virginia
about one hour south of Huntington. The 14 sampling stations (Table 1) are distributed
through the east fork subwatershed of the Twelvepole Creek drainage system. Target
streams include Alex Branch, Old House Branch, Caney Fork, Pretty Branch, and East
Fork Twelvepole Creek; these range in size from first to third or fourth order. In general,
the streams are shallow and low gradient. Many of the first and second order streams
have a bedrock substrate; the larger streams (third or fourth order) have gravel and
cobble substrates. At the time of this pilot study, most of the sites had experienced
minimal habitat disruption. East Fork mainstem was an exception in that an asphalt road
parallels it on the north side providing runoff through a buffer zone reduction.
Although the majority of this paper deals with the DQOs and QC elements of the
pilot study, a brief overview of the 5-year phased project is presented (Table 2). If mining
activities proceed as expected, primary sampling events will be annual and will roughly
correspond to the stages of mine development. Thus, incremental monitoring of full
impact will be achieved throughout continued operations.
Response Variables and Specific Questions
For the identification of more specific questions it becomes necessary to define
"biological integrity". Biological integrity is functionally defined as "the condition of the
aquatic community inhabiting the unimpaired waterbodies of a specified habitat as
measured by community structure and function" (EPA 1990). Karr and Dudley (1981)
presented an earlier, more specific definition as "the ability of a habitat to support and
maintain a balanced, integrated, adaptive community of organisms having a species
composition, diversity, and functional organization comparable to that of natural habitat
of the region".
Variables potentially affecting resolution of the central problem can be divided into
abiotic and biotic considerations. Abiotic variables selected for evaluation are the habitat
assessment parameters included in the habitat assessment procedure of Rapid
Bioassessment Protocols (RBPs; Plafkin et al. 1989) as modified by Barbour and Stribling
-------
98
(1991, in press). These habitat assessment parameters deal with characteristics of
instream structure, canopy cover, channel morphology, riparian vegetative community,
and bank structure (Table 3).
Biotic variables are within the communities to be sampled for evaluation of these
sites: benthic macroinvertebrates and fish. A multimetric community analysis procedure
(RBPs, Plafkin et al. 1989) will be used for evaluation of these components of the biota.
Macroinvertebrate community health (as biological condition) is estimated by calculation
of metrics dealing with community composition, structure, and function (Table 4); fish
communities are evaluated by calculation of the Index of Biotic Integrity (Karr et al. 1986,
Plafkin et al. 1989) which also deals with community attributes (Table 5). With this
approach, biological condition is integrated with habitat quality for the overall assessment.
TABLE 1 SAMPLING STATIONS FOR THE U.S. EPA REGION III RARE PROJECT.
ALL ARE LOCATED IN THE EAST FORK OF TWELVEPOLE CREEK IN
MINGO COUNTY, WEST VIRGINIA
Station
T1
T2
T3
T4
T5
O6
C7
C8
C9
C10
C11
C12
A13
P14
Stream
Order
Twelvepole Creek 3-4
Twelvepole Creek 3-4
Twelvepole Creek 3-4
Twelvepole Creek 3-4
Twelvepole Creek 3-4
Old House Branch 1
Caney Fork 2
Caney Fork 2
Caney Fork 1
Caney Fork 1
Caney Fork 1
Caney Fork 1
Alex Branch 1
Pretty Branch 2
-------
TABLE 2 EPA REGION III RARE PROJECT OVERVIEW.
99
A. Central Question:
B. Specific Questions:
C. Phase
Pilot study
3
4
5
Effects of surface-mining activities on the aquatic community.
1. Effects on habitat quality.
2. Effects on biological condition of macroinvertebrate
assemblage.
3. Effects on biological condition of fish assemblage.
Site Condition
Minimal disturbance
100% deforestation;
post-grubbing
active mining completed
reclamation
2 years post-reclamation
Subquestions
1. appropriate response variables
2. appropriate methods / procedures
3. sampling stations / reference sites
4. natural variability of habitat and populations
1. appropriate analyses to differentiate among
stations
2. effects of habitat alteration on aquatic
community
3. adequate study design
1. effects on habitat quality
2. effects on macroinvertebrates
3. effects on fish
same questions as Phase 2
same questions as Phase 2
same questions as Phase 2
-------
100
TABLE 3 HABITAT ASSESSMENT PARAMETERS FOR RIFFLE/RUN PREVALENT SITUATIONS.
TAKEN FROM BARBOUR AND STRIBLING (1991, IN PRESS).
Primary -- Substrate, Instream Cover, and Canopy
1. Substrate Variety/I nstream Cover
2. Embeddedness
3. Flow or Velocity/Depth
4. Canopy Cover (Shading)
Secondary -- Channel Morphology
5. Channel Alteration
6. Bottom Scouring and Deposition
7. Pool/Riffle, Run/Bend Ratio
8. Lower Bank Channel Capacity
Tertiary - Riparian and Bank Structure
9. Upper Bank Stability
10. Bank Vegetative Stability (Grazing/Disruptive Pressure)
11. Streamside Cover
12. Riparian Vegetative Zone Width
-------
101
TABLE 4 BIOLOGICAL CONDITION METRICS TO BE CALCULATED ON
MACROINVERTEBRATE SURVEY DATA. FROM RBP III (PLAFKIN ET
AL [1989]).
Community Structure
1. Taxa Richness
2. EPT Index
3. Pinkham and Pearson
4. QSI-Taxa
Community Balance
5. HBI
6. % Dominant Taxon
7. DIC-5
8. Hydropsychidae/Trichoptera
Functional Feeding Group
9. Scrapers/(Scrapers + Filter Collectors)
10. Shredders/Total
11. QSI-Taxa
-------
102
Logic Statements
A logic statement is a characterization of the expected value or range of values to
be obtained for a given variable or composite of variables. Outliers from the established
expectations are considered deviations from the norm at which management decisions
are made. The entire QA/QC process is dedicated to the ability to make these decisions
with some level of confidence. The relationship between habitat quality (as percent of
reference) and biological condition (as percent of reference) can be established as
illustrated in Figure 2 to provide for an integrated assessment of composited variables.
The sigmoid curve represents a conceptual expectation of the relationship between
habitat quality and biological condition.
From this relationship, the ability to differentiate between water quality effects and
habitat alteration is dependent upon the integrity of the reference data. Essentially, three
major outcomes (scenarios) in this integrated assessment approach are possible (Figure
3):
Scenario A - If biological condition is rated as "non-impaired," and habitat quality
as "supporting" or "comparable," then biological integrity is indicated;
Scenario B - If biological condition is rated as "slightly impaired" or better, and
habitat quality as "supporting" or "comparable," then contaminant effects are
present;
Scenario C - If biological condition is rated as "slightly impaired" or lower, and
habitat quality as "partially supporting" or lower, then a combination of habitat
quality and water quality effects are present.
These scenarios are based on the assumption that acceptable and unacceptable
conditions have been ascertained through the establishment of a threshold value. Using
the values suggested by Plafkin et al. (1989), separate threshold values can be
determined for habitat quality and biological condition.
For habitat - If the habitat assessment score is less than 90 percent of the
reference site, then it is less than comparable.
For the benthic macroinvertebrates - If the total of the metric scores is less than
83 percent of reference conditions, then the site is impaired.
The pilot study will be used to evaluate the appropriate threshold values to
differentiate between nominal and subnominal conditions. The logic statement for the
integrated assessment will be developed to incorporate the composite of variables and
ultimately, the judgment of impairment from the mining activities.
-------
103
Acceptable Uncertainty
The more critical the consequences of ecological analysis for decision-making, the
narrower one wants confidence intervals. The capacity to increase confidence in
decisions is directly dependent on the available resources and on cost investment.
In this RARE project, the consequences of a false negative would be that stream
degradation could continue unabated and there would be no indication of a need for
further pollution abatement requirements as mining activities proceed. A false positive
might cause unnecessary regulations to be enacted, or, at additional costs for verification,
the study to be repeated because of weaknesses in the study design.
Uncertainty in ecological studies arises in several tiers, for purposes of this
discussion, roughly corresponding to five of the six major activity classes discussed
earlier: sampling design, field operations, laboratory activities, data analysis, and
reporting. Hart (1990) and Taylor (1988) point out that components of uncertainty from
various sources are additive. It seems that specification of acceptable levels of
uncertainty for rapid bioassessment protocols (benthos) must be developed from
consideration of specific components to reduce compounding error from an additivity
perspective. However, quantifiable confidence limits cannot be developed for some of
these activities in the absence of parallel studies. Funding levels usually cannot support
simultaneous and complete parallel studies. The level of uncertainty associated with
these must be qualitatively related to the implementation of QC elements for control of
specific potential error sources. A number of these elements are discussed below.
-------
104
TABLE 5 BIOLOGICAL CONDITION METRICS TO BE CALCULATED ON FISH
SURVEY DATA. FROM THE IBI (RBP V) (PLAFKIN ET AL [1989]).
1. Number native species.
2. Number darter or benthic species.
3. Number sunfish or pool species.
4. Number sucker or long-lived species.
5. Number intolerant species.
6. Proportion green sunfish or tolerant individuals.
7. Proportion omnivorous individuals.
8. Proportion insectivores.
9. Proportion top carnivores.
10. Total number individuals.
11. Proportion hybrids or exotics.
12. Proportion disease/anomalies.
-------
PwteMy - >,J Com
Supporting \'\ Supporting || parabto
- ?''
10
100
Habitat Quality (% of Reference)
Figure 2. The relationship between habitat and biological condition.
o
01
-------
Scenario
A
§
i
S
o
^Supporting = = parable
Scenario
B
Habitat Quality
Figure 3. Illustration of three potential results of RBP analyses.
o
o>
-------
107
One of those activities for which quantifiable confidence intervals can be calculated
is replicate sampling. Duplicate processing of these replicates provides information on
sorting efficiency, habitat and biological population/community variability, and suitability
of sampling equipment. It also provides for evaluation of the taxonomic treatment.
Relative percent difference (RPD), calculated as
_ (C1-C2)X100%
(Cl+C2)/2
where C1 = number of taxa in the larger sample and C2 = number of taxa in the smaller
sample, will be used to evaluate precision between duplicate samples. An RPD of 0.5 (50
percent) or greater will be considered unacceptable and corrective action taken. For
three or more replicates, relative standard deviation (RSD) will be used:
=
V
where s = standard deviation and y = mean of replicate analyses (EPA 1989).
Habitat assessments will be performed by more than one observer; each observer
will also perform duplicate assessments. Comparisons using RPD, RSD, and coefficients
of variability (CVs) will allow evaluation of both intra- and inter-observer variation.
QUALITY ASSURANCE PROGRAM
Quality Management
Quality management entails a number of activities including delineation of lines of
responsibility, outlining of the quality management plan, designation of a quality assurance
officer (QAO) and ensuring availability of standard operating procedures (SOPs). Thus
far, project organization and lines of responsibility have been illustrated for field
operations, laboratory activities, and data analysis (Figures 4-6). The EMSL-Cincinnati
Aquatic Biology Branch Chief will serve as QAO. All field and laboratory data sheets from
the pilot study have been assembled from various project personnel by EMSL-Cincinnati.
The project team consists of the following agencies or groups:
-------
108
A) U.S. Environmental Protection Agency
1) Environmental Monitoring Systems Laboratory - Cincinnati
(EMSL-Cincinnati)
2) Wheeling Region III Support Office
B) West Virginia
1) Division of Natural Resources (WVDNR)
2) Division of Energy (WVDOE)
C) Marshall University, Department of Biology
D) EA Engineering, Science, and Technology, Inc.
There are four components of field operations: benthic macroinvertebrates, fish,
habitat assessments, and water quality (Figure 4). Benthic macroinvertebrates (benthos)
are sampled by personnel of EMSL-Cincinnati, WVDOE, and EA Engineering, Science,
and Technology. Field leaders of the benthos team are from EMSL-Cincinnati. The fish
survey crew and leadership is provided by the WVDNR. Habitat assessment is performed
by EA and WVDOE. EPA Region III handles all water quality sampling. Field leaders
report to the principal investigators who are from EMSL-Cincinnati and EPA Region III.
-------
FIELD OPERATIONS
Principal Investigators
I
Benthos
EPA-EMSL
Field Leaders
WV-DNR
EA
WQ
EPA-Region III
Crew 1
Crew [|
Crew
EPA-EMSL WV-DNR WV-DOE
WV-DOE EA
EA
EF
Crew 1
JA-Regior
Figure 4. EPA Region III RARE project; field
operations organizational chart.
-------
LABORATORY ACTIVITIES
Principal Investigators I
Identification:
Macroinvertebrates
L
Identification:
Fish
Water Quality
replicates
Marshall
University
\
EA
wv-DNR
EPA Region III
Figure 5. EPA Region III RARE project; laboratory
activities organizational chart.
-------
DATA ANALYSIS
Principal Investigators
Rapid Bloassessment
Protocols
Hi V
(invertebrates)
(fish)
/~"~~~~"~~
'
EPA Region III
/
f
/— --— ~~~--|
WV-DNR ;
Water Quality
. r
1.
EPA Region III
Figure 6. EPA Region III RARE project; data
analysis organizational chart.
-------
112
Laboratory activities essentially are divided into three groups (Figure 5):
macroinvertebrate identifications, fish identifications, and water quality. Macroinvertebrate
taxonomy is being performed by Marshall University, Huntington, WV, EMSL-Cincinnati,
and EA. Fish identifications are being performed by WVDNR. Water samples are being
collected and shipped to the laboratory by EPA Region III. All data sheets are submitted
to EMSL-Cincinnati.
Data analysis is primarily by EPA Region III with assistance from EA (RBP III metric
calculation) and WVDNR (RBP V metric calculation) (Figure 6). Water quality data are
being interpreted by EPA Region III. Results are reported to EMSL-Cincinnati principal
investigator.
RARE Project Quality Control Elements
For each of the five ecological project activity classes (design, field, laboratory, data
analysis, and reporting) there are a number of potential sources of error. Barbour and
Thornley (1990) presented a series of matrices illustrating interactions between these
potential error sources and various QC elements which might be implemented to minimize
error. These matrices are reproduced in this document (Tables 6-10) with accompanying
tables indicating QC features specific to this RARE project.
In the left column of the upper part of the table are listed some of the potential
error sources; some of which are unique to each activity. The QC elements that will
control these potential errors are labelled at the top of the table. The "X" indicates
interaction between these two factors. When examining effectiveness of the QC elements,
those with the greatest number of interactions (X) are probably most significant.
Conversely, those potential sources of error without at least one QC interaction are likely
to have adverse influence and thus serious consequences on the performance of this
activity. In the lower part of each table are listed specific RARE activities related to these
QC elements.
Pilot studies are critical in design of sampling programs for controlling most
potential error (Table 6). Environmental strata have been taken into account in the
sampling design. The number of sampling sites (14) is manageable for the project
personnel, both in terms of fieldwork and data analysis. Sites are distributed among first
through third order streams, primarily around two areas designated to become strip
mines. Several sites in both areas are located so as to be upstream from mine runoff
influences. Historical data for area fisheries surveys in the area have been located and
procured for preliminary analysis. Replicate benthic samples are being taken with
different collecting gears to examine habitat and population variability as well as
equipment efficiency. Sampling equipment was chosen to target riffles, generally
recognized as the most productive habitat in freshwater streams. At several sites,
duplicate habitat assessments will be completed by different observers. Comparison of
-------
113
site rank-order should provide an estimate of consistency within methods. The pilot study
has demonstrated that some of the streams to be sampled have a complete bedrock
substrate with no gravel or cobble. For these cases, pools with accumulation of detritus
were sampled using a D-frame dip net.
For field operations, the most important QC element is that of training and
experience relative to project responsibility (Table 7). In the RARE project, all involved
personnel have a number of years of experience along with appropriate professional and
academic training. Calibration and maintenance of dissolved oxygen and pH meters
follow manufacturer specifications and SOPs. SOPs for benthic sampling are as
presented in the workplan prepared by EMSL-Cincinnati and in Plafkin et al. (1989).
Sample handling procedures including labelling, logging, and transportation, have been
established and are recorded in the EMSL-Cincinnati workplan.
For laboratory operations, training is also the most critical QC element for error
control, minimizing human error, and affecting virtually every potential error source (Table
8). Duplicate processing of samples is being performed for the benthos data. Primary
samples are sorted and identified by Marshall University; duplicates are processed by
EMSL-Cincinnati and/or EA. Taxonomists in these laboratories have from several to
many years of training and experience in invertebrate taxonomy. Data sheets from all
laboratories have been assembled, duplicated, and distributed to all members of the
project team.
In the data analysis phase, personnel training and standardization of the database
are the most critical QC elements (Table 9). Data handling and reporting and database
standardization are closely tied. All data are being computerized by WVDOE. A
professional biostatistician of EMSL-Cincinnati is providing statistical advice and
performing analyses. Calculation of rapid bioassessment biological condition metrics is
as in Plafkin et al. (1989) and Barbour et al. (1991, in press).
Reporting of the project is primarily supported by training and peer review (Table
10). Several personnel have experience and training in preparing manuscripts for
publication in peer reviewed journals. Results and conclusions will be presented at
professional meetings and published in peer review journals (such as the Journal of the
North American Benthological Society). Technical editing will be performed by in-house
personnel and journal editors and reviewers. Standard format will conform to journal and
EPA guidelines.
-------
TABLE 6 CONSIDERATIONS FOR DESIGN AND APPLICATION TO THE RARE PILOT STUDY
Design Considerations (Taken from Harbour and Thornley 1990)
Potential Error
Resources Available
Logistics
Response Variables
Ueather
Seasonality
Site Location
Habitat Variability
Population Variability
Equipment
Duality Control Elcmonts/RARE Implementation
PC Concepts
1. Pilot Study
2. Environmental Strata
3. Historical Data
4. Replicates
5. Equipment Choice
QC ELEMENTS
Pilot
Study
X
X
X
X
X
X
X
X
Environmental
Strata
Reg ton III RAKE
Historical
Data
Replicates
X
Equipment
Choice
la. Pilot study.
2a. Stream-order variability (let - 3rd).
2b. Site locations upstream and downstream of mine runoff.
2c. Annual samples on same date as previous year.
3a. Benthic macroinvertebrates. Marshall University.
3b. Fish. UVDNR.
4a. Benthic sampling--with available habitat, kicknets (2).
4b. Habitat assessment—multiple observers; true replicates.
5a. Square meter kicknet, double composite.
-------
TABLE 7 CONSIDERATIONS FOR FIEU) OPERATIONS AND APPLICATION TO THE RARE PILOT STUDY
Considerations for Field Operations (Taken from Barber and Thomley 1990)
QC ELEMENTS
Potential Error
Climate
Site Location
Sampling Equipment Efficiency
Human (Equipment Use)
Field Notes
Sample Processing
Sample Transportation
Sample Tracking
Sample
Maintenance
X
X
X
X
X
X
Effort
Evaluation
X
X
X
X
X
X
X
X
Additional
Equipment
X
X
X
X
X
X
Calibration
t Handling
X
X
X
X
Instrument
Training t Checks
X
Quality Control Ele*enta/RARE Implementation
QC Concepts
1. Instrument Calibration/
Maintenance
2. Crew Training/Evaluation
3. Field Equipment
4. Sample Handling
5. Additional Effort Checks
Region 111 RARE
la. SOPs, manufacturers specifications.
2a. All involved personnel have education and number of years' experience relating to project
responsibilities.
3a. SOPs for kicknet and Surber provided in uorkplan and Plafkin et al. (1989)
4a. SOPs for labelling, preserving, logging, and transportation provided in workplan.
SB. Check adherence to SOPs.
-------
TABLE 8 LABORATORY DESIGN CONSIDERATIONS AND APPLICATION TO THE RARE PILOT STUDY
Laboratory Considerations (Taken fron Bartoour and Thornley 1990)
Potential Error
Sampling Tracking
Improper Storage
Sample Preparation
Reference Error (Taxonomy)
Taxonomic Error (Human)
Counting Error
Sorting Efficiency
Data Records (Labs)
Data Records (Computer)
Quality Control Elewents/RARE Implementation
QC Concepts
QC ELEMENTS
Sort rig &
Verification
X
X
Duplicate
Taxonomy Process
X
Archives
X
X
Training
X
X
X
Data
Handling
X
1. Sorting and
Verification
2. Taxonomy
3. Duplicate
Processing
Region III RARE
la.
2a.
2b.
2c.
3a.
Experience.
Intralaboratory re-checking of sample residue for Missed specimens.
Interlaboratory re-checking of sample residue for Missed specimens.
Academic training.
Professional experience.
Investigation of unusual erroneous/records.
Interlaboratory sample processing; sorting, taxonomy.
TABLE 8 CONTINUED
QC Concepts
4. Archives
5. Training
6. Data Handling
Region 111 RARE
Aa. Sample logs--EHSL-C notebooks.
4b. Sample preserve!(on--ethanol.
4c. Taxonomy--investigate unusual/erroneous records.
5a. Personnel with academic and/or professional background relating to project responsibilities.
6a. SOPs for sample logging, with identification/serial numbers.
6b. Data transcript ton--checked by trained professionals. Data computerization--entry by trained technicians, output
reviewed by trained personnel; check with original datasheets.
O)
-------
TABLE 9 DATA ANALYSIS CONSIDERATIONS AND APPLICATION TO THE RARE PILOT STUDY
Data Analysis Considerations (Taken fro» Barbour and Thornley 1990)
QC ELEMENTS
Handling Standar
Potential Error
Inappropriate Statistics
Errors in Database
Database Management
Programming Errors
Misinterpret Analysis
Quality Control Eloents/RARE IspU
QC Concepts
1. Training
2. Handling & Reporting
3. Standardized Database
it. Standardized Analysis
5. Peer Review
6. Range Control
Handling
Training
X
X
X
X
X
Standardized
I Reporting
X
X
X
Standardized
Database
X
X
X
Peer
Analysis
X
X
Range
Review
X
X
Control
X
itation
Region HI RARE
2a.
2b.
3a.
4a.
4b.
5a.
6a.
All involved personnel with academic training and/or professional experience relating to data
analysis; RBP workshop participation.
Proofing—compare output with original datasheets, approval of transcription.
Documentation--datasheets completed/copied, complete set sent to four different locations.
Data entry/computerization into UVDOE sy steal.
RBP Metric calculation; SOPs. Plafkin et al. (1989) and Barbour et al. (1991; in press).
Statistics—Spearman's rank correlation coefficient, t-test; EMSL-C.
Professional biostatisticians, benthic and fisheries biologists.
Investigate data variability, outliers.
-------
TABLE 10 REPORTING CONSIDERATIONS AND APPLICATION TO THE RARE PILOT STUDY
Reporting Considerations (Taken fro* Harbour and Thornley 1990)
Potential Error
Transcription Error
Poor Presentation
Obscure Language
Did We Address Question?
duality Control Eleaents/RARE l«pl(
PC Concepts
1. Training
2. Peer Review
3. Technical Editor
4. Standard Format
ntation
la.
2a.
2b.
2c.
2d.
3a.
3b.
3c.
4a.
4b.
QC ELEMENTS
Training
X
X
X
X
Technical
Editor
X
X
Standard
format
X
X
Region 111 RARE
All involved personnel with training and/or experience in project reporting.
Address project organization.
Address presentation of data and discussion of results; merit (project personnel).
Objectives addressed? (project personnel).
Publication in peer reviewed journals.
Document--organization, appropriateness.
Basic wording.
Journal editors, anonymous reviewers.
U.S. EPA guidelines.
Journal format.
CO
-------
119
Remaining QC Considerations
Several factors remain to be addressed in conception and implementation of
appropriate QC elements. For example, contingencies must be developed to handle
variability of habitat characteristics. From the pilot study experience, it was determined
that a number of factors could potentially influence the effectiveness of sampling gear.
Among these are channel width, substrate quality, and flow/depth. At some sites,
alternative sampling methods were used. Appropriate statistical treatment of this potential
sampling gear variability should be determined. This focus would add to QC elements
of the design, field, laboratory, and analysis activity classes. Another consideration will
be data differences resulting from duplicate processing, particularly in the areas of
taxonomic identification levels and sorting efficiencies among laboratories. Verification of
sample processing and laboratory operations will be conducted by the QAO.
Perhaps the most important consideration to be resolved by the pilot study will be
selection of reference sites and development of a reference database necessary to assess
mining impacts throughout this proposed 5-year project. Historical data will be useful to
characterize past conditions and ecological potential of this region of West Virginia.
Reference sites corresponding to stream classification types (stream orders 1-4) will be
selected for comparisons to the ambient sites receiving mining activity influences.
REFERENCES
Barbour, M.T., and S. Thornley. 1990. Quality Assurance Issues Related to Field
Assessment of Aquatic Ecosystems, in Proceedings of the Third Annual Ecological
Quality Assurance Workshop, Dan Hart (ed.). Burlington, Ontario. April, pp. 167-183.
Barbour, M.T. and J.B. Stribling. 1991 (in press). Use of Habitat Assessment in
Evaluating the Biological Integrity of Stream Communities. Proceedings of U.S. EPA
Workshop on Biocriteria, Arlington, Virginia. 12-13 December 1990.
Barbour, M.T., J.L Plafkin, B.P. Bradley, C.G. Graves, and R.W. Wisseman. 1991 (in
press). Evaluation of EPA's Rapid Bioassessment Benthic Metrics: Metric Redundancy
and Variability Among Reference Streams Sites. Journal of Environmental Toxicology
and Chemistry.
Hart, D. (editor). 1990. Proceedings of the Third Annual Ecological Quality Assurance
Workshop. Burlington, Ontario. 214 pp.
Karr, J.R., and D.R. Dudley. 1981. Ecological Perspective On Water Quality Goals.
Environmental Management 5:55-68.
-------
120
Karr, J.R., K.D. Fausch, P.L Angermeier, P.R. Yant, and I.J. Schlosser. 1986. Assessing
Biological Integrity In Running Waters: A Method and Its Rationale. Special Publication
5. Illinois Natural History Survey, Champaign, Illinois.
Keith, LH. (editor). 1988. Principles of Environmental Sampling. ACS Professional
Reference Book, American Chemical Society. Washington, D.C. 458 pp.
Plafkin, J.L., M.T. Barbour, K.D. Porter, S.K. Gross, and R.M. Hughes. 1989. Rapid
Bioassessment Protocols for Use in Streams and Rivers: Benthic Macroinvertebrates
and Fish. EPA/444/4-89-001. U.S. EPA, Office of Water, Washington, D.C.
Smith, F., S. Kulkarni, LE. Myers, and M.J. Messner. 1988. Evaluating and Presenting
Quality Assurance Sampling Data, ]n Principles of Environmental Sampling, Chapter 10,
LH. Keith (ed.). American Chemical Society, Washington, D.C. Pages 157-168.
Taylor, J.K. 1988. Defining the Accuracy, Precision, and Confidence Limits of Sample
Data, in Principles of Environmental Sampling, Chapter 6, L.H. Keith (ed.). American
Chemical Society, Washington, D.C. Pages 101-107.
U.S. Environmental Protection Agency. 1989. Preparing Perfect Project Plans.
EPA/800/9-89/087. October. Office of Research and Development, Washington, D.C.
U.S. Environmental Protection Agency. 1990. Biological Criteria: National Program
Guidance for Surface Waters. EPA/440/5-90-004. April. Office of Water Regulations
and Standards, Washington, D.C.
-------
121
COMPARISONS OF LABORATORY AND FIELD
GOOD LABORATORY PRACTICES AND
QUALITY CONTROL VIEWPOINTS
A. F. Maciorowski
S. R. Brown
B. W. Cornaby
R. A. Mayer
Battelle Health and Environmental Group
Columbus, Ohio
ABSTRACT
As Good Laboratory Practices (GLPs) and Quality Assurance/Quality
Control (QA/QC) procedures are applied to field studies, especially large field
programs, similarities and differences between field and laboratory
environments must be understood. Further, this understanding must be
expressed in operational terms. Logistical similarities include the use of
written protocols for instrument calibration; sampling; sample shipment and
storage; sample analysis; and operation of a QA unit. Logical differences in
ambient biological, chemical, and physical endpoints include controlled age-
distribution vs. age-variable populations, controlled vs. uncontrolled doses
and environmental/substrate variables and easy-to-establish vs. difficult and
multiple cause/effect relationships. Such differences create intellectual
tensions that influence the acceptable bounds of data accuracy, precision,
completeness, comparability, acceptability, and reproducibility. We provide
lists of major GLP requirements for field studies. Situations where GLP and
QA/QC procedures can be applied as written, require modification and/or
are beyond investigative control are also discussed. A case history of a large
research program is also provided.
-------
OVERVIEW
Benefits
Comparisons
10 parameters
laboratory situations
field situations
differences/similarities
Rules of thumb from examples
parameter by parameter
problems and solutions
Working conclusions
IV)
-------
BENEFITS
Confirmation of common sense field experience
Logical treatment of laboratory and field
environments
Problem identification with solutions
Avoidance of some pitfalls
GJ
-------
TEN PARAMETERS IN COMPARISON
Study plan and data quality objectives
Study personnel and management
Test substance
Test system
Equipment
Facilities
Samples and specimens
Records and data control
Data analysis
Operation of QA unit
-------
STUDY PLAN AND DATA QUALITY OBJECTIVES
Location
Parameter S/D Laboratory Field
Accuracy S Similar because defined a priori, even though values may be
Completeness S ' r
Measurability S
Precision D Higher due to better control Lower due to lesser control
Comparability D
Representativenss D
»S = Similar, D = Different
IV)
CJI
-------
STUDY PERSONNEL AND MANAGEMENT
Parameter
Location
S/D
Laboratory
Field
Protective Clothing
Same strictness off measures regardless off location
Nature of Personnel
Use off existing qualified staff
Study Director Duties
Upper Management
Oversight
Generally, easier exercised
when entire study at one
location
Easier due to location
1. Supplemental cm-site
staff needed
2. Increased attention to
training
Fulfillment presents a
challenge due to multiple
sites
Harder because of distance
*S = Similar, D = Different
O)
-------
TEST SYSTEM
Location
Parameter
Laboratory
Field
Age
Origin
Population Size
Habitat
Security
Measurements
Observations
D Known
D Known
D Known exactly
D Artificial
D Standard measures
D Direct on all planned
specimens
D Planned and well controlled
Not as well known
Not as well known
Estimated
Natural with multiple
interacting variables
Additional measures due to
site vulnerability
Indirect estimate based on
sample
Planned but opportunistic
* S = Similar, D = Different
-------
EQUIPMENT
Parameter
Location
BID
Laboratory
Field
Standard Operating
Procedures
Transport
Calibration/
Standardization/
Cleaning
Maintenance/
Repair/
Backup
S
D
Should exist regardless of location
Within existing buildings
Normal schedules
Use of available resources
Better care due to
increased distances
More frequent intervals due
to moving
Better planning needed due
to lack of resources
*S = Similar, D = Different
-------
FACILITIES
Location
Parameter
S/D*
Laboratory
Field
General Nature
Separate Areas
Ambient Conditions,
e.g., ventilation, water)
Safety Apparatus
Maintenance and
Repair
D Indoor controlled
infrastructure
D Usually well defined
D Dictated and well controlled
D Hoods, eyewashes, other
already established
D Resources available
Limitations due to
temporary areas
Not as distinct due to space
limitations
Acceptance of natural
conditions
Needs to be planned for
and installed
Procuring of resources at
site
* S = Similar, D = Different
CO
-------
SAMPLES AND SPECIMENS
Parameter
S/D
Location
Laboratory
Field
Identification
Analytical Lab Storage
Collection
Preservation &
Storage When
Collected
Shipping
S
S
D
Similar regardless of collection location
Well controlled
Less controlled due to
weather conditions
Good control with refrigerators Less control due to use of
and freezers coolers and increased
logistics
Low concern if close to
sampling and analysis
locations
Stringent controls needed
due to distance between
collection and analysis
*S=Similar, D=Different
-------
RECORDS AND DATA CONTROL
Parameter
Standard Operating
Procedures
Protocols
Data
Archival
Protection
S/D*
s
s
s
s
D
Location
Laboratory Field
Existence and controls similar
Easier to control during study Increased attention due to
multiple transfers of
records
*S = Similar, D = Different
-------
DATA ANALYSIS
Parameter
Location
S/D
Laboratory
Field
Statistical
Extrapolations
- Tissue
- Organ
- Organism
- Population
- Community and
Ecosystem
Interpretations
S Similar regardless of data origin
S
S
S
S
D
All directly calculated from data gathered during study
Not generally appropriate Estimated based on data
gathered
Mechanistic Holistic
*S = Similar, D = Different
CO
IV)
-------
OPERATION OF QA UNIT
Location
Parameter
Independence
Master Schedule
Data and Report
S/D*
s
s
s
Laboratory
Field
Maintained regardless of location
Same regardless of location
Same methods regardless of data/report origin
Audits
Inspections
Inspector D
Communication With D
Study Team
Cooperator Oversight D
1. Coordination easier
2. Adaptable to schedule
changes
'nput available from QA Unit
Better due to closeness
Minimal; not a problem
1. Harder logistics
2. Not easily adaptable to
schedule changes
Independent individual
making quick decisions
Harder due to dispersed
personnel
Differing levels of
compliance
*S = Similar, D = Different
CO
CO
-------
EXAMPLES OF FIELD PROGRAMS
Large pond study for pesticide client
Several field sites for decontamination
Grain field study
Orchard field study
Other field programs
GO
-------
FINDINGS
FROM
FIELD PROGRAM EXPERIENCE:
RULES OF THUMB
Parameter
Problem
Solution
STUDY PLAN AND DATA
QUALITY
OBJECTIVES
Not an issue once the data quality
objectives
are set
Not an issue once the
data quality objectives
are set
01
-------
RULES OF THUMB
Parameter
Problem
Solution
STUDY PERSONNEL
AND
MANAGEMENT
Skill mix, motivation
Part-time and new staff
• Safety
Careful selection of personnel-
more ingenuity/tolerance
required
Intensive, on-site training before
study starts
1. Protective clothing
2. Qualified hoods
3. Packaging and storage
CO
O)
-------
RULES OF THUMB
Parameter
Problem
Solution
TEST SUBSTANCE
Geographical distance of
mixing and applications
Contamination
Transportation demands of
mixed materials
Adequate storage of test
substance
Retention of containers
Accidental releases
Careful planning and handling
1. Containment
2. SOPs for decontamination of
equipment and people
3. Establishment of clean/dirty
areas in field
1. Complete knowledge of
degradation properties
2. Careful logistics
3. Portable environmental control
1. Temperature and light control
2. Ventilation
Written waiver from EPA
1. Safety officer
2. Training
-------
RULES OF THUMB
Parameter
Problem
Solution
TEST SYSTEM
Lack of security, e.g.,
poaching, fishing
Untimely weather, e.g., flash
storm
Irregularity of application,
e.g., gullies, winds
Unexpected nearby events,
e.g., aerial spraying, wind
drift from roadside spraying
1. Posting of signs
2. Public announcements about
poisoning
Best informed decisions about
prevailing conditions, e.g.,
thresholds of wind speed
Regulation of dose, e.g., wind
shields
Avoidance of roads
CO
03
-------
RULES OF THUMB
Parameter
Problem
Solution
EQUIPMENT
• Breakdown, loss, malfunction, Backups are vital!
theft
• Lack of equipment at needed Check lists are essential!
time
Vulnerability of equipment to
hot sun and other weather
elements
Calibration relative to
movement of equipment
1. Use of shade to reduce
overheating
2. Back-ups are vital!
Calibration plan with on-site
verification
-------
RULES OF THUMB
Parameter
Problem
Solution
FACILITIES
Lack of facilities
Safety issues
• Pests, e.g., mice, gnats
• Inadequate space planning
Maintenance and repair
Development of same, e.g.,
trailers, mobiles
1. Experienced field foreman
2. Lease/upgrade space,
installation of hoods,
showers, and eye washes
3. Fire protection
Control with care and avoid
contaminating chemicals
Provision of space for:
• Test substance
• Samples
• Controls
• Equipment and supplies
Close cooperation with local
craftsmen
-------
RULES OF THUMB
Parameter
Problem
Solution
SAMPLES AND
SPECIMENS
• Lack of dry and wet ice
Time delays to move fresh
samples to freezers
Shipping confusion
Handling of large number of
samples in small space
Overall protection
Two or more suppliers, including
overnight access
Shade and ice
1. SOP development for
shipping
2. Assigned persons
Limiting factors defined and
worked out
1. Rigorous checklists and
follow-ups
2. Care in handling storage
containers
-------
RULES OF THUMB
Parameter
Problem
Solution
RECORDS AND DATA
CONTROL
• Storage space for records
Control of access
Regular transfers to central
collection in fire proof containers
Transfer to study director or
designate
Lack of designated custodian Specific assignment
-------
RULES OF THUMB
Parameter Problem Solution
DATA ANALYSIS • Generally not an issue Differences are addressed in
study plan
CO
-------
RULES OF THUMB
Parameter
Problem
Solution
OPERATION OF QA UNIT
Accessibility
1. Timing of field audits to
observe most events
2. Cooperation of field
operators
Varying degree of compliance On-site visit/evaluation/
among cooperators corrective actions per
cooperator's compliance
Unqualified inspector
Assignment of experienced field
inspectors
Nine inspectors and one worker 1. Better planning
2. Simplification of regulations
Overanxious auditors,
researchers, and other tense
situations
Service with a smile
-------
WORKING CONCLUSIONS
1. Complete a good program plan with appropriate SOPs
2. Use well-trained and experienced field staff
3. Assure all back-up equipment is ready for use
4. Use checklists for all activities
5. Respect fully the importance of the specimen/sample
6. Maintain a commitment to GLP from the entire team
7. Solve problems within above context
01
-------
146
MANAGEMENT AND RESULTS OF A QUALITY ASSURANCE PROGRAM
FORA
CANADA-UNITED STATES ATMOSPHERIC FIELD STUDY
Robert J. Vet
Atmospheric Environment Service
4905 Dufferin St.
Downsview, Ontario, Canada M3H 5T4
ABSTRACT
From June 1988 to June 1990, a cooperative Canada-US field study,
known as the Eulerian Model Evaluation Field Study (EMEFS), was carried
out in eastern North America. Its objective was to collect regional-scale air
and precipitation chemistry data for the evaluation of Canadian and US
atmospheric long range transport models. Two Canadian and three US
monitoring networks operated simultaneously using comparable, but not
identical, measurement methods. Considerable effort was expended
throughout the study on a quality assurance (QA) program focused on
accuracy, precision, comparability, completeness, and representativeness.
Work on the study's QA Program began during the design phase by
addressing data quality objectives (DQOs) and network comparability.
During the study itself, the QA Program was managed by a working group
responsible for network operations.
The culmination of the QA Program is to be a document known as
the Quality Assurance Synthesis Report, in which quantitative estimates of
accuracy, precision, comparability, completeness, and representativeness will
be summarized. The results to date indicate good comparability of the air
and precipitation chemistry data (somewhat better for precipitation than air),
that the accuracy of most measurements can be assured within the DQOs,
and that the precision of the measurements is excellent for most chemical
species (again, somewhat better for precipitation than for air). These results,
and a number of innovative estimators of precision and comparability, are
discussed in detail in the paper.
-------
147
INTRODUCTION
From June 1988 to May 1990, a cooperative Canada-United States field study,
known as the Eulerian Model Evaluation Field Study (EMEFS), was carried out across
eastern North America. The objective of the study was to collect regional-scale air and
precipitation chemistry data for the evaluation of Canadian and U.S. Eulerian long range
transport models [Hansen, 1989]. These models were designed to simulate the
complicated atmospheric processes of pollutant emission, transport, chemical conversion,
and deposition (wet and dry). They focus on the oxides of sulphur and nitrogen with a
view to determining the relationships between various pollutant emission sources and
receptors in eastern North America. The Eulerian Model Evaluation Field Study was
designed to obtain sulphur and nitrogen species concentrations in air and precipitation
for comparison against model predictions. It is the quality assurance of these
measurements that is discussed in this paper. Complete details of the EMEFS and its
field measurement program are given in Hansen [1989].
EMEFS ORGANIZATION AND MANAGEMENT
Data required for the model evaluation exercise were obtained from five air and
precipitation monitoring networks operated simultaneously across eastern North America.
Of the five networks, two were Canadian and three were American (see Table 1). The
networks operated with comparable, but not identical, measurement methods.
Representatives of the five agencies sponsoring the networks formed a Project
Management Group (PMG) responsible for coordinating all activities within the study.
Direct operational responsibilities were delegated to four teams - one for estimating
pollutant source emission rates, one for carrying out special aircraft and ground-based
atmospheric studies, one for the model evaluation activities, and the last for making the
network measurements. This paper focuses on the QA/QC activities of the network
measurement team (known as the Operational Measurements Team or OMT).
MEASUREMENT METHODS
The five monitoring networks operated in eastern North America at the sites shown
in Figure 1 (from Hansen, 1989). Note that some of the sites shown did not operate
continuously throughout the two year period.
The measurement methods used by the five networks were similar in approach but
varied somewhat in detail. Standardization and commonality of the measurement
methods was addressed in several workshops held before the study began. Wherever
possible, field, laboratory, QA/QC and data management methods were standardized.
-------
148
TABLE 1 CANADIAN AND US NETWORKS OPERATING DURING EMEFS
NETWORK
ACRONYM
US Networks
OEN
Acid MODES
FADMP
NETWORK NAME
Operational Evaluation Network
Acid Model Operational
Diagnostic Evaluation Study
Florida Acid Deposition
Management Program Network
SPONSORING AGENCY
Electric Power Research Institute (EPRI)
US Environmental Protection Agency
Florida Electrical Utilities
The Acid Deposition in Ontario
Study Network
The Canadian Air and
Precipitation Monitoring Network
Ontario Ministry of the Environment
Atmospheric Environment Service/
Environment Canada
The five networks all made three types of measurements (certain networks also
made other measurements):
1. Precipitation Chemistry
2. Air Chemistry
3.
Ozone
using precipitation chemistry collectors
and standard precipitation gauges;
using multi-stage filter packs;
using continuous monitors.
The precipitation and air chemistry measurements were made over 24 hour
periods, the ozone measurements were made continuously. The emphasis of the
precipitation and air chemistry measurements was on species of sulfur and nitrogen,
however, other analytes were also measured.
-------
149
The full suite of precipitation measurements included SO4, NO3, CI", pH, NH4, Na+, K+,
Ca++ and Mg++ and air measurements included paniculate SO4, NO3, NH4 and gaseous
SO2 and HNO3 (and, in some networks, NH3). The methods employed were roughly
similar in all networks in that they used commercial precipitation chemistry collectors (of
three different types) and, for air, 3 or 4-stage filter packs mounted on 10 m towers (3
different kinds of filter packs and four different kinds of flow systems were used). Four
different air and precipitation chemistry laboratories analyzed the air and precipitation
samples collected at the field sites. Further details of the measurement methods can be
found in Hansen (1989).
THE EMEFS QUALITY ASSURANCE PROGRAM AND ACTIVITIES
To assure the collection of a high quality data base, the OMT designed an
extensive quality assurance program for the network measurements. The implementation
of the quality assurance program began during the study design phase when a number
of workshops were held to determine the types of measurements to be made, the
appropriate measurement methods, the degree of commonality between the various
networks, and the quality assurance procedures needed to ensure accurate, precise,
comparable, representative and complete data. The outcome of the workshops was a
common set of within- and between-network QA/QC procedures. A summary of these
procedures in the precipitation and air chemistry field monitoring programs is given in
Table 2 [McNair and Allan, 1991].
Aside from the within-network procedures of:
o Measuring field blanks,
o Carrying out regular site inspections/audits, and
o Quality controlling the laboratory operating system,
the most notable QA/QC procedures were those addressing within-network precision and
between-network comparability. To address the former, four of the five networks
operated several sites where duplicate sets of sampling instrumentation were run
simultaneously (these sites are referred to hereafter as 'duplicate sites'). The air and
precipitation chemistry results from the duplicate instruments were used to determine
within-network precision. Between-network comparability was addressed by colocating
the instrumentation of several networks at the same sites (these sites are hereafter
referred to as 'colocated sites'). Two colocated sites were particularly important. One
was located at the Pennsylvania State University, where four of the five networks operated
a duplicate set of sampling instruments (the purpose of which was to determine precision
and comparability simultaneously). The other site was located at Egbert, Ontario where
the same four networks colocated one set of instruments each. At selected other sites,
two or more networks also colocated instruments.
-------
150
QUALITY ASSURANCE PROGRAM RESULTS
The eventual goal of the OMT's quality assurance program is to document the
quality of the multi-network data base. To do this, the QA Program results will be
synthesized and documented in a report known as the Quality Assurance Synthesis
Report. This report will be used by the model evaluators and other data users to
understand the uncertainties in the measurement data. Since the report is highly oriented
toward data users, it may serve as a useful model for other QA Programs, and, for this
reason, the contents are given here in Table 3.
There are several important notes to be made about the QA Synthesis Report:
1. The quality assurance information is to be presented in the context of the five
standard data quality attributes, namely, accuracy, precision, comparability,
representativeness and completeness.
2. The information on the five data quality attributes will be combined (if possible) into
a single measure of data uncertainty - ideally an error bar.
3. An extensive discussion will be presented on the suitability of the measurement
methods used by the OMT.
-------
151
Figure 1. EMEFS monitoring site locations. ME-35, GRAD, and VAR sites
constitute the Acid MODES network.
-------
FIELD
OA/OC PROGRAMS
PRECIPITATION & FILTER PACK PROGRAMS
Container
Procurement
Controls
Container
Inventory
Controls
Container
Preparation
Site Inspection
Frequency
Field Blanks
>uplicai* saaniing
laua utwork
comparison site*
Filter
Procurement
Controls
Filter
Inventory
Controls
Filter
Preparation
Filter Pack
joadiegAJeloadini
Field Blanks
Flow Audits
>iplicaae samplin
Comparison site*
APIOS
CAPMoN
FADMP
M£-35
OEN
PRECIPITATION CHEMISTRY
ag blanks tested
or presence of
nalytes
ag batches shipped
to all regional offices
imultaneously
Semi-tnnually
Dry bag at least
monthly
6 of 11
9 of 11
i Ol 1 •
Blanks tested
for presence of
analytea
Single supplier
Staged shipments
Batches tracked
Impregnation
by ENSR
Regional Offices
Passive filler pack
Weekly
Quarterly
2 of 10 Site*
2f%f \(\ Cat**.
Ol l\J OllEV
Bag blanks tested
for presence of
•nalytes
Bucket blanks tested
for presence of
snalytes
Bag batches shipped Bucket batches shipped
lo all site*
simultaneously
-
Quarterly
Dry bag
Weekly
4 of 17 site*
ft nl \ 7 titefl
o 01 i / siiea
Blank* tested
for presence of
analytea
Single supplier
Staged shipments
Bate he* tracked
Impregnation
by ENSR
Central facility
Passive filler pack
Weekly
Quarterly
1 of 12 Sites
*> f\f I")
-------
153
TABLE 3. CONTENTS OF THE QA SYNTHESIS REPORT
Overview of the Measurement Program
Measurement Methods
QA/QC Methods
Quality Assurance/Quality Control Results
Accuracy
Precision
Comparability
Representativeness
Completeness
Suitability of Measurement Methods
Demonstrated Performance Characteristics of Methods During
EMEFS
Comparison of Methods Against Other Methods
Estimation of the 'Overall Uncertainty' from the QA/QC Information.
Error Bars
Although the QA Synthesis Report has not yet been written, some preliminary
quantitative results are available. A flavour for these results can be obtained from the
discussion below.
ACCURACY
Of the measurements made during the study, flow rate and ozone were the only
ones for which absolute accuracy could be determined directly. This is because 'transfer
standard' flow meters and ozone monitors traceable to primary standards were available
for auditing the field instrumentation. An idea of the achievable accuracy of the filter pack
flow measurement system is given in Figure 2. Shown are the results of a flow
measurement audit carried out on the Acid MODES filter pack sampling system at the
Egbert, Ontario site [Bowen and Dowler, 1990]. The results indicate an absolute
accuracy of -0.53 L/min (relative accuracy = -2.6% of the audit flow rate). Three such
sets of independent audits were done at the colocated sites.
-------
FLOW AUDIT RESULTS
EGBERT, CANADA (AMP)
AUDIT DATE: APRIL 18,1990
AUDITORS: BOWEN. DOWLER
SAMPLERS
TYPE
ID*
FILTER
PACK
FLOW
CONTROLER
SETTING
3.33
FLOW
CONTHOLEH
READING
3.33
LEAK CHECK •••
CONT. REACMNQ
AUDIT/SITE
0.11/0.11
TEMP.
DEO. C
9.86
BAHO.
PRESS.
MM Hfl
748.5
AUDIT FLOW
L/MIN.
20.53
SITE FLOW
L/MIN.
20.00
FLOW
OFF.
L/MIN.
-0.53
PERCENT
DIFF
S-A/A * 100
-2.6
COMMENTS: FLOWS AT STP ( 0 DEO. C ; 760 MM Hg )
••• LEAK CHECK: AUDIT VALUE IS CONTROLLER HEADING WITH FLOW LINE PLUGGED (PUMP HUNNNG).
SITE VALUE IS CONTROLLER HEADING OBTAINED FROM OPERATORS LEAK CHECK.
AUDIT VALUE AND SITE VALUE SHOULD BE THE SAME f THERE ARE NO LEAKS.
Figure 2. US EPA audit results of flow rates on the Acid NODES filter pack
sampling system at Egbert, Ontario.
-------
155
For the other measurements, i.e., filter pack loadings and precipitation composition,
no primary standards existed - making it therefore impossible to determine the absolute
accuracy of the measurements. As the next best alternative, measurements were made
of a number of individual sources of error known to have a potential influence on the
overall accuracy. The concept behind these measurements was to determine
quantitatively that the sources of error were relatively small, and therefore had little
influence on the overall accuracy of the measurements. The measurements included
taking field blanks of the filter packs and precipitation collection vessels and analyzing
reagent blanks in the laboratory.
PRECISION
The within-network precision of each network's air and precipitation measurement
systems was determined by operating duplicate sampling instruments at several sites.
The duplicate data made it possible to calculate the precision of each network's
measurement systems. Unfortunately, final data are not yet available, but certain insights
regarding measurement precision are available. Two of the main points are:
1. The duplicate air and precipitation data are characterized by a highly non-normal
frequency distribution. This causes problems with the selection of a representative
measure of precision, e.g., the standard deviation of duplicate measurements is
not a good measure of precision when the underlying distribution is not normal.
Even though only preliminary data are available, we do have some insight into the
shape of the underlying distribution and have proposed a suitable measure of
precision. To describe this, we begin by defining the 'error' between duplicate
measurements at the same site [Vet and Sirois, 1987], i.e.,
ERROR = Cj = [1/ 2] [C, - C2], Eq. 1
where C, and C2 represent the concentrations measured in samplers
1 and 2, respectively at the same site,
1/2 accounts for the variance of both measurements, and
i represents the precipitation event number.
Unfortunately, the distribution of this error term is highly non-normal, a fact easily
seen in Figure 3 [Sirois, 1991]. Shown in the solid line is the shape of the error
term distribution for duplicate precipitation chemistry measurements. These
particular measurements were made in the CAPMoN network over a five- year
period - considerably longer than the two-year EMEFS program. However, the
distribution is still representative of the EMEFS results.
-------
156
It is clear from Figure 3 that roughly 90 percent of the errors are near zero. The
remaining 10% are relatively large and located at the extreme tails of the
distribution. Two sources of variance are responsible for this error distribution -
field variance and laboratory variance. While field variance is very difficult to
measure, laboratory variance is not, and is shown by the dashed line in Figure 3.
It is apparent from this figure that the 'overall precision' and the 'laboratory
precision' are quite similar in magnitude except at the extreme tails. There, the
large errors appear to be due to field-induced variance alone.
2. As mentioned above, the highly non-Gaussian nature of the error distribution
precludes the use of traditional measures of precision. As a result, the OMT has
adopted a non-parametric measure of precision called the Modified Median
Absolute Difference or M.MAD [Vet and Sirois, 1987]. The M.MAD is simply a
non-parametric (i.e., independent of the underlying distribution) estimator of scale
adapted from Randies and Wolf [1979]. It is defined as follows:
It is defined as follows:
M.MAD=—-—medianl Aq-M), . . ., | ACn-Af|] Eg. 2
where the 0.6745 term makes the numerator a consistent estimator of the
standard deviation of the errors when the underlying distribution is
normal,
AC term represents the individual errors of the duplicate data from samples
1 to n (as defined in Eq. 1 above)
and M = the median of the AC terms.
Since the initial value of AC normally equals zero (i.e., no biases exist between duplicate
collectors), Eq. 2 reduces to:
median[— |q-C|
6745
-------
157
Note that, in this form, the M.MAD approximately equals the simple Median
Absolute Difference (MAD) between collectors.
For the Quality Assurance Synthesis Report, all air and precipitation precision
values will be reported in terms of the M.MAD. Also reported will be a
non-parametric coefficient of variation equal to the M.MAD divided by the median
air or precipitation concentration.
-------
8
-8
S04
NA = 604
NT = 2056
SOURCE OF ERROR
J l_J—L
J L
ANALYTICAL
J—i—i—i—'
TOTAL
J L
158
.01 .1 .5 2 10 30 50 70 90 98 99.9
Standard Normal Probability Distribution(%)
Figure 3. Normal probability distributions of: (a) total error (from Eq. 1)
obtained from duplicate site concentrations of sulphate in
precipitation (solid line) and (b) analytical error (from Eq. 1)
obtained from between-run laboratory duplicate analyses of sulphate
(dashed line). Data taken from EMEFS and pre-EMEFS sampling
periods.
-------
159
COMPARABILITY
Comparability was determined through the following activities:
o Colocation of the networks at specific sites,
o Precipitation laboratory intercomparison studies,
o Air filter pack laboratory intercomparison studies,
o Use of a single filter supplier to all five networks, and
o Analysis of a single low concentration standard by all
laboratories to determine a consistent measure of analytical
detection limit.
A brief description of the methods and preliminary results is given below.
Colocation: Multi-network colocation of precipitation and filter pack instrumentation took
place at Penn State University, Pennsylvania (in duplicate) and Egbert, Ontario
(singularly). To date, the colocated network data have not be analyzed but will eventually
be tested to (1) determine whether statistically significant biases occurred between the
different networks and (2) quantify the magnitude of such biases. It is expected that
these intercomparison results will be very important for ensuring that the various networks
were indeed comparable.
Precipitation Chemistry Laboratory Intercomparison Study: The National Water
Research Institute of Environment Canada was contracted for the Eulerian Model
Evaluation Field Study to operate a precipitation chemistry laboratory intercomparison
study. Once per month, ten water samples (covering a concentration range similar to that
of real precipitation samples) were sent to each network's precipitation laboratory. The
results were analyzed for bias and precision using the non-parametric Youden technique
[Aspila, 1989]. Reports were sent to the individual laboratories within two to three months
to ensure that corrective action could be taken at any out-of-control laboratories. In fact,
none of the laboratories experienced out-of-control situations, although all experienced
at least one blunder. Currently, the data from the 27 monthly intercomparison studies are
being analyzed to quantify between-laboratory biases. Preliminary results, shown in
Figure 4 for SO4, indicate very small between-network biases (typically <0.1 mg/L)
compared to typical sample concentrations [Aspila, 1991]. Note that the magnitude of
the biases was determined using the 'median polish technique' [Hoaglin et al., 1983].
-------
160
Air Filter Laboratory Intel-comparison Studies: One of the participating members of
the OMT was contracted to carry out a filter pack laboratory intercomparison study for
the EMEFS project. This study was considerably more difficult than the aforementioned
precipitation laboratory intercomparison study because of major difficulties in producing
filters of comparable loadings for each of the pollutant species being measured. Despite
the difficulties, the Operational Measurement Team designed a study in which three sets
of comparably-loaded filters were distributed to the participating air filter laboratories.
The studies were designed with the following criteria in mind:
o The intercomparison filters were to be the same as the filters used
in the networks;
o The filter loadings were to be representative of the range of
loadings obtained throughout the EMEFS measurement
program;
o The loadings on the filters were to be obtained from ambient
air sampling rather than from solution spikes;
o The pollutant species to be measured in the intercomparison were
to be the same as those measured in the routine network operations.
The study design was as follows: of the three intercomparison studies carried out,
one was in the middle of the EMEFS project, one at the two thirds point, and one at the
end. In each study, 18 sets of filters were distributed to each of the four labs - the 18
being nine sets of duplicate filters. All but one set of filters, which was a set of unused
blanks, were collected under ambient sampling conditions by placing eight filter packs on
the same sampling manifold at the same time and carefully controlling the flow rate to
each. This was done over eight different sampling periods in order to collect the requisite
number of ambient samples. Once collected, two filter packs from each sampling period
were randomly selected for distribution to individual laboratories. Each lab then received
eight sets of duplicate ambient filters and one set of duplicate blank filters. In addition,
before each study, a pilot study was undertaken in which all eight filter packs sampled on
three different occasions and were analyzed by one laboratory only. The resultant data
were used to determine whether the within-network precision (affected by collection and
analysis sources of variance) was good enough to allow us to distinguish
between-network biases. It was concluded that this was indeed the case and the studies
went forward.
The data from the three studies are currently being analyzed and no quantitative results
are available to date. The method, however, appeared to work and holds considerable
promise for wider applications in the future.
-------
161
Sole Supplier of Filters: Throughout the two-year sampling period, one supplier of filters
was used by all five networks. The supplier obtained the necessary filters from the
manufacturers, quality controlled the blank levels, prepared the filters that required
chemical impregnation, and distributed them to the five networks. The use of the single
supplier of filters assured, at minimum, a comparable starting point for the filter sampling
program.
Consistent Measure of Detection Limit: Unfortunately, the various laboratories carrying
out air and precipitation analyses used a number of different methods to determine their
analytical detection limits. Since most of these laboratories were unwilling or unable to
modify these methods, the OMT designed a method of quantifying comparable detection
limits across all labs. To do this, the OMT selected the International Union of Pure and
Applied Chemistry (IUPAC) definition of detection limit, namely,
"the limit of detection expressed as a concentration of CL, or quantity, QL, is
derived from the smallest measure, XL, that can be accepted with confidence
as genuine and is not suspected to be only an accidentally high value of the
blank measure" [IUPAC, 1978].
Here, the value for XL is given by:
XL = Xb, + kSb|
where Xbl and Sbl are the mean and standard deviation, respectively, of the blank
measure, and k is numerical factor based on the desired confidence interval -
chosen as 3 in this case. The values for Xbl and Sb, are measured experimentally
by making 20 or more measurements of a blank solution [Tropp et al., 1991].
For the EM EPS program, the blank measures were determined from surrogate
anion and cation samples produced and distributed to all the laboratories by the National
Water Research Institute of Environment Canada. These solutions contained less than
0.1 mg/l per analyte and were measured daily (or as frequently as possible). Two sets
of the solutions were used over the EMEFS study period. All EMEFS network laboratories
were required to analyze the solutions on an ongoing basis and submit the data for
independent detection limit determination.
-------
0.10-
0.06-
0.02-
-0.02-
-0.06-
-0.10
162
I UVOIB
001 oca 10. on 001 oos OM ao. oo? on in on a. 010 on 013 on on a. on 017 on M. on tao
O.iO-i
0.06-
0.02-
-0.02-
-0.06^
-0.10-
1
1
r
1 r
p
' ' ' 1
i i
1 ' ' 1 ' ' ' "Woo*
0.10-
0.06-
0.02-
-0.02-
-0.06-
-0.10
IB. on on i*. on an am n. ear on an. on XL ou on ou 013 on ois a. on 01? on ML
0.10-]
0.06-
0.02-
-0.02-
-0.06-
-0.10-
|
.(111. , ..IN, 1
III ' '
1
i«.ooiooai«.oooiHooioo«anLoon»Ju.
oia on oia 013 oit ois a. on 017 m» M. on
1988 1989 1990
Study'* Start Dot* (Rpproxlmat*)
Figure 4. Results of the 27 EMEFS precipitation chemistry laboratory
intercomparison studies for SOT. Shown on the x-axis are the study
numbers and approximate dates. Shown on the y-axis is the magnitude
of the between-network bias in mg/L. Arrows indicate values
exceeding the maximum or miniumum scale. The top 4 graphs represent
the participating laboratories, the bottom graph represents the
average of the 4 laboratories. All biases were estimated using the
median polish technique.
-------
163
Preliminary results are published in Tropp et al. [1991] and are summarized for SO4
and NO3 in Table 4. It is clear from Table 4 that the detection limits of most of the
laboratories were comparable, however, two of the laboratories had detection limits
roughly an order of magnitude higher than the others. It remains for the OMT to decide
how to handle this dichotomy in the multi-network data base. It is worth noting that for
SO4 (the primary analyte of interest to this study), the measured detection limits of two
of the six laboratories were lower than the nominal values stated by the laboratories
(using their own definitions), two were higher and two were roughly equivalent.
COMPLETENESS
Data completeness was recognized early in the study as a major factor affecting the
usefulness and accuracy of the EM EPS air and precipitation chemistry measurements
(see, for example, Sirois, 1990). To minimize missing data errors, the five networks
agreed to a 90 percent Data Quality Objective for completeness. They then instituted
effective and timely corrective action programs to meet this DQO.
For the filter pack measurements in particular, data completeness proved to be an
even more important factor because of the 24-hour-integrated nature of the filter pack
sampling, i.e., if sampling time was lost during a 24 hour sampling interval, then the
integrated filter pack loadings would be unrepresentative of the 24 hour period. To
accommodate this, the networks validated only those data from filters that sampled for
more than 75 percent of the sampling interval.
REPRESENTATIVENESS
In the context of this study, the representativeness of greatest concern to the model
evaluators was the so-called 'regional representativeness' of the monitoring sites.
Unfortunately, regional representativeness is not an easily-measured attribute so the
networks adopted the following subjective and qualitative methods for handling
representativeness:
1. Measurement sites were initially located using accepted siting criteria for
regionally-representative sites, and
2. Networks adopted the site rating scheme of Olsen et al. [1990] whereby each
site was rated subjectively for its relative regional representativeness. Table
5 summarizes the rating scheme. In the EM EPS data transmittal documents,
these site ratings will be published for each site so that model evaluators and
other data users will have some opportunity to judge the relative
representativeness of the sites they use.
-------
164
SUMMARY
From its inception, the Eulerian Model Evaluation Field Study was carefully designed
to produce a high quality model evaluation data set. To assure such quality, specific
procedures were used by the various networks, including: the adoption of standard
measurement methods to the extent possible by the five participating networks, the
implementation of similar within-network QA/QC procedures, and the operation of an
extensive set of between-network colocated site intercomparisons. The results of these
QA/QC activities will be documented in a Quality Assurance Synthesis Report that
focusses on the accuracy, precision, completeness, comparability and representativeness
of the data. Preliminary results suggest that a quality multi-network data base will be
available for use by the model evaluators.
TABLE 4. PRELIMINARY RESULTS OF THE STANDARDIZED DETECTION LIMIT STUDY
S04 (mOg/L)
Laboratory
12 3 4 56
1 0.124 0.030 0.034 N/A 0.031 0.030
2 0.868 0.058 0.056 1.423 0.022 0.022
N03 (mg N/L)
Laboratory
12 3 4 56
Set
1 0.252 0.011 0.045 0.094 0.008 0.017
2 0.104 0.065 0.026 0.052 0.008 0.010
-------
165
VII. REFERENCES
Aspila K.I. (1989) A Manual for Effective Interlaboratory Quality Assurance.
National Water Research Institute Report 89-99, Canada Centre for Inland Waters,
867 Lakeshore Rd., Burlington, Ont.
Aspila K.I. (1991) Personal Communication.
Bowen J.A. and Dowler O.L. (1991) Performance and Systems Audits on EMEFS Field
Sites in Egbert, Ontario, Canada and State College, PA. US EPA Internal Report,
Quality Assurance Division, Research Triangle Park, NC.
Hansen D.A. (1989) Project Plan for the Acid Deposition Eulerian Model
Evaluation and Field Study. Electric Power Research Institute, Palo Alto, CA.
Hoaglin D.C., Mosteller F. and Tukey J.W. [Eds.] (1983) Understanding Robust
and Exploratory Data Analysis. John Wiley and Sons, New York.
IUPAC, International Union of Pure and Applied Chemists (1979) Compendium of
Analytical Nomenclature. Pergamon Press, New York.
McNair C.S. and Allan M.A. (1991). Personal Communication.
Olsen A.R., Voldner E.G., Bigelow D.S., Chan W.H., Clark T.L, Lusis M.A.,
Misra P.K., and Vet R.J. (1990) Unified Wet Deposition Data Summaries for North
America: Data Summary Procedures and Results for 1980-1986. Atmos. Envir.
24A:3, 661-672.
Randies R.H. and Wolfe D.A. (1979) Introduction to the Theory of Non-Parametric
Statistics. John Wiley and Sons, New York.
Sirois A. (1990) The Effects of Missing Data on the Calculation of
Precipitation-Weighted-Mean Concentrations in Wet Deposition. Atmos. Envir.
24A:9, 2277-2288.
Sirois A. (1991) Personal Communication.
Tropp R., Tulis D. and Ringler E. (1991) OMT Report on Analytical Detection
Limits for EMEFS Laboratories. ABB Environmental Services Inc., Chapel Hill, NC.
Vet R.J. and Sirois A. (1987) The Precision of Precipitation Chemistry
Measurements in the Canadian Air and Precipitation Monitoring Network (CAPMoN).
Proceedings of the 80th Annual Meeting of the Air Pollution Control Association,
June 21-26, 1987, New York, New York.
-------
166
TABLE 5. REGIONAL SITE REPRESENTATIVENESS RATING SCHEME [Olsen et al, 1990]
LEVEL RATING DESCRIPTION
1 Regionally Representative Site free of non-regional influences or
contamination.
2a Potentially Regionally Local Sources (i.e., within 40 km) of
Representative interference exist but are judged to have
little effect.
2b Potentially Regionally Local sources of interference exist with
Unrepresentative potential loss of regional
representativeness.
3 Regionally Unrepresentative Local sources of interference exist and
are known or strongly suspected to have
significant effects.
-------
167
FOURTH ANNUAL ECOLOGICAL QUALITY ASSURANCE WORKSHOP
POSTER SESSION
-------
168
ECOLOGICAL SURVEY OF LAND AND WATER IN BRITAIN
John C. Peters
United Kingdom Department of Environment
Rural Affairs Directorate
Bristol, England
ABSTRACT
The paper describes the amalgamation of methods, costs and reasons
for a survey of the British countryside during 1990.
Changes in agricultural activity fuelled by changes in support mechanisms
has resulted in a loss of wildlife habitat and decrease in the varied landscape
of Britain. The UK Department of the Environment together with its agencies
and the Natural Environment Research Council have embarked upon a
survey of land cover features utilising satellite remote sensing, low level air
photography and detailed ground survey of a sample of 1 km squares to
obtain a database. These data will not only provide a platform for
comparison with future surveys but also enable comparisons to be made with
past work of a less complete nature but which examined some sectors
covered in the 1990 Survey.
The paper refers to important past work indicates the efforts made to
ensure quality of survey and lists the features and species recorded.
INTRODUCTION
Until Sir Dudley Stamp set out to obtain a Land Use statement in England and
Wales, in the mid-1930's there had been no comprehensive statement of these areas
since the Doomsday Book of 1066. Stamp's land use maps and statistics have been
augmented by the mapping activities of the UK Ordnance Survey and by a survey in
England and Wales in the mid-1960s "The Land Utilisation Survey" of Dr Alice Coleman
utilising students as surveyors. None of these surveys covered Scotland, but this is being
rectified by the analysis of complete aerial cover obtained in 1989 and 1990. In addition
to these country surveys there have been various sectoral surveys of land use such as
those by the UK Ministry of Agriculture, Fisheries and Food (MAFF), Forestry Commission
(FC) and Nature Conservancy Council (NCC). It is these sectoral surveys which have
provided the main basis of national statistics of land use and land use change. Their
strength lies in systematic recording, but their limitation in the difficulties of combining
data.
-------
169
In response to economic, social and environmental pressures land cover and the
use to which land is put has changed dramatically since the Second World War.
Agricultural Statistics reveal complicated shifts in the balance of areas of land under
different crops. Forestry statistics show an expansion of conifers in the uplands and
changes in the management of broadleave woodlands in the lowlands. Small changes
in management, e.g., grazing intensity, time of cropping and fertiliser use are not
discernible from generalised land use statistics. Ecological information necessary to
identify and quantify the consequences for wildlife of such changes in land use are
lacking, however. Three major but unconnected pieces of work took place in the 1970s
and early 80s and it is the tying together of these into a major survey of Great Britain (GB)
in 1990 that this paper describes.
AMALGAMATION OF THREE SEPARATE STUDIES
The Institute of Terrestrial Ecology Land Classification
In the 1970s the Natural Environment Research Council, Institute of Terrestrial
Ecology set out to provide national statistics on vegetation cover and its species
composition. The Institute of Terrestrial Ecology, National Ecological Survey, which took
place in 1978 used an approach developed by Dr R Bunce of that Institute.
The Survey required detail that could only be provided by surveyors working on the
ground. The three distinct levels of information were:
A land classification based on multi-variate analysis of physical
variables (geology, climate, topography) designed to provide a
framework for stratified sampling. The resulting hierarchy of 32
land classes defining the environmental variation of Britain was
based on measurement of map data for each 1 km square from
the national grid. (220,000 in all).
Within each of the 32 environmental strata, random 1 km squares
were used for field sampling of the vegetation and features of land
cover. (8 from each in 1978 in 1978, 12 from each in 1984 (384
squares) and now increased to 528 proportionalised to each land
class).
Within each square random quadrats of both open vegetation
and linear features (hedge, roadside, stream banks) were
surveyed to give details of the species composition and cover.
-------
170
The national extent and distribution of each type of land cover, vegetation type of
species is then calculated from the detailed field survey by summating data for each class
according to the area and distribution of each land class throughout the country. The
land class therefore provides descriptions of the environmental conditions to which
particular land use or ecological features are related through the data generated from
ground surveyed 1 km squares.
Monitoring Landscape Change
At the time that the Institute of Terrestrial Ecology were embarking on their national
vegetation survey, the Department of Environment together with its agencies the
Countryside Commission and Nature Conservancy Council wanted to assess the rate of
change that was taking place both nationally and regionally in the main countryside
features so that some assessment could be made of the implications for both landscape
and wildlife. It was soon recognised by the participants that the only retrospective
datasets available to enable change assessment to be made was by utilising the national
archive of aerial photographs.
The study was mounted to:
i) Examine major landscape features and their distribution from
three periods of historic air photography.
ii) From the aerial photography available the dates chosen were the 1950s
to 60 around 1970 (within two or three years) and around 1980 (within
about two years).
The Department of Environment only has jurisdiction in England and Wales and not
Scotland or Northern Ireland. The study was therefore embarked upon jointly with the
Countryside Commission for these two countries.
The survey consisted of:
i) A sample of 707 sites of approximately 5 sq kms.
ii) At 140 of these sites a 1 km sq was ground surveyed.
iii) The rest of the sites were surveyed from 1:50,000 scale air
photography.
-------
171
iv) The sample of 2.3 percent of the land area of England and
Wales collected data on a number of similar features for each of
the three dates of air photography. (See Table 1).
v) The sample was stratified according to one of 16 generalised soil
types within each county of England and Wales. This
combination was chosen because county boundaries would
enable grossing up to DOE regions.
vi) The size of sample sites was considered to be the smallest area
for which the results would reach the specification of +/-5
percent error where features had a coverage of at least 15
percent of the area.
vii) Landsat Thematic Mapper census survey for England and Wales
using a reduced number of amalgamated broad land cover
features.
This "Monitoring Landscape Change" (MLC) study, produced a mass of net change
statistics indicating the magnitude of change from one landscape feature to another as
well as an estimate of the stock of these features at each of the three time periods.
Because the survey sample was limited in both numbers of samples and the total area
surveyed it was not possible to provide better than regional scale data on change. This
went some way to answering the problem of how much change was occurring in
countryside features, but it was not possible to answer the question as to the implications
in terms of significance to landscape or wildlife because such data resolution was such
that assessment could not measure feature quality and/or what mixture of features make
up landscape value (Table 1).
In order to overcome some of the shortcomings of a limited sample in assessing the
stock of features a parallel investigation using Thematic Mapper (TM) satellite images
took place. Investigation of the accuracy of area prediction by confusion matrices
indicated the importance of technique in expressing the pattern of features in the
landscape but showed that accuracy even at 85 percent was insufficient in monitoring
change. Change requires high mapping accuracy, particularly when the amount of
change is small.
The MLC contract was carried out by Hunting Surveys and Consultants, part of the
Hunting Group of companies. They completed the study in two years of intensive air
photographic interpretation at a cost of some £400,000
-------
172
In order to conduct the MLC survey a joint steering committee had been set up
between the Countryside Commission, Department of Environment and the Nature
Conservancy Council. Although funds were provided by both the Countryside
Commission and Department of Environment, the Nature Conservancy Council was
unconvinced that the scales of air photography used and the sampling intensity were
sufficient to describe features accurately enough to assess wildlife habitat.
The Nature Conservancy Council therefore decided to go its own way and set up
a National Countryside Monitoring Scheme (NCMS) using low level air photography
(1:25,000) which attempted to determine ground features in greater detail. Although NCC
managed to complete a study for Scotland using this approach, it was for only two
periods: 1945 (approx) and 1976. In spite of increasing the sample to 10% of the land
area, they also found that the results were poorly translatable in terms of habitat and
landscape.
Ecological Classification of Running Waters
In parallel with these efforts to obtain statistics on change in the landscape and its
implications DOE started a study in 1975 with the objective of determining the effect of
regulating the flow of river water by headwater storage reservoirs on the ecology of rivers.
Terms of reference were:
i) To examine ways of improving prediction of after effects of river
flow control.
ii) To obtain some advance warning of expected effects.
iii) To explain the observations made from monitoring of river
systems.
The UK Department of Environment contracted the Natural Environment Research
Council/Institute of Freshwater Ecology (IFE) - then known as the Freshwater Biological
Association (FBA) - to:
a) Sample a wide variety of British rivers in areas known to be
as free of pollution as possible.
b) To describe the fauna in terms of its distribution according to physical and
chemical data sampled for the same stretches of rivers.
c) Analyse this dataset by multi-variate analysis to develop a river
classification capable of describing any reach of river on the basis of the
faunal community associated with chemical and physical status.
-------
173
This system has now been developed for the UK. Measurements of variables can
be input into a computer model (RIVPACS) in order to enable a description of the
expected fauna to be output. Departures from this prediction are indicative of the effect
of changes in river regime outside normal limits of variation.
Research over the past 15 years has enabled improved predictions to be made and
then to place in the context of observed changes produced by such changes as pollution
as well as river regulation.
The Countryside Survey 1990
During the period 1987-1990 research was conducted into the opportunities for co-
ordinating the types of approach used in the three surveys described above. This
research programme was entitled "Ecological Consequences of Land Use Change"
(ECOLUC).
Only the ITE survey provides a general classification to which all land use and cover
types can be related as well as details of the ecology. The field survey information has
been repeated in 1990 for comparison with 1978 and 1984 and provides quantitative data
on land cover, vegetation type, and plant species composition. This database includes
information on the distribution of plant species in open land, at its margins and in linear
features, such as roadsides and can be summarised to include their spatial relationships
or purely as summaries by 1 km square or land class. (See Annex B).
-------
TABLE 1 MONITORING LANDSCAPE CHANGE PROJECT
Area feature: •••uriMd results for MM whole of England and Wales, 1947-1980
Feature
Broadleaf
Coniferous
Nixed
Woodland
Upland heath
Upland grass (smooth)
Upland grass (coarse)
Blanket bog
Bracken
Lowland grass heath
Lowland heather
Semi -natural vegetation
Cultivated land
Inproved grassland
Rough grassland
Neglected grassland
Farmed land
Water /Wet land
Built-up land
Urban open space
Transport routes
Other land
Total
1947
Cover
per cent
5.6
0.7
0.7
7.0*
3.0
1.Z
4.6
0.7
1.1
1.5
0.4
12.6*
26.1
38.1
2.9
3.7
72.7*
1.3*
4.5
0.7
0.5
6.4*
100.0*
Relative
standard
error*
X
5.7
24.0
14.6
5.3
20.3
20. B
14.6
27.7
20.4
16.0
22.8
7.4
2.7
2.8
9.0
8.8
1.4
18.3
5.8
13.5
19.1
5.5
1969
Cover
per cent
4.7
2.2
1.0
7.9*
2.4
0.9
4.3
0.7
1.0
0.4
0.2
10.1*
31.7
34.5
2.2
3.6
72.1*
1.1*
6.5
1.1
0.5
8.8*
100.0*
Relative
standard
error*
X
5.7
19.8
12.0
6.8
26.2
24.8
17.5
27.4
23.4
24.3
34.5
9.6
2.5
3.1
8.7
8.1
1.4
17.2
5.2
10.2
14.3
5.0
1980
Cover
per cent
4.2
2.7
0.9
7.9*
2.4
0.6
3.9
0.7
1.0
0.3
0.2
9.2*
35.4
31.0
2.2
3.1
71.8*
1.1*
7.3
1.3
0.5
10.0*
100.0*
Relative
standard
error"
X
*.i
16.5
11.7
6.8
25.5
28.0
18.8
28.5
25.2
29.5
31.6
10.2
2.6
2.9
9.2
9.7
1.4
16.7
5.1
9.6
11.7
4.9
100.0*
•Relative standard error Is equal to coefficient of variation.
*Figures for sub-totals include rare features for which percentages are not presented In this
the Final Report.
ry table. Full tables ere presented In Voluaes 3. 4 and 5 of
-------
175
Definitions
In Table 2, a number of British land assessment studies are compared. Although
they measure the same broad categories of land cover, close examination indicates that
features such as permanent grassland have a different definition because they have been
measured from aerial photographs, satellite scenes or ground survey. In the latter
instance, species composition has been used to define the feature.
It is vital in any survey that undertakes to link past datasets that data are collected
that can be manipulated in the same way as the previous surveys. In the Countryside
1990 Survey the 1 km squares data are recorded by ground survey, low level air
photography as well as satellite remote sensing. Study of the ground survey information
has demonstrated the importance of linear features such as hedgerows and stream sides
in maintaining diverse ecological communities. The importance of surveying these
landscape features in order to provide data on implications for species content arose from
the three-year period of research leading to the present survey (ECOLUC).
The MLC project identified the scale of changes recorded in major features. The
scale of these changes has been applied to ecological data from a number of other
studies derived from many sources in order to see whether the resolution at which MLC
worked and at which changes were observed, was meaningful in ecological terms. It is
apparent that measures of fragmentation and isolation of habitats such as woodlands and
the frequency of occurrence of these fragments in the landscape have important
implications for the make up and variety of bird species that may be associated with
woods and hedgerows. A number of algorithms were derived to measure both buffer
zones and distances from different landscape features to analyze these associations.
Freshwater Studies
The analyses to link these into the land classification have evaluated the extent to
which topographical, meteorological, geographical and cartographic variables (landscape
variables within the ITE Classification) can be used independently or in conjunction with
physico-chemical descriptors of river type used in RIVPACS.
Linkage between change in the land cover and land use in river catchments has
been associated with change in the datasets on invertebrate communities utilised in
RIVPACS. Analysis of these data show a correlation coefficient of 0.946 between land
class mean on the first access of the multi-variate analysis and the macro-invertebrate
assemblages used to place GB rivers in one of 32 river classes. Studies took place on
eight river catchments - a total of 34 sub-catchments. Each was sampled in 1980 and
again in 1989 and comparisons made with land use changes in that period.
The success of this approach is so promising that the UK National Rivers Authority
(NRA) has included as part of their quinquennial series of river surveys, RIVPACS
-------
176
The success of this approach is so promising that the UK National Rivers Authority
(NRA) has included as part of their quinquennial series of river surveys, RIVPACS
compatible samples from 4000 sites totalling 9000 samples. This dataset will allow
examination of the relationship between water quality and land use. As a result it should
now be possible to upgrade the RIVPACS system hitherto based on the relationship
between macro-invertebrate assemblages and physical and chemical characteristics of
the river, to include landscape and land use variables. This is particularly important in
assessing the true significance of point source pollution incidents against diffuse effects.
It will improve environmental options set according to measures of "best practicable
environmental option", i.e., balancing the possible effects of pollutant disposal to water
as opposed incineration, dumping at sea or disposal to land.
One-Kilometer Square Land Classification - Ground Survey of 528 Squares
Through the stratification system provided by the land classification and the sample
survey of a proportion of individual 1 km square on the ground, it is possible to determine
sample means and their variance within each land class. For example this variance can
be represented in terms of numbers and lengths of hedgerows, areas of different crops,
quantity of woodland, transport routes etc. Hitherto survey results are applied as class
means for each class to the 1 km squares that make up a particular area. An area of
interest can be described in terms of the proportion of the 32 land classes that may be
present.
It is also possible to describe each land class in terms of features in the form of a
landscape sketch. This shows the numbers of a particular feature distributed as an "ideal"
landscape. As the quantity of each feature can also be described in terms of its 95
percent confidence limits it can also be presented in the form of the highest and lowest
numbers of features superimposed on the same landscape. By this method policy
makers can have a visual representation of the possible variation in a landscape that may
come about in response to different economic support strategies for land management.
Subjective measures of like/dislike of a particular landscape change can be translated
into objective terms. Summation of the total proportion of squares that may be affected
by changes can then be used to measure the implications regionally, nationally or by area
of interest eg, National Park or water catchment, of altered land management practices
(see Table 2).
-------
TABLE 2 FEATURES IN EACH SURVEY
CS 1990
water
beach
saltmarsh
maritime grass
amenity ley permanent
pasture
herb rich grass
fen marsh
upland I moorland grass
bog
moor shrub heath/burnt
IANDSAT CLASS
sea/estuary
inland water
tidal flats/beach
saItmarsh
•and dune/grass heath
managed grass
rough grass
fen/mar»h/rough grass
herbaceous
montane grass
upland bog
heather moor
NLC
coast/estuarine
open water
saltmarsh
sand/shingle
pasture amenity
rough pasture, neglected
pasture
freshwater marsh
upland grass moor
peat bog
upland heath
LANDCOV
SCOTLAND
water
beach
saltmarsh
marram dune
improved grass airfield
smooth grass, golf links
wetlands
Nardus/Molinia
bogs
wet heather undefined
»CMS
open water
semi-improved I improved
grass, recreational
unimproved grass
lowland mire wet ground
blanket mire
heather moor, montane
heather
Pteridiura
bracken
bracken
bracken
bracken
-------
TABLE 2. CONTINUED
CS 1990
lowland heath
scrub
trees/deciduous species
trees/evergreen species
mixed wood
-
arable
vacant land, abandoned
and fallow
built
bare
felled wood
LAMDSAT CLASS
heather heath
scrub/orchards
decisuous/mixed
evergreen
-
-
arable
rudiral weed
suburban, urban
bare
felled
££
lowland heath gorse
scrub
broadleaved
coniferous
mixed wood
orchard/hops
arable/market garden
-
housing transport
derelict
rock, mineral
-
LANDCOV
SCOTLAND
dry heather
scrub
broadleaved
coniferous
undifferentiated nix
-
arable
-
developed rural, built-
up road, rai I
bare
felled
Nog
maritime heath
scrub
b1 leaved wood/plantation
conifer wood/plantation
mixed wood
orchard
arable
-
built
bare rock/soil
recent felled
linear/points aquatic,
flush, burnt arable,
misc. spp/uses
CLASSES EXCLUDED FROM
LAHOSAT CLASSIFICATION
A-linear/small features
B-isolated features
cloud, snow, mixed
mosiac, lines/points,
recent plough, wood,
open canopy, plantation
young plantation
marginal inundated
parkland
CO
-------
179
For each ground surveyed 1 km square, maps are produced Annex A shows the areal
extent of habitats and cover features as well as the extent of linear features (hedges,
walls, fences, ditches, roads - see Annex B). These data are being digitised for entry into
the ARCINFO geographic information system (GIS) and can be output in both vector and
raster format. This flexibility enables the areal data to be used to correct feature
estimates obtained from the satellite remote sensing programme.
Satellite Remote Sensing
This part of the survey has the following aims:
i) To compile a digital map of land cover in Great Britain based on a
hierarchical classification of land cover types (Annex B).
ii) To make quantitative assessments of accuracy of end products.
iii) To integrate this map with 1 km square topographic and
thematic data obtained from the land classification and
ground survey.
Landsat Thematic Mapper (TM) data will be geometrically corrected to OS national grid
using 25m output pixels. Summer/winter composites will be made by co-registering
scenes or part of scenes. The baseline will be 1990 +/- two years.
Images will be classified using maximum likelihood supervised classification.
Training of the classifier will use sub-division of target classes to ensure low within-class
variance and the number of training areas per sub-class has been adjusted to adequately
define sub-class-statistics. Consistency at the national scale will only be obtainable for
the aggregated 20- class product. Accuracy levels are likely to vary according to the
detail of level of sub-division, but the expectation from initial tests is to classify target
classes with 80 to 85 percent accuracy measured pixel by pixel or of 90 percent if
measured per land parcel using a "majority verdict" of component pixels. Checks will be
made against the ITE 528 1 km square ground data to enable correction factors to be
applied using the 528 squares as control points. By extracting 1 km sub-images from the
TM class maps and co-registering these with the 1 km field data a confusion matrix and
correspondence statistics will be derived to enable tabulations of accuracy by land cover
type and derive coefficients to correct the maximum likelihood classification.
Financial Investment
Some indication of the total cost of Countryside Survey 1990 and the lead in
research programmes is necessary in understanding the attention paid to stringent data
collection.
-------
180
The ground survey of 525 one-km squares has amounted
to E700K approx. out of a total for all aspects, including
data analysis and remote sensing of £1.8 million.
The lead-in research within the ECOLUC project cost
£350,000 over three years and the research to develop the
RIVPACS system amounted to £1.6 million over 10 years.
In addition experience gained from MLC of E400K should
be added to make the total direct investment just over £4
million.
Added to this there has been basic scientific research
undertaken by the Natural Environment Research Council
for some 15 years in developing and testing the land
classification and the river classification. It is likely that the
scientific investment required to attract applied customer
funding from the Government has not been far short of
another £0.5 million.
Discussion
Research that brings methodologies into applied use and requires investment over
periods of time and is of this magnitude tend only to be linked to policy needs in the later
stages of development at a time when their outputs can be identified as being able to
derive information within short timescales. This means that identification of resources in
the early stages can only be made if sufficient professional individuals working in the
particular field of inquiry, can be motivated and convinced of the likely eventual promise
of the method. This means that the science involved must receive adequate peer review.
Throughout the development of the method used for this 1990 Countryside Survey
there has been a peeling off of sections of the work which could be demonstrated as
being useful to a variety of users prepared to invest in the project and support continued
development. The river classification was linked to feasibility studies of engineering
proposals for regulating reservoirs which have themselves a long lead-in period and
therefore able to support a low level research initiative for some years. The land
classification approach proved to be useful at surveying a single county in England
(Cumbria, where the research station initiating the work was situated).
Development of the satellite remote sensing study has perhaps been less reliant
upon a customer identifying interest and more on a methodology chasing a use. But
even here it has been the UK Forestry Commission wishing to find ways of rapidly
updating stock maps that has provided long term funding and stimulus to link the three
levels of survey, ground, air photography and satellite data as is now being done in the
1990 Survey.
-------
181
Stimulus for the survey to take place at this point in time has arisen from public
pressure on politicians to preserve the landscape and wildlife of the British countryside,
seen to be threatened by the rapid development of farm intensification and associated
amalgamation of farms into larger and more efficient food production units. This has
been fuelled by a post-war Government policy to make the UK self-supporting in food
production. Since our entry into the Common Market supports in the European
Commission for the agricultural sector have further stimulated the process and confirmed
the perceptions of the general public that they do not like to see the changes taking place
in the countryside.
Emphasis on the use of survey is changing from a wish to know where changes and
their magnitude take place toward using these results as an objective description of
different areas so that perceptions of what is wanted can be turned into objective to
measures and used in a process of habitat restoration. A policy is beginning to develop
towards setting "Environment Quality Objectives" for land. This may possibly be an
ambitious use of the data which in consequence may be criticised as being inadequate
in hindsight. Quality control must therefore include a strong element of understanding
present objectives. Otherwise the very best of data can be devalued as being irrelevant
to new objectives.
Acknowledgment
This paper represents the views of the author and is not an official statement of the
UK DOE. I am grateful for the assistance in providing information and comment on this
paper, particular Dr. R. Bunce, Dr. C. Barr, Dr. M. Furse, Dr. T. Parr, Mr. R. Fuller and
Professor O. W. Heal.
-------
182
BIBLIOGRAPHY
Armitage, P. D., Moss, D. Wright, J. F. and Furze, M. T. (1983)
The Performance of a New Biological Water Quality Score System Based on Macro-
invertebrates Over A Wide Range of Unpolluted Running Water Sites. Water
Research Volume 17, Number 3.
Barr, C. J., Bunce, R. G. H., Riddsdale, H. and Whittaker, H. A. (1986).
Landscape Changes in Britain. Institute of Terrestrial Ecology, Monks Wood,
Huntington.
Bunce, R. G. H. and Heal, O. W. (1984)
Landscape Evaluation and the Impact of Changing Land-Use of the Rural
Environment: The Problem and an Approach.
In: Planning and Ecology. Ed. R. D. Roberts, T. M. Roberts, Chapman Hall:
London.
Bunce, R. G. H. and Smith, R. S. (1978)
An Ecological Survey of Cumbria. Cumbria C C and Lake District Special Planning
Board: Kendal.
Coleman, A. and Maggs, K. R. A. (1968).
Land Use Survey Handbook (5th edn). Isle of Thanet Geographical Association.
Furze, M. T., Wright, J. F., Gunn, R. J. M., Clarke, R. T., Johnson, H. A.,
Blackburn, J. H., Armitage, P. D., and Moss, D. (1991 in press)
The Impact of Land Use Change on Aquatic Communities Department of the
Environment Report.
Griffiths, G. H. and Williams, Ceri. (1990)
The Impact of Land Use Change on Aquatic Communities - Mapping Land Cover
Change in Water Catchments from Satellite Imagery. National Remote Sensing
Centre, Farnborough, Hants. GU14 6TD.
Griffiths, G. H., Wooding, M. G., Jewell, N. and Batts, H. A. (1988)
Use of Satellite Data for the Preparation of Land Cover Maps and Statistics. Report
to the DOE. National Remote Sensing Centre: Farnborough.
Harvey, D. R., Barr, C. J., Bell, M., Bunce, R. G. H., Edwards, D., Errington A. J., Jollans,
J. L, McLintock, J. H., Thompson, A. M. M. and Tranter R. B. (1986).
Countryside Implications for England and Wales of Possible Changes in the CAP.
Centre for Agricultural Strategy: Reading.
-------
183
Hill, M. O. (1989)
Computerised Matching of Releves and Association Tables, with an Application to
the British National Vegetation Classification. Vegetatio 83, 187-194.
Hooper, A. J. (1988)
Monitoring of Landscape and Wildlife Habitats EC. Environmental Management in
Agriculture - European Perspectives edit J. R. Park.
Nature Conservancy Council. (1987).
Changes in the Cumbrian Countryside. NCC, Peterborough.
Natural Environment Research Council. (1991 in press)
The Land of Britain. Natural Environment Research Council, Polaris House, North
Star Avenue, Swindon, Wiltshire SN2 1EU.
Stamp, L D. 1962.
The Land of Britain: Its Use and Misuse, 3rd edn. Longman, London.
-------
184
ANNEX A
COUNTRYSIDE SURVEY 1990
FIELD SURVEY QUALITY ASSURANCE by Dr. C. Barr and Dr. R. Bunce (ITE)
1. Countryside Survey 1990 is ITE's third major national field survey of the rural
environment. As part of a continuous process of improvement in methods and
standards of field survey, greater emphasis than before has been focused on quality
assurance. In planning the survey, time has been set aside to consult widely and
to analyse the collective experience of ITE staff and those in other institutions and
agencies with responsibility for the planning and execution of relevant field surveys.
2. This paper summarises the main ways in which quality control has been exercised,
before, during and after the field survey.
Pre-survey
3. 1984 surveyors' recommendations on mapping: At the conclusion of ITE's 1984
survey, a meeting was held at ITE Merlewood during which more than 30 major and
minor recommendations were made for future surveys of land cover and landscape
features. These were accepted and included in the planning of Countryside Survey
1990.
4. Consultants' recommendations on vegetation recording: As part of the 3 year
period of research "Ecological Consequences of Land Use Change" (ECOLUC)
project, ITE commissioned independent consultants to evaluate ITE's methods and
to make recommendations for vegetation recording in further survey work. A sub-
sample of 64 ITE sites was re-visited by Consultants in 1988, and vegetation
quadrats were recorded. Data were compared with those from the 1977/8 ITE
survey and the accuracy of change was assessed. These results, together with a
further exercise, involving an examination of observer variability, led to a series of
recommendations (eg the need to permanently mark quadrats, and to employ
experienced botanists). The consultants' major recommendations were
incorporated into the methods employed in the current survey.
5. Internal appraisal document: In February 1990, also resulting from ECOLUC, ITE
produced a publication titled "ITE Land Classification and its application to survey:
an internal appraisal". This review, which accommodated the comments of
international referees, included recommendations in field survey methods, especially
relating to sampling strategies (eg the need to move towards proportional sampling)
and statistical aspects (eg development of suitable statistical procedures for dealing
with data sets containing a high proportion of zeros). These recommendations were
implemented in plans for the current survey.
-------
185
6. Qualified survey staff: To mount the survey it was necessary for ITE to recruit 24
temporary field staff. These staff were selected from a total of over 200 applicants,
most of whom had considerable experience of botanical surveys. The 18 survey
teams, each of two persons, included at least one member or an experienced
consultant, especially in the early part of the field season.
7. Field handbook: A comprehensive handbook was prepared, based on the lessons
learned from previous surveys and recommendations, and incorporating the ideas
and advice of staff in government agencies and other interested organisations. The
handbook included both details of the standard methods to be used, and definitions
of categories to be recorded. (Plans are in hand to make this available for general
information and use).
8. Field training course: A two-week training course was held immediately prior to the
field season (in late May). The main objectives of the course were to teach and
standardise procedures, and to assess the botanical expertise of the surveyors.
More than 50% of the course was spent in the field, learning methods through
practical demonstrations and experience. The course was intensive but held in
comfortable surroundings with time available for a wide range of activities relating
to field work (eg seminars, botanical identification, first aid courses). Staff from a
range of Government agencies, academic institutions and others (eg Department of
the Environment, NCC, FC, IFE, Newcastle University and consultancies) were
invited to instruct surveyors in particular areas of expertise and to provide policy
background to the survey. The course was particularly valuable in bringing together
a large group of taxonomists who were able to work and learn together.
9. Aerial photographic interpretation: As an aid to field survey, aerial photographs
(taken post-1984) were obtained for each sample square and comparisons made
with the Ordnance Survey (OS) base maps. All physical boundary changes were
marked on the based map, as were isolated features such as individual trees.
Recognisable differences in ground vegetation types, especially in the uplands, were
marked as an aid to field mapping. All extraneous information (eg house names)
was deleted to give a clearer revised base map.
During survey
10. Mixing of survey teams: Although different regions of GB were allocated to the six
ITE Research Stations which are scattered throughout GB observer bias was
reduced by mixing the members of survey teams within a region, at intervals. As
well as reducing the chances of bias, this strategy ensured that surveyors were
frequently reassessing their performance against new partners.
-------
186
11. Permanent plot marking: To meet the overall objective concerning relocation of
vegetation plots, metal marker plates were placed in field boundaries, or at the plot
location in open land. Sketches were made showing the location of each plot and
its marker plate, and lengths and bearings to nearby landmarks were measured and
mapped. Finally, a photograph was taken showing the relationship of the quadrat
to its surroundings.
12. Supervision and expertise at each ITE Station: To guide the field surveyors with
administrative, logistical and procedural aspects, a Survey Coordinator was
appointed at each ITE Station. Additional botanical expertise was also made
available to allow surveyors to cross-check and confirm taxonomic identifications.
13. Field supervision: Apart from day-to-day supervision by Station Coordinators, each
field team was visited independently by the Project leader on five occasions
throughout the season, and checks were carried out on general procedures and
mapping. In addition, checks on vegetation recording and botanical identification
were carried out at least four times by independent external consultants.
14. Desk-checks of recording sheets: Data recording booklets were returned to ITE
Merlewood Research Station on completion, throughout the season. Checks were
carried out to ensure that 100% mapping has been completed in each site, and that
all quadrats were recorded and samples taken. Any problems were notified
immediately to Station Coordinators who ensured that omissions and errors were
corrected. In the event, less than 10% of recording booklets were affected and re-
visits were necessary in only two recorded cases.
15. Newsletter: During the survey, staff were circulated with six editions of a newsletter
which were useful in updating and clarifying points in the Field Handbook, as well
as providing a focal point for communication between staff.
Post-survey
16. Coordinators' feedback meeting: Having canvassed the views of field
surveyors based at their ITE Station, the Coordinators reported back to the
Project Management team in early December. Points to be discussed included
the use and interpretation of codes (and the identification of any limitations in
their use), and recommendations for future surveys.
17. Repeat survey: A sample of 30 sites (6%) was revisited to collect a second set of
data from each. The exercise was being carried out by project management staff
and consultants and allows an assessment of the quality of data-recording during
the season. Because the repeat survey was carried out at the end of the normal
field season, there will be an opportunity to examine the effects of temporal variation
and, in some areas, the effects in drought on vegetation. It is intended that an
-------
187
independent panel of expert field botanists should carry out a quantified assessment
of the two sets of field records and should carry out a quantified assessment of the
two sets of field records to identify the causes of any differences between them.
18. External checking of data recording forms: It is intended that an independent
consultant will check and comment on all data recording forms, especially as a
means of validating the botanical components of the land cover mapping, and the
vegetation quadrats. In addition, cross-checks will be made between the location
of species recorded in the Countryside Survey 1990 and information held at the UK
Biological Records Centre (part of the Environmental Information Centre at ITE
Monks Wood) which holds records and publishes maps of 10 km square scale of
the distribution of over 7000 UK species.
19. Checking of machine-readable data: All information collected during the field
survey is being entered into computers. Typing and coding errors will be reduced
by double punching by two separate firms entering data. There is partial repeat
digitising of cartographic data. Checks for legitimate code combinations will be
carried out.
20. Advisory Committee: General progress of the project is steered by a 15 strong
committee which includes sponsors, academics, government and international
observers from Europe.
-------
188
ANNEX B
ITE LAND CLASSIFICATION
Descriptions of main classes
1. Undulating country, varied agriculture, mainly grassland
2. Open, gentle slopes, often lowland, varied agriculture
3. Flat arable land, mainly cereals, little native vegetation
4. Flat, intensive agriculture, otherwise mainly built-up
5. Lowland, somewhat enclosed land, varied agriculture and vegetation
6. Gently rolling enclosed country, mainly fertile pastures
7. Coastal with variable morphology and vegetation
8. Coastal, often estuarine, mainly pasture, otherwise built-up
9. Fairly flat, open intensive agriculture, often built-up
10. Flat plains with intensive farming, often arable/grass
mixtures
11. Rich alluvial plains, mainly open with arable or pasture
12. Very fertile coastal plains with very productive crops
13. Somewhat variable land forms, mainly flat, heterogeneous land use
14. Level coastal plains with arable, otherwise often urbanised
15. Valley bottoms with mixed agriculture, predominantly
pastural
16. Undulating lowlands, variable agriculture and native vegetation
17. Rounded intermediate slopes, mainly improvable permanent pasture
18. Rounded hills, some steep slopes, varied moorlands
19. Smooth hills, mainly heather moors, often afforested
20. Midvalley slopes, wide range of vegetation types
21. Upper valley slopes, mainly covered with bogs
22. Margins of high mountains, moorlands, often afforested
23. High mountain summits, with well drained moorlands
24. Upper, steep, mountain slopes, usually bog covered
25. Lowlands with variable land use, mainly arable
26. Fertile lowlands with intensive agriculture
27. Fertile lowland margins with mixed agriculture
28. Varied lowland margins with heterogeneous land use
29. Sheltered coasts with varied land use, often crofting
30. Open coasts with low hills dominated by bogs
31. Cold exposed coasts with variable land use and crofting
32. Bleak undulating surfaces mainly covered with bogs
-------
189
SUMMARY OF LAND COVER DATA
Recorded for 528 1 km squares
Lolium perenne
Lolium/Dactylis glomerata
Dactylis glomerata
Mixture/unspecified
Hay/silage
PERMANENT PASTURE
Lolium perenne dominant
Lolium perenne present but also Holcus lanatus or Poa trivialis or Agrostis spp. in various
mixtures.
Unspecified/mixtures or generally improved pasture.
Cynosurus cristatus/Agrostis spp. or Holcus pasture that is not in good condition being
somewhat neglected.
ROUGH PASTURE
Agrostis/Fescue
Mixture/unspecified
Rush infested but not entirely rushes
Bracken infested but not entirely bracken
Mixtures of Deschamosia flaxuosa and Nardus stricta
OTHER DOMINANT SPECIES
Calluna vulgaris
Vaccinium myrtillis
Pteridium aquilinum
Juncus effusus/Juncus atriculatus and mixtures or marshland.
Molinia caerulea
Eriophorum vaginatum
Tall herb vegetation
Herb-rich grazed grassland
Ploughed land or fallow
Derelict
Wheat
Barley
Oats
-------
190
Sugar Beet
Kale/fodder species
Turnips/roots/swedes/rape
Potatoes
Beans/peas
General horticultural crops
Orchards
Roads
Built up land
Footpaths
Railways
Cliffs, sand and mud
Canal/Stream
Lake
PHYSIOGRAPHY/INLAND WATER/COASTAL
INLAND PHYSIOGRAPHIC FEATURES COASTAL FEATURES
Cliff > 30m high Rocky shore
Cliff 5-30m high
Pebble/gravel shore
Rock outcrop & cliff >5m high Sandy shore
Scree Sandy dune
Surface boulders Bare mud
Isolated boulder
Eroding raw peat
100% rock
>50% rock
10-50% rock
Stable raw peat
Current domestic peat workings
Current commercial peat workings
Old peat workings
Soil erosion
Ground levelling
INLAND WATER FEATURES
Lake natural Signs of drainage
Lake artificial Rock
Pond natural Sand/Gravel
Pond artificial Mud
River Peat
-------
191
Canalised river
Canal
Stream
Roadside ditch
Other ditch
Spring
Well
Levee
Lake shore
River bank
River substrate
Stream substrate
Waterfall
Rapids
Gorge
AGRICULTURE/NATURAL VEG ETC.
COVER TYPES
Amenity grass > 1ha
Ley
Permanent pasture
Upland grassland
Moorland grass
Moorland - shrub heath
Herb-rich grassland
Maritime grass
Lowland heath
Aquatic macrophytes
Aquatic marginal veg
Bog
Fen
Marsh
Flush - calcareous
Flush - non calcareous
Garden Centre/Nursery
Saltmarsh
Wheat
Barley
Oats
Mixed grain
SPECIES (IF > 25%)
Corsican pine
Scots pine
Lodgepole pine
Norway spruce
Sugar beet
Turnips/Swedes/Roots
Kale
Potatoes
Field beans
Peas
Lucerne
Maize
Rye
Oilseed rape
Other crop
Flowers
Commercial horticulture
Commercial glasshouse
Soft fruit
Ploughed
Vacant
Abandoned/neglected
Burnt
Fallow
PROPORTIONS
25-50%
50-75%
75-95%
95-100%
-------
192
Sikta spruce
Douglas fir
Larch
Western hemlock
Western red cedar
Other conifer
Elm
Oak
Beech
Ash
Sycamore
Birch
Poplar
Alder
Lime
Willow
Hawthorn
Gorse
Bramble
Other broadleaf
Mixed softwoods
Mixed hardwoods
Game/Sporting
DESCRIPTIONS/FEATURES
Undamaged
Cutting/Brashing
Felling/Stumps
Natural regeneration
Underplanting
Plantation
Planted
Ploughed land
Staked trees
Tuley tubes
Fenced single trees
Windblow
Dead standing trees
Re-growth - cut stump
USE
Commercial
Domestic
Timber production
Fuelwood production
Conservation
Amenity
Recreation
Grazing-Agricultural
Shelter
Game/Sporting
COVER TYPES
Scattered trees
Woodland/forest
Coppice
Scrub
Line of trees
Belt
Individual trees
Hedgerow tree
AGE
1-4 years
5-20 years
21-100 years
>100 years
-------
193
BOUNDARIES
Dry stone
Mortared
Other
FENCE
Wood only
Iron only
Wire
Other
HEDGE
>50% Hawthorn
> 50% Willow
>50% Beech
> 50% Gorse
> 50% Other
Mixed hedge
Hedge trimmed
Hedge uncut
Hedge derelict
Line of relict hedge
Laying
Flailing
RECREATION
FORMAL
School playing fields
Other playing fields
Race track
Tennis courts
Boating area
Static informal caravans
Golf courses
Static formal caravans
Information point
BANK
Stone
Earth
DESCRIPTIONS
>2m High
>2m High
>1m High
Stockproof
Not stockproof
Filled gaps <10%
Filled gaps >10%
Signs of replacement
Signs of removal
No longer present
Derelict
Burnt
INFORMAL
Horse jumps
Other horse accessories
Angling notice
Angling platform
Boat-house
Boat-inland water
Nature trail
-------
194
Touring caravan park
Camp site
BUILDING/STRUCTURES/COMMUNICATIONS
BUILT COVER TYPES
Building
Garden/grounds with trees
Garden/grounds without trees
Public open space
Allotments
Car park
Other land
USE
Residential
Commercial
Industrial
Public Service and
facilities
Institutional
Educational/cultural
Religious
DESCRIPTION
New
Vacant
Derelict
STRUCTURES
Bridge
Tunnel
Dam
Pipeline
Pylon
Other pole
Silo
Silage pit/clamp
Other Agri store
Snow fence
Speed restriction
Quarry/pit
Recreation areas
Sea
Rock
Forestry
Sporting/recreational
Waste domestic
Waste industrial
Quarry/mine
COMMUNICATIONS
Road (tarmac)
Verge <1m
Verge <5m
Verge <5m
Constructed track
Unconstructed track
Footpath (exclusive)
Footpath (other)
Railway track
Other railway land
Embankment
Airport/aerodrome
Informal barrier
-------
195
WOODLAND
Broadleaved copse
Mixed copse
Conifer copse
Broadleaved shelterbelt
Mixed shelterbelt
Conifer shelterbelt
Gillside woodland
Scrub
Broadleaved woodland
Conifer woodland
Mixed woodland
OTHER CATEGORIES
Phleum pratense
Lucerne
Maize
Nardusstricta
Molinia/Trichophorum/Eriphorum recorded as mixtures
subartic type vegetation
Vaccinium myrtillus
Erica tetralix
Rye
Juncus squarrosus
Unspecified mixed mountain grassland
Unspecified mixed mountain moorland always with some Calluna
Trichophorum - always with a proportion of Calluna
Calluna recorded as co-dominant with Eriophorum
Calluna recorded as co-dominant with Vaccinium
Burnt
Parkland
Maritime grassland recorded on sea cliffs
Mustard and oilseed rape
Mixed grain oats and barley
Saltmarsh
New built-up land
-------
196
Aggregation of classes for accurate map outputs from satellite remote sensing
water
sea/estuary
inland water
wetland
wetland
intertidal
vegetation
'woody'
man-made
suburban
bare
man-made
bare
lowland
upland
herbaceous
shrub
woodland/
trees
arable
non-vegetated
intertidal flats/
beach
saltmarsh
sand dune/grass
heath
managed grass
fur/marsh/rough herb
montane grass
upland bog
bracken
rudiral weed
heather moor
scrub/orchards
heather heath
deciduous/mixed
evergreen
arable land
urban/industrial
felled
bare rock
-------
197
FOURTH ANNUAL ECOLOGICAL QUALITY ASSURANCE WORKSHOP
WORKGROUP SESSION REPORTS
-------
198
THE DEVELOPMENT OF GUIDELINES
for use of
DATA QUALITY OBJECTIVES
in
AQUATIC MONITORING PROGRAMS
Leader:
Dennis M. McMullen
Technology Applications, Inc.
Rapporteur:
Michael Guill
AScI Corporation
I. Introduction
To open each work group session,the workgroup leader, Mr. Dennis McMullen,
provided an overview of the Data Quality Objectives (DQO) process. He described the
process as consisting of seven stages: (1) statement of the problem to be resolved, (2)
identification of the research question, (3) statement of the inputs, (4) narrowing of the
boundaries of the study, (5) development of a logic statement, (6) development of
uncertainty constraints, and (7) optimization of design for obtaining data.
To assist the groups in understanding and applying the DQO process, he
presented ten questions categorized into three Stages. Stage one questions address the
problem of concern: What is the purpose of the environmental data? What are the
resources and time constraints? What are the consequences of Type I and Type II errors?
Stage two questions address information needs: What is the population of interest? What
level of confidence must attend results? Do pertinent, usable data currently exist? What
new data are needed? Stage three questions address the scientific approach: What
approaches to data collection are available? Which approaches provide data quality
commensurate with Stage two requirements? What research and development activities
are needed to meet Stage two requirements?
The workgroups were then given a brief overview of EPA's Environmental
Monitoring and Assessment Program (EMAP) objectives, namely: (1) to estimate current
status, extent, changes and trends in indicators of the nation's ecological resources on
a regional basis with known confidence, (2) to monitor indicators of pollutant exposure
and habitat condition, and to seek correlative relationships between human-induced
stresses and ecological condition that identify possible causes of adverse effects, and (3)
to provide periodic statistical summaries and interpretive reports on ecological status and
trends to the EPA Administrator and the public. Three basic types of indicators to be
-------
199
estimated were identified: Response Indicators (e.g., indices of biotic diversity), Exposure
Indicators (e.g., toxicity tests and habitat assessments), and Stressor Indicators (e.g.,
population density and land use).
Mr. McMullen introduced the concept of the DQO Hierarchy, with Measurement
Quality Objectives as the first tier, addressing baseline data; Indicator Quality
Objectives as the second, addressing the indicators of ecological interest; Ecosystem
Quality Objectives as the third, addressing the consideration of multiple indicators
together; and Societal/Environmental Endpoints as the fourth and final tier, addressing
cross-ecosystem assessments.
Mr. McMullen's presentation then focused on the EMAP Surface Waters
objectives, which involves investigation of three specific endpoints: Trophic State (in
particular, the occurrence of eutrophication due to anthropogenic nutrient loading),
Fishability ("Are there fish in these systems, can I catch these fish, and if I catch them,
can I eat them?"), and Biological Integrity (defined as the ability to support and maintain
a balanced, integrated, adaptive community of organisms with a species composition,
diversity, and functional organization comparable to that of natural habitat of the region).
Mr. McMullen then reviewed the development of the logic statement for the EMAP
Surface Waters sediment toxicity indicator. The DQO process forces the investigator to
remain focused on the central questions and the information required to answer these
questions, as can be seen in the following DQO-based question/answer pairs:
1. Q: What is the problem or environmental concern that this indicator will
address?
A: Whether the bottom sediments of lakes in the U.S. are toxic
to aquatic life.
2. Q: How does this problem relate to societal interests or
concerns?
A: Sediments may be hazardous to humans. They may be
critical habitat for ecologically important species, or loss of
critical species resulting from sediment toxicity may lead to
loss of fisheries and lower aesthetic value.
3. Q: What information is needed to address the problem?
A: A method of measuring sediment toxicity is needed; a
definition for "indicator species" is needed; and a method for
assuring that the laboratory test design reflects in-vivo toxicity
is needed.
-------
200
4. Q: What data will be collected in the field and by what method?
A: A field investigation plan is needed that will provide data to
describe, at a minimum, the types and levels of contaminants
in sediment; fish community structures, including species
identification, abundance, and diversity; and the
physical/chemical environment, including water chemistry and
the physical characteristics of water bodies studied. This plan
must include a sampling plan that identifies specific methods
to be used and describes the rationale for selecting sampling
stations and sample collection frequencies.
5. Q: What laboratory analyses will be performed and by what methods?
A: Workgroup participants suggested the following investigations
to determine sediment toxicity: (1) a 10-day acute, solid-
phase test with amphipods using survival as the endpoint to
be measured; (2) a 7-day chronic, solid-phase test with fish
using survival and growth as endpoints; and (3) a 7-day
chronic, solid-phase test with Daphnia, using survival and
reproduction as endpoints.
6. Q: How will the data provided be used to address the problem?
A: The laboratory data will demonstrate the relative toxicity of
sediment to a single or multiple indicator species. From this
information, population and community level effects can be
extrapolated.
After working through the process of answering each question, workgroup
participants concluded that the original question to be addressed should be further refined
and restated as: What number/percentage of lakes in a given region of the U.S. have
sediments that are toxic to aquatic life, as demonstrated by effects observed relative to
controls in laboratory bioassays? The work group also identified the sources of variability
in the data. They included: (1) measurement variability (including analytical measurement
variability and other measurement variability, such as from sample collection, and
handling); and (2) sampling variability (including spatial and temporal representativeness
of the sample, and the representativeness of test method (i.e., laboratory test vs. field
effects).
-------
201
II. Summary of Discussions
Following the introduction to DQOs, each work group session applied the
suggested process to development of DQOs for a program designed to select a fish
tissue contaminant indicator to be used by the Surface Water component of EMAP.
Available time was not sufficient for either session to complete all seven steps of the DQO
process. Following is a summary of discussions regarding the first four steps.
State the Problem to be Resolved. Since the goal of the case study was selection of
appropriate indicators of fish tissue contaminants for the Surface Water component of
EMAP, workgroup participants focused on problems inherent to making scientific
generalizations over a wide range of species, contaminants, and environmental settings.
The workgroup leader suggested that the general underlying problem to be resolved in
the case study is the lack of information and understanding of fish tissue contaminants
and contaminant levels in the U.S.
Identify the Research Question. The two sessions arrived at fairly similar preliminary
research questions:
Is the consumption of fish from a particular reach of river hazardous to
human health?
Are the fish in a given area safe to eat? Is it necessary to issue a fish
consumption advisories for specific areas?
State the Inputs. In order to characterize the information needed to answer the research
questions, participants in both sessions developed extensive lists of issues to be
addressed. The issues fell into several general categories, as follows:
Definition of terms: Participants in both sessions identified
numerous terms that must be clearly defined and understood
at the outset of the project. These included terms such as
"contaminant", "area", "fish", "safe", and "hazardous".
Identify contaminants of concern: Participants agreed that the
contaminants of concern must be determined early in the
planning process. Questions were raised concerning how a
contaminant list could most effectively be generated; whether
it should include all compounds for which toxicological data
exists, or whether it should be limited to contaminants that
have been identified by a field reconnaissance program. It
was suggested that the investigator should identify: (1)
potential sources of contaminants (conspicuous point
sources, industry in the region, effluents, contaminated
-------
202
runoff); (2) information available from data generated in past
epidemiological and toxicological studies; and (3)
contaminants of regional concern historically. It was further
suggested that the investigator use this information to plan
and conduct a pilot study in order to refine the study and
assure that results of future data collection and analysis will
be of sufficient quality to answer the research question.
Method detection limits: Both sessions addressed the
problem of interpreting "Not Detected" results and
acknowledged the problems that arise when currently
available measurement techniques are not sufficiently sensitive
to detect concentrations of concern.
Participants identified specific information that would be necessary to answer the
research questions, including: (1) the trophic status of the body of water in question; (2)
the natural history of the fish in question (including the location of feeding, whether they
are resident or migratory, and size and age distributions); and (3) data on the
bioconcentration factors of the contaminants.
Each session discussed the topic of variability and the consequences of error.
Two possible undesirable effects of error in this instance were identified: (1) the closing
of fisheries which are not dangerously contaminated, and (2) the failure to regulate unsafe
fisheries.
Narrow the Boundaries of the Study. Four general topics of concern were identified
as points where the scope of the investigation could be narrowed: (1) What are the water
bodies of concern? (2) Which contaminants are to be investigated? (3) What fish (or
other) species are of concern? and (4) What human populations are placed at risk by the
contaminants?
III. Conclusions and Recommendations
Participants in both sessions agreed that the DQO process provides a good tool
for identifying potential sources of error in a research study design and for determining
acceptable levels of uncertainty in decisions based on environmental data. Participants
further agreed that, in order to be successful, the DQO process must be supported by
strong management endorsement, in terms of both philosophical commitment and
commitment of resources needed to implement the process. Other general conclusions
reached by the sessions and offered as guidelines for using the DQO process in aquatic
ecological applications include the following:
-------
203
Collection and review of historical data is a critical first step in
study design. Investigators should examine all relevant
studies and literature in detail prior to designing the
investigation. Federal and state agencies should also be
consulted to obtain unpublished information.
The use of pilot studies is an essential element in meeting the
objectives of the DQO process.
Investigators should be careful to determine that all data to be
collected will provide information useful in addressing the
research question.
-------
204
THE DEVELOPMENT OF GUIDELINES
for use of
STATISTICAL TOOLS
in
AQUATIC MONITORING PROGRAMS
Leader:
Dr. Peter MacDonald
McMaster University
Rapporteur:
Jan Edwards
Edwards Associates
I. INTRODUCTION
Workgroup sessions on the use of statistical tools in aquatic monitoring involved
two discussion periods. In the first, each session discussed aquatic monitoring programs
and problems that were of interest to individual participants. Each participant described
a case or problem of interest and identified key concerns or objectives related to the use
of statistical tools and monitoring program design. In the second discussion period of
each session, the workgroup reviewed some important statistical concepts and their
potential applications in aquatic monitoring.
II. SUMMARY OF DISCUSSIONS
As each workgroup participant described a study or monitoring program of interest
to them, the group identified the potential applications for statistical tools. The following
types of problems were discussed:
Questions of data comparability: Several participants described situations
where, because of budget constraints or changes in programs made over
time, data collected over periods of many years that varies in terms of
sample numbers and locations and/or analytical technique must be used
to characterize trends or draw conclusions. Problems inherent to making
comparisons of data sets from different sources (e.g., two different
laboratories) were also discussed. One challenging example involved
evaluating the quality of macroinvertebrate species identification and
enumeration data from numerous laboratories. The workgroup agreed that
a simple contingency table is a useful tool for comparing data sets from two
sources.
-------
205
Using small data sets: Workgroup participants identified cases where
conclusions must be drawn on the basis of relatively small data sets.
Participants noted that often, budget constraints or the limitations of
historical data require aquatic biologists to rely on relatively small data sets
to draw significant conclusions or make decisions. In such cases, it
becomes necessary to make an explicit evaluation of sample
representativeness.
Interpreting data outliers: Several participants described situations that
required techniques for explaining or interpreting data outliers. The
workgroup agreed that data interpreters should rely on all possible sources
of information, including notebooks kept by field personnel, which can
sometimes provide the key to otherwise inexplicable results.
Establishing baseline conditions or identifying "reference" lakes and rivers:
One participant described a case that required identification of a reference
river and noted that the wide range of variables to be considered makes
identification of an appropriate reference very difficult in aquatic
environments. Nonetheless, participants agreed that establishing baseline
conditions or a reference from which to evaluate change or trends is a
critical element of study design.
Data useability issues: Several participants described situations that
required application of data known to be flawed or of limited applicability.
They cited a need for tools that assist in determining when bias or error is
sufficient to render data unusable.
Developing sample stratification regimes: One participant described a study
that involved developing techniques for delineating regions within a wetland
or estuary environment. Participants noted that environmental regions or
strata often are not distinguishable in obvious ways. The workgroup
discussed use of multivariate analysis to examine numerous variables and
their relationship to each other in order to define regions. Participants
agreed that pilot studies should be used to identify and understand strata
so that effective sampling regimes can be developed.
At the close of each session, the workgroup reviewed some statistical principles
and tools that are generally helpful in designing environmental studies. The workgroup
participants concluded that application of statistical tools is helpful in data interpretation
in cases where there is evidence of data correlation, where effects are masked, where
there are missing data, where the wrong variable has been measured, or where other
confounding factors or variables are at work.
-------
206
Participants discussed the advantages of consulting statisticians early in the
development of a study design and agreed that statistics should be viewed by scientists
as an integral component of study design rather than as a "widget" or enhancement
added after the data are collected. Participants concluded that there are several basic
considerations that should be taken into account when designing an aquatic monitoring
program: balance, control, replication, reproducibility, and checks for correlation and
dependence. Balance or weighing considerations ensure that the correct suite of
variables are chosen for measurement. In general, participants in both sessions agreed
that it is advisable to collect as much data as possible while in the field, taking into
account budget considerations. Once the data are collected, they need not all be
evaluated at once. Establishing a control or reference for a study ensures that change
or trends over time can be reliably detected. The concept of replication allows for
distinguishing between measurement error and natural variability. Replication should be
achieved by repeating measurements at the most elementary level. Checks for
correlation allow the scientist to evaluate data in time and space to determine whether any
patterns of variability can be ascertained.
The workgroup addressed two different types of sampling regimes: stratification
and cluster sampling. Stratified sampling techniques are applicable when strata can be
defined within which there is relatively low variability for the parameters of interest. The
numbers of samples needed within strata increase as variability within strata increases.
Pilot studies are needed to define and understand strata within a population. In general,
stratified sampling regimes are more cost-effective because they allow collection of fewer
samples.
Cluster sampling techniques are applicable in cases where a cluster of sample
locations is representative of a whole population. Cluster sampling techniques simplify
the logistics (and potentially the cost) of sample collection. Their effectiveness however,
depends on the extent to which the cluster is representative of the whole population.
Again, pilot studies are needed to determine whether a cluster sampling technique will be
effective.
III. Conclusions and Recommendation
Based on the two sessions, the workgroup proposed the following guidelines for
the use of statistical tools in aquatic monitoring programs.
1. Statisticians should be involved in study design from the beginning, but only if the
statistician is experienced and interested in environmental survey design.
-------
207
2. Optimal statistical design requires information. Pilot studies should be used to
provide specific information needed for sampling design. Pilot studies should be
used to:
Understand the performance characteristics of the
measurement technique;
Investigate the scale of population variation sufficient to
determine whether stratified or cluster sampling is appropriate;
and
Select appropriate sampling intervals in space and/or time.
3. Pure measurement error expresses the limitations of the methodology and should
be measured at the most elementary level. Variation between laboratories will not
be less than variations within one laboratory.
4. Measure as many variables as practical. They need not all be analyzed at once.
In studying ecosystems, multivariate analysis is useful to an extent, but in
examining multiple variables, the scientist runs the risk of confusing the issue with
statistics. On the other hand, if measurement of additional variables can be
accomplished for relatively little extra cost, more information increases the chances
of measuring the "best" variable.
5. The role of control or comparison groups may differ from study to study, however,
in all cases, a baseline or standard for comparison should be established. New
studies can be used as a baseline for future study.
6. In cases where more than one measurement method is applicable, selecting the
best method should involve balancing decreasing cost with increasing variance.
7. Methods for combining results from different studies, called meta analysis, may be
borrowed from clinical studies for use in ecological studies. Such combining
depends on an ability to measure the reliability of each data-generating study.
8. In studies conducted over very long periods of time that experience a change in
measurement technology, new and old technologies should be run in parallel until
sufficient calibration is achieved.
9. Aggregating data in space or time reduces variability but may introduce bias.
10. Managers should make their own evaluations of monitoring data. Numerous
statistical support and graphical software packages are available to assist
managers with data evaluation.
-------
208
11. Graphical displays of data should be carefully designed to take into account the
concerns, needs, and perspective of the intended audience.
-------
209
DEVELOPMENT OF GUIDELINES
for use of
DATA QUALITY OBJECTIVES
in
TERRESTRIAL MONITORING PROGRAMS
Leader:
Elizabeth Leovey
USEPA
Rapporteur:
Allison Cook
AScI Corporation
I. Introduction
The leader of this workgroup, Elizabeth Leovey of EPA's Office of Pesticide
Programs, provided a case study as the context for application of the Data Quality
Objectives (DQO) process. The case involved EPA review of the registration for a
fictitious herbicide, "Power Wilt", under the Federal Insecticide, Fungicide, and Rodenticide
Act (FIFRA). Workgroup participants acted the roles of interested parties in applying the
DQO process and designing a monitoring study (or series of studies) needed to
determine the potential environmental consequences caused by application of Power Wilt.
This case was a relatively complex one because the hypothetical pesticide affects
several different components in the environment (i.e. plants, soil, water, and air), and
because its impact varies between sites due to factors such as climate and geography.
Also, the fact that the case involved an applied research study mandated under an EPA
regulation introduced policy concerns into the DQO process. By taking on different public
and private sector roles, the workgroup participants obtained first-hand experience in
solving an inherently adversarial problem.
Consequently, this workgroup examined the following types of questions:
How realistic is the DQO process in planning field research for a
perceived environmental problem?
-------
210
Since the DQO process is intended for research planning, does the
process in fact perform well in a situation that involves both scientific,
policy, and economic concerns in an inherently adversarial setting?
And finally, what are the strengths and weaknesses of the DQO process?
What improves the likelihood of success, and what can make the DQO
process work better in complex situations?
At the outset of the sessions, Ms. Leovey reviewed the five main steps in the DQO
process:
1. State the problem to be addressed,
2. Identify variables and select those to be measured,
3. Develop a logic statement for each variable to be measured,
4. Specify an acceptable level of uncertainty, and
5. Optimize the study design.
In the case study, the Environmental Protection Agency is requesting that the
manufacturer of "Power Wilt", Herbicides Inc., conduct studies to respond to public
concerns and resolve data gaps. Since the compound was originally registered in 1975,
additional adverse impacts have been discovered which have generated significant public
concern. Power Wilt was described as a highly toxic herbicide that kills broadleaf plants
with virtually no effects on mammals. Levels toxic to plants can be measured by present
analytical methods.
The specifics of the field study required may vary depending on the negotiation
format chosen. Different types of public participation are encouraged by the Agency. In
this example, participants could refine the required study through a public hearing, at
which all interested parties present their positions and specify the questions they would
like to see answered, and the problem-solving strategy they recommend. Or they could
choose to negotiate directly with the registrant instead. In this instance, representatives
from various interest groups may provide input into the study design process through
their state representatives, by attending some negotiation sessions, or by writing to EPA
staff directly.
The following perspectives were identified for role-playing purposes:
-------
211
The Environmental Protection Agency
EPA is responsible for registration of pesticides, including specification of label
instructions. Although the use of Power Wilt, as registered in 1975, was judged not to
cause unreasonable adverse effects on man and the environment, EPA is now concerned
about some of the new findings mentioned below. The Agency would like the registrant
to perform a new study, due to increased public interest in the material. EPA is interested
in discussing these issues with the registrant, particularly the fate of the chemical under
actual use conditions. The Agency's objective is to arrive at a reasonable solution that
can resolve most concerns.
The Registrant
"Power Wilt" is manufactured and formulated by Herbicides, Inc., a subsidiary of
a Fortune 500 Company. The registrant is motivated to maintain sales of the material
because it is a very remunerative product. Herbicides, Inc. attempts to maintain a
positive public image and claims that when used properly, the chemical is environmentally
safe. The registrant argues that if this product is banned, users will increase application
of materials that are more toxic to fish and wildlife.
Agricultural Interests
Agricultural interest groups have a range of concerns. Growers of crops that are
sensitive to Power Wilt claim that applications of the herbicide on grains and evergreens
have caused yield reductions on adjacent plantings of broadleaved crops. The material
is highly soluble in water and can be transported by spray drift, runoff of irrigation water,
and occasionally by fog. Some agriculturalists are also concerned about contamination
of surface water and groundwater supplies. The chemical is known to persist in the soil,
especially at higher latitudes, and in soils with high organic content. This characteristic
impairs farmers' ability to use crop rotations: for example, use of Power Wilt on a grain
crop will reduce yields on a crop of soybeans or sugar beets planted at the same site the
following season.
Environmental Interests
Environmental advocacy groups share some of the concerns of the agriculturalists.
They are also concerned with the movement of Power Wilt into surface water and
groundwater. In addition, they contend that airborne droplets of the herbicide, carried
into forests and other protected areas by low-lying fog, can threaten populations of rare
broadleaved plants.
-------
212
Other Interested Parties
Several less vocal parties are also stakeholders in the negotiation process.
Members of the State Highway Departments Association use this material in management
of roadside weeds. Consequently, the Association is concerned about liability from
growers affected by drift. Pesticide applicators are also sensitive to claims that they are
responsible for misuse of the material. Range farmers are in favor of continued use of
the compound, because the herbicide effectively removes weeds, improving rangeland
yields while not endangering grazing cattle. Forest managers like to apply Power Wilt to
prepare fir plantation lands before planting. Finally, U.S. Department of Agriculture is
interested in obtaining more information on the chemical before publicly advocating a
course of action.
II. Summary of Discussions
The process used to address this case varied considerably between the two
sessions, due primarily to different group dynamics and the different negotiation formats
selected. In the first session, participants chose to have EPA negotiate with the registrant
and representatives from different interested parties provided indirect input to the
negotiations. The workgroup member who played the role of the registrant was familiar
with the chemical company's perspective. His forceful negotiation style and argument
about the type of study that was affordable strongly influenced the DQO process. In the
second session, a public hearing format was selected. This allowed for more equitable
participation by all members. In this session, less time was spent discussing the problem
of interest, and more time was spent optimizing the research design.
Participants in the first session initially attempted to summarize the problem to be
addressed (Step 1 of the DQO Process). There was some controversy surrounding the
fundamental question posed.
Variants of the key question included:
Can losses to nontarget crop plants actually be attributed to "Power
Wilt"? Are they due to misuse of the material?
Are label requirements adequate to protect non-target plants?
What are the fate and transport of this material in the environment?
The participant taking the role of the EPA representative opened the workgroup
session and stated her position: that the registrant should conduct some additional
testing in order to determine whether or not losses of nontarget crops are in fact due to
misuse of "Power Wilt". The position taken by the environmental advocacy group was
-------
213
that in addition to the type of study envisioned by EPA, one or more field studies were
necessary to measure the detrimental impacts of the materials on surface water,
groundwater, and endangered species. Environmentalists also asked how the stated
acute toxicity measure of 40 parts per trillion (ppt) had been obtained, due to the fact that
this concentration is much lower than the amount measurable by gas chromatography
technology. This session then asked the registrant to conduct a fate and effects study
for a worst-case scenario. Late in the session, they discussed Step two of the DQO
Process, "Selection of Variables to be Measured in the Study". Variables of interest were
determined to be:
Fate and effects over time (persistence of the material)
Fate in different parts of the environment (such as water, soil, and
plants)
Fate under varying climate conditions (such as winter temperatures
and foggy conditions)
Effects on different nontarget crops of interest
Effects under different geographic conditions
Effects with different application techniques
The workgroup finally discussed the worst-case scenario (or set of scenarios) to
be chosen to optimize the study so that results could be obtained which pertained to the
majority of questions raised during the session. Conditions that were considered to
contribute to a worst-case scenario were:
Sensitive soil composition, high soil organic matter;
Timing in life cycle when broadleaf plants are most sensitive (i.e.,
early stages);
Application to different crop types:
forests
roadside weed control
grain crops
rangeland
Time of year when herbicide is normally applied; and
-------
214
Significant wind speed conditions, and possible foggy conditions.
It thus became apparent that several field studies would be needed in order to test all
variables of interest under worst-case conditions. In conclusion, the first session pursued
a fate and effects study, in part because this research option was familiar to the registrant.
It was decided that conducting such a study under worst-case conditions for all pertinent
variables would require setting up field plots at several sites.
During the second workgroup session, which involved a public hearing format,
there were three main players: 1) Environment and Agriculture as a coalition; 2) the
registrant; and 3) the Environmental Protection Agency. In this session, the
Environment/Agriculture coalition was vocal and well-prepared with data to strengthen
their argument.
The environmental and agricultural interest groups echoed concerns voiced during
the previous session: growers near roadside spray programs cited a significant decrease
in yield; these yield losses apparently had not responded to improved applicator training;
crop rotation systems were at risk; and rare forest species appeared to be vulnerable to
the herbicide. Canadian growers also requested that "Power Wilt" be tested under
northern conditions where it is apparently more persistent, as well as in the southeastern
U.S., which was the study site that the registrant proposed.
The Environment/Agriculture coalition thus helped identify variables that needed
to be considered in any studies. They were effective in holding this position because they
requested that the herbicide be otherwise substantially restricted in use or suspended.
This second workgroup session focused on the following research questions: 1)
is "Power Wilt" toxic?; and 2) At what concentrations? They questioned the 1975 data.
In order to obtain up-to-date phytotoxicity data, initial laboratory studies - bioassays of
plants such as potatoes, tomatoes, and soybeans at known dilutions of the chemical --
were eventually proposed. In addition to these laboratory studies, the registrant was also
asked to monitor sensitive crops in the field under different herbicide application regimes
and review the half-life of the compound in the soil under various climate conditions.
An important point that was raised during both sessions was that it was unclear
how the cited toxic dose to plants of 40 ppt had been obtained. Participants requested
that the registrant perform new toxicity measurements using state-of-the-art techniques.
During both workgroup sessions, the different interests of the main stakeholders
in the problem assured that certain concerns would be consistently raised. However,
each session approached the same problem in a different way. This was primarily due
to differences in group dynamics and the format in which participants chose to negotiate.
-------
215
III. Conclusions and Recommendations
Because of the differences in format used to address the case study, the two
workgroup sessions reached differing conclusions with regard to the utility of the DQO
process. The first session concluded that the DQO process is more appropriate for
simple monitoring studies, and less appropriate for basic research or complex problems.
Participants believed that in basic research, investigators must be free to modify the
fundamental question and the study design as the problem becomes better understood.
They concluded that the structure imposed by the DQO process would limit investigators
and does not encourage such mid-course correction during an investigation. Participants
in the second workgroup session concluded that the DQO process was generally useful
as a checklist, but need not necessarily be followed in the prescribed sequence of steps.
For example, participants found that Step 5: "Optimization of Study Design" could be
addressed before Step 4: "Specification of an Acceptable Level of Uncertainty". The
steps in the process were continually repeated as questions arose later in the process
that required going back to Step 1.
Workgroup members concluded that several common factors drove the DQO
process during both sessions. Time was considered to be one such factor. The fact that
the problem negotiation and study design needed to be completed during the finite
workgroup period put pressure on the participants to arrive at a workable solution.
In both sessions, it was evident that genuine participation by all interested parties -
- listening with a desire to cooperate and arrive at an agreeable solution contributed to
the success of the data quality objectives process. The fact that this case study was
regulatory in nature made the process adversarial. However, somewhat counterintuitively,
the adversarial nature of the situation helped speed the DQO process along and forced
all participants to arrive at a consensus solution. The fact that EPA might take regulatory
action provided further motivation for compromise.
Participants concluded that the process was also driven by the need to preserve
good science, as an end in itself, and also to maximize program effectiveness. Research
results that lack credibility can be interpreted as wasted program resources and leave
EPA (or any research organization) vulnerable to criticism by the public and during peer
review. Also, realistically speaking, other nonscientific factors drove the process, such
as cost and social concerns.
At the conclusion of the two sessions, the workgroup offered the following
recommendations as general guidelines for application of the DQO process to terrestrial
monitoring programs conducted in a regulatory setting such as that characterized by the
case study:
-------
216
The success of the DQO process will depend on the individual
participants and group dynamics. Although the DQO process lays
out a series of steps that can aid problem-solving, each outcome
depends on the players' willingness to participate, their individual
communications and negotiations skills, and other such factors, in
addition to scientific considerations.
All participants should provide input into the initial problem definition
stage. Such initial communication is vital to arriving at the best
research design that is also agreeable to all parties involved.
A facilitator is key, particularly at the problem definition stage, to
improve participation by all parties, and enhance process flow
according to a structured format. A skilled facilitator helps ensure
that the views of all parties are expressed, and thus ultimately
factored into the DQO process.
The most difficult parts of the DQO process, in both sessions, appeared to be:
Specifying the acceptable level of statistical uncertainty needed in
study results (this step requires that participants have some
background in statistics and a detailed understanding of the
biological system under study); and
Identifying the single study or series of studies that will answer the
questions posed.
In conclusion, the DQO process was considered to provide a helpful checklist in
the design of appropriate and scientifically valid field studies for the terrestrial monitoring
case examined. However, the sessions showed that in the case of "real world" situations
in which scientific considerations must be balanced with economic, social, and other non-
scientific factors, multiple valid solutions to the same problem can be possible, the non-
scientific factors may drive the process, and good communication and equitable group
participation are important in applying the DQO process successfully.
-------
217
THE DEVELOPMENT OF GUIDELINES
for use of
STATISTICAL TOOLS
in
TERRESTRIAL MONITORING PROGRAMS
Leader:
Dr. James Gore
Austin Peay University
Rapporteur:
Michael Piehler
AScI Corporation
I. Introduction
Both sessions of this work group were conducted as informal open discussions
that addressed the difficulty of applying appropriate statistical tools to data acquired from
terrestrial monitoring programs. Dr. James Gore, the work group leader, led the
discussion by posing questions to the participants pertaining to specific phases of a
scientific investigation and allowing the participants to discuss how procedures at each
stage of investigation affect the use of statistical tools. The objective of the discussions
was to recommend general guidelines for designing a terrestrial monitoring study that
lends itself to the use of effective statistics. Participants in both sessions agreed to focus
on monitoring studies designed for observational purposes (i.e., research-oriented
monitoring) rather than for purposes of investigating specific sources of pollution.
II. Summary of Discussions
Participants in both sessions agreed that there is a need for increased
communication between scientists and statisticians during all phases of the investigation.
Scientists should inform statisticians about specific factors in their studies that may
influence the statistical analysis of their data. Participants further agreed that, in general,
statisticians should understand certain aspects inherent to biological investigations at the
outset. These include:
1. That dependent and independent variables in a natural system are
often arbitrarily determined and sometimes influence each other.
There is often mutual causation (e.g., the level of moisture in a forest
influences the growth of trees and the growth of the trees increases
the level of moisture).
-------
218
2. That "statistical" significance is not always biologically meaningful.
Statisticians need to have some information about the system that is
being studied in order to effectively determine the appropriate
increment of analysis.
3. That biological measuring systems have inherent limitations in
precision and accuracy. This affects the level of detected change
which can be considered biologically significant. Furthermore, it
indicates that significant digits should be determined appropriate to
the measuring system's limitations.
4. That the full range of sampled phenomena must be considered in
order to detect a trend.
In addition, participants identified those considerations that statisticians expect from
scientists during a study. The statistician expects to be consulted before the scientist
begins sampling. The statistician also expects to have some preliminary data to predict
the required sample size and statistical design. Such data are often provided through a
pilot study.
The groups agreed that the first requirement for a statistically sound study is the
ability to state the question that is being investigated. Each work group session
encountered some difficulty in identifying the question typically asked in terrestrial
monitoring studies, but eventually participants agreed that the question most often asked
is: "Is there change over time?"
In the first session, it was established that a general framework for terrestrial
monitoring studies is needed. Participants suggested that the framework should include
a reference to the successional state of the area in question. Incorporating consideration
of successional change allows for separation of spatial and temporal variability by
providing a baseline of natural variability. The goals for detecting change can then be
defined as either deviation from a trend or deviation from a characteristic variance.
Both sessions identified many logistical problems that occur in large scale
terrestrial monitoring programs. A baseline measurement is often difficult to make in the
environment. Few historical data sets available are supported with full QA/QC
documentation and consequently often must be considered unreliable. Another logistical
concern in a large scale terrestrial monitoring project is the frequency of sampling. Both
groups agreed that because of constraints on time and money, extensive sampling at
high frequency is often prohibitive. Consequently, accurate characterization of temporal
variability can be difficult to achieve. A final problem with large monitoring programs
identified by participants was the assessment of the extent and focus of the sampling.
Participants agreed that terrestrial monitoring programs should have as one of their goals
providing data that will be useful for future studies. The difficulty is in creating a study
-------
219
design that is sufficiently robust to capture enough important information, while avoiding
unnecessary measurements.
Given these inherent dilemmas in terrestrial monitoring, the participants concurred
that a pilot study is generally necessary to focus monitoring on the relevant issues.
Several requirements were recommended for designing an effective pilot study, as follows:
1. Separate spatial and successional variation.
2. Sample at a frequency sufficient to assess annual variation so that
the investigator can determine the appropriate time of year to sample
and steps can be taken to avoid sampling rare conditions (i.e., those
existing less than 5 percent of the time).
3. Develop specific protocols for sampling, with different protocols for
discrete regions. Determine plot size and shape based on the
composition of the discrete areas.
5. Study plots should include ecotonal areas to provide an early
indication of change.
6. Always include QA and QC procedures.
7. When possible, pilot studies should not occur simultaneous with the
parent study, but should be a precursor to the main study. This
prevents the use of "ready-shoot-aim" strategies of data collection.
In each session, the participants agreed that large scale terrestrial monitoring
programs should not be based on arbitrary or political boundaries. By determining
natural boundaries and designing monitoring programs accordingly, available statistical
options are maximized. Finally, the workgroup sessions concluded that when designing
terrestrial (or any other) monitoring programs, regulators should not set goals for the
program without determining whether they are realistically attainable.
III. Conclusions and Recommendations
Based on the two sessions, the work group leader summarized conclusions and
recommendations concerning guidelines for the use of statistical tools in terrestrial
monitoring as follows:
-------
220
1. Scientists should consult with statisticians before beginning sampling.
Communications should continue throughout the study.
2. Before beginning a terrestrial monitoring program, there must be a
concise statement of the question to be addressed.
3. Incorporating an understanding of successional change is useful in
that it allows for the separation of spatial and temporal variability.
4. In designing a terrestrial monitoring program, the goal should be to
create a design sufficiently robust to capture information for future
use, while avoiding unnecessary sampling.
5. A pilot study is necessary before sampling begins to accurately
assess the scope and direction of sampling efforts and allow for the
use of effective statistics.
6. Large scale monitoring studies should be stratified using ecological
boundaries and not arbitrary or political boundaries.
7. Before setting goals for a terrestrial monitoring study, their
attainability should be assessed.
-------
221
DEVELOPMENTS OF GUIDELINES
for use of
DATA QUALITY OBJECTIVES and STATISTICAL TOOLS
in
ATMOSPHERIC MONITORING PROGRAMS
Leader:
Dr. Neville Reid
Ontario Ministry of the Environment
Rapporteur:
John Willauer
AScI Corporation
I. Introduction
Led by Dr. Neville Reid, the atmospheric workgroup addressed a single case study
from both the perspectives of the Data Quality Objectives (DQO) process and the use of
statistical tools in atmospheric monitoring. At Dr. Reid's suggestion, the group designed
an atmospheric monitoring network intended to provide data needed to support
development of broadly applicable emissions standards and other regulatory controls.
The group first applied the DQO process to develop a reasonable, cost-effective network
design. Then, in the second session, the group considered application of statistical tools
to enhance the utility of the monitoring data.
To develop DQOs for the case study, the workgroup structured its discussions
according to the following topics:
The Problem
The Issue
The Sources
The Customers/Users
Statement of Intent
What to Measure
How to Measure
When to Measure
Where to Measure
Possible Monitoring Networks
-------
222
II. Summary of Discussions
The Problem
The workgroup began by examining the case study presented by Dr. Reid, an
atmospheric monitoring network for toxic chemicals in the Great Lakes region. The
objectives of the study were to (1) monitor for ambient concentrations of toxic chemicals,
(2) follow time-trends in the atmosphere, (3) identify the sources, and (4) feed data back
to the development of regulations. Participants agreed that the scope of the case study
should be narrowed somewhat to make the task practical within the time allowed.
Consequently, the focus was narrowed to one chemical at a specific location. The
specific problem which the group decided to address was to develop DQOs and statistical
tools for a regional scale monitoring network designed to determine the input of PCBs to
Lake Ontario by wet and dry deposition from the atmosphere.
The Issue
The PCB concentration in Lake Ontario has declined and stabilized in the past, but
new evidence shows a recent increase. The major issue is the accumulation of PCB in
biota. Bioaccumulation of PCBs in fish tissue, especially Coho Salmon (Oncorhynchus
kisutch), has raised concern about the edibility of Great Lakes' fish. This increase could
also be of ecological concern to the region as weli. The workgroup agreed that this
matter may pose a health risk and may endanger the economies of the communities
along the lake which depend on the lake for their livelihood.
The next step involved examining preliminary studies of PCBs in Lake Ontario.
Preliminary studies suggest that atmospheric input accounts for approximately 50 percent
of the total PCB loading in the Lake. Isolated studies indicate that spatial variability may
be small. These studies also show that the temporal variability is small from a seasonal
stand point, but daily variability may be greater, and may need to be examined closely.
The Customers/Users
The workgroup identified possible customers/users for monitoring data. These
included provincial, state, and federal government organizations, the tourism industry, and
the sports and recreation industry in both Canada and the U.S. Participants agreed that
the customer's/user's questions should be understood and clearly stated. Questions
regarding measurement precision, accuracy, and species tolerance to PCBs at various
concentrations must be identified and addressed. The process should be designed to
understand the principal concerns of different user groups and the degree of clean-up or
reductions in PCB concentrations that each group believes would be acceptable. To
accomplish this goal, workgroup participants agreed that the DQO process should
incorporate everyone involved in the study: the researchers, the statisticians,
customers/users, and managers.
-------
223
At this stage in the process, participants agreed that the researcher should consult
a statistician to determine what data are needed to address the question, and then
determine what statistical tools are available to best illustrate how the data will answer the
customer's/user's questions. This information should be furnished to the
customers/users to show precisely the information that the study will produce and its
utility in solving the problem. Participants also stated that, when communicating with the
user community, it is not only important to state the objectives of the study clearly and
concisely, but also, to state the non-objectives — specifically what the study will not
address — as well.
The workgroup participants agreed that there should be an attempt to maximize
the utility of the data. In other words, as the DQO process proceeds, researchers should
recognize that not all of the customers/users can be determined from the outset. Other
customers/users might become apparent as the study progresses. Consequently, the
workgroup agreed that field studies should be planned to acquire as much data as is
reasonably feasible recognizing, however, that collecting too much collateral data can be
wasteful.
The Sources
There was considerable discussion and some disagreement about the sources of
the PCB input to Lake Ontario. The principal disagreement concerned whether the
sources of the PCB input could be located. The PCB concentration in Lake Ontario had
declined and stabilized in the past, but new evidence shows an increase in the PCB
concentration. This increase has been attributed to a shift in the ecological equilibrium.
The workgroup concluded that, since commercial production and use of PCBs have been
banned both in Canada and the United States, the increase in PCBs is likely to be
attributable to other sources and possibly the surrounding environment. Participants
believed that ascertaining what these sources are would be difficult and possibly not
feasible.
Initially, the workgroup agreed that locating specific sources of PCBs would be
unrealistic owing to the fact that the PCB input was coming from non-point sources. Two
known sources of PCBs are electric transformers and toxic waste sites, but these sources
do not account for the quantity of PCBs found in the Lake. It was suggested that the
increase may be occurring through terrestrial soil disturbance and through re-suspension
of the PCBs in lake sediment, caused by wave action.
-------
224
Statement of Intent
The workgroup agreed that the intent of the case study was to monitor
concentrations of total PCB in air and precipitation, and derive wet and dry deposition
rates for PCBs.
What to Measure
The workgroup agreed that the following data should be gathered by the
monitoring network:
PCB concentration in air and precipitation (congener specific)
Precipitation quantities
Precipitation type (rain, snow, etc.)
Water content in air
Total particulate matter and particle size distribution
Meteorological parameters
How to Measure
The workgroup recognized that atmospheric monitoring has inherent limitations in
precision and accuracy, and that these limitations should be clearly stated for the DQOs.
Participants identified the following measurement techniques:
Sorbent equipped high volume samplers with denuder for air
Wet-only collectors with sorbent for precipitation
Standard rain-gauge and meteorological instruments
Wind directional sampling was also mentioned, but it was pointed out that this type of
sampling may contribute bias to the study.
-------
225
When and Where
Determining when and where to sample was difficult for the workgroup. In the time
allowed, recommendations for actual sampling locations and frequencies could not be
developed. However, the workgroup identified several important issues that would affect
the final location of monitoring stations. These included:
The location of known or potential PCB sources to the Lake;
The degree of mapping in the atmosphere surrounding the
Lake basin; and
The importance of temporal variation in PCB loading rates.
Possible Monitoring Networks
The workgroup then developed some possible monitoring network schemes,
considering best-case scenarios first, and then examining the network taking budgetary
constraints into consideration. The first iteration utilized 30 sampling sites in a tesselated
pattern around lake Ontario, with two samples collected per week. The estimated cost
was $300,000 for the start-up and $3,000,000 per year for operation and analysis.
The second iteration sacrificed spatial resolution. This was done with a radial
distribution of the sampling sites, resulting in a decrease in the number of sites from 30
to 12, with two samples collected per week. The estimated cost of this scheme was
$150,000 for start-up and $1,500,000 per year for operation and analysis. The third
iteration sacrificed temporal resolution by cutting back sampling to one sample per week.
The workgroup concluded that a combination of the second and third iterations,
a compromise in both spatial and temporal resolution, should be developed for the final
version. Participants agreed that it would be important to sample at as many sites as
possible, but composite the samples for purposes of analysis, and integrate over longer
times.
III. Conclusions and Recommendations
The workgroup developed the following conclusions and recommendations
regarding the use of DQOs and statistical tools in atmospheric monitoring:.
1. DQOs should be set by a project team that includes the scientists,
statisticians, managers, and data users.
-------
226
2. In atmospheric monitoring programs, as well as other studies, the
question to be addressed must be properly and completely posed,
and especially with monitoring programs, there should be emphasis
on determining the importance and relative magnitude of temporal
variations in the parameters to be measured.
3. The customer/user should be identified and involved in the DQO
process. Their input will be required, to establish the question and
to set the quantitative expectations for data quality.
4. An attempt should be made to anticipate future data requirements
and to maximize the utility of the data but, there should also be an
awareness of the risks of becoming too diffuse and of producing
data of little or no utility.
5. Full specification of required data quality is needed, including
specifying the level of precision required or requested, how precision
will be calculated, and what the acceptable detection limits are.
6. Once the question is identified and defined and the types of data
needed are determined, combine this information with budgetary
constraints to obtain a practical study design.
7. In preparing alternative study scenarios, document what is lost or changed
between scenarios. This information will allow the researcher to illustrate
how all reasonable scenarios were investigated in the DQO process.
8. One of the principal benefits of the DQO process is that it documents
the study design and the process by which it is developed.
-------
APPENDIX A
LIST OF PARTICIPANTS
-------
APPENDIX A
FOURTH ANNUAL ECOLOGICAL QUALITY ASSURANCE WORKSHOP
Cincinnati, Ohio
February 26-28, 1991
List of Participants
Name
Group
Alvo, Dr. Mayer
Atmospheric
Bigelow, Dr. David
Atmospheric
Bisson, Dr. Robert
Atmospheric
Black, Mr. Paul
None
Bowers, Dr. James A.
Aquatic
Affiliation and Address
University of Ottawa
585 King Edward
Ottawa, ON K1N 6N5
CANADA
Ph: (613) 564-9532
FAX: (613) 564-5014
Colorado State University
Fort Collins, CO 80523
U.S.A.
Ph: (303) 491-5574
FAX: (303) 491-1965
Environment Canada
867 Lakeshore Road
P.O. Box 5050
Burlington, ON L7R 4A6
CANADA
Ph: (416) 336-4877
FAX: (416) 336-4989
Decision Science
Consortium/ICF
1895 Preston White Drive, Suite 300
Reston, VA 22091
U.S.A.
Ph: (703) 715-3402
Westinghouse
126 Riverera Rd.
Aiken, SC 29803
U.S.A.
Ph: (803) 725-5213
FAX: (803) 725-4704
A-1
-------
APPENDIX A, continued
FOURTH ANNUAL ECOLOGICAL QUALITY ASSURANCE WORKSHOP
Cincinnati, Ohio
February 26-28, 1991
List of Participants
Name
Group
Burgeson, Ms. Cindy
Aquatic
Burkman, Mr. Bill
Terrestrial
Byers, Mr. Gerald E.
Terrestrial
Cantillo, Dr. Adrianna
Aquatic
Coffey, Ms. Deborah
Terrestrial
Affiliation and Address
Management Technology
200 S.W. 35th Street
Corvallis, OR 97333
U.S.A.
Ph: (503) 757-4666
FAX: (503) 757-4335
U.S. Forest Service
100 Matsonford Rd., Suite 200
Radnor, PA 19087
U.S.A.
Ph: (215) 975-4155
FAX: (215)975-4200
Lockheed
1050 E. Flamingo Road, Suite 200
Las Vegas, NV 89119
U.S.A.
Ph: (702) 734-3337
FAX: (702) 796-1084
National Oceanic/Atmospheric
Administration
Rockville, MD 20852
U.S.A.
Ph: (301) 443-8655
FAX: (301) 231-5764
Management Technology
1600 S.E. Western Blvd.
Corvallis, OR 97333
U.S.A.
Ph: (503) 757-4666
A-2
-------
APPENDIX A, continued
FOURTH ANNUAL ECOLOGICAL QUALITY ASSURANCE WORKSHOP
Cincinnati, Ohio
February 26-28, 1991
List of Participants
Name
Group
Collins, Mr. Gary
Aquatic
Conkling, Ms. Barbara L.
Terrestrial
Cook, Ms. Allison
Terrestrial
Cuffney, Mr. Thomas F.
Aquatic
Danner, Mr. Robert
Aquatic
Affiliation and Address
USEPA
26 W. Martin L. King Drive
Cincinnati, OH 45268
U.S.A.
Ph: (513) 569-7325
FAX: (513) 569-7424
University of Nevada
4505 S. Maryland Pkwy.
Las Vegas, NV 89154
U.S.A.
Ph: (702) 597-4124
FAX: (702) 739-3094
AScI
1365 Beverly Road
McLean, VA 22101
U.S.A.
Ph: (703)847-0001
U.S. Geological Service
3916 Sunset Ridge Rd.
Raleigh, NC 27607
U.S.A.
Ph: (919) 571-4019
FAX: (919) 571-4041
USEPA
26 W. Martin L. King Dr.
Cincinnati, OH 45268
U.S.A.
Ph: (513) 569-7409
FAX: (513) 569-7276
A-3
-------
APPENDIX A, continued
FOURTH ANNUAL ECOLOGICAL QUALITY ASSURANCE WORKSHOP
Cincinnati, Ohio
February 26-28, 1991
List of Participants
Name
Group
Dixon, Mr. Tom
Atmospheric
Dwire, Ms. Kate
Terrestrial
Edwards, Ms. Jan
Aquatic
El-Shaarawi, Mr. Abdul
Atmospheric
Flueck, Mr. John A.
Terrestrial
Affiliation and Address
USEPA
401 M Street, S.W.
Washington, DC 20460
U.S.A
Ph: (202) 382-7238
FAX: (202) 260-4346
Management Technology
1600 Western Blvd.
Corvallis, OR 97333
U.S.A.
Ph: (503) 757-4666
FAX: (503) 757-4335
Edwards Associates
1302 Gibson Place
Falls Church, VA 22046
U.S.A.
Ph: (703) 237-1220
Environment Canada
867 Lakeshore Road, P.O. Box 5050
Burlington, ON L7R 4A6
CANADA
Ph: (416) 336-4584
FAX: (416) 336-4989
University of Nevada
4505 S. Maryland Parkway
Las Vegas, NV 89154
U.S.A.
Ph: (702) 739-0838
FAX: (702) 739-3094
A-4
-------
APPENDIX A, continued
FOURTH ANNUAL ECOLOGICAL QUALITY ASSURANCE WORKSHOP
Cincinnati, Ohio
February 26-28, 1991
List of Participants
Name
Group
Gerlach, Mr. Robert W.
Terrestrial
Gore, Dr. James
Terrestrial
Graves, Mr. Robert
Atmospheric
Guill, Mr. Michael
Aquatic
Heath, Mr. Richard H.
Aquatic
Affiliation and Address
Lockheed
1050 E. Flamingo Road
Las Vegas, NV 89119
U.S.A.
Ph: (702) 734-3293
FAX: (702) 796-1084
Austin Peay State University
P.O. Box 4718
Clarksville, TN 37044
U.S.A.
Ph: (615) 648-7019
USEPA
26 W. Martin Luther King Dr.
Cincinnati, OH 45268-1525
U.S.A.
Ph: (513) 569-7197
FAX: (513) 569-7115
AScI
1365 Beverly Road
McLean, VA 22101
U.S.A.
Ph: (703) 847-0001
SERC, University of Maine
Orono, ME
U.S.A.
Ph: (207) 581-3287
FAX: (207) 581-1604
A-5
-------
APPENDIX A, continued
FOURTH ANNUAL ECOLOGICAL QUALITY ASSURANCE WORKSHOP
Cincinnati, Ohio
February 26-28, 1991
List of Participants
Name
Group
Johnson, Ms. Lora
Terrestrial
Kawahara, Dr. Fred
Aquatic
Kern, Ms. Ann
Aquatic
King, Mr. Larry
Aquatic
Kirkland, Ms. Linda
Aquatic
Affiliation and Address
TAI
26 W. Martin Luther King Drive
Cincinnati, OH 45268
U.S.A.
Ph: (513) 569-7299
USEPA
26 W. Martin Luther King Drive
Cincinnati, OH 45268
U.S.A.
Ph: (513) 569-7313
FAX: (5130 569-7424
USEPA
26 W. Martin Luther King Drive
Cincinnati, OH 45268
U.S.A.
Ph: (513) 569-7635
FAX: (513) 569-7276
Ontario Environ Studies and Assessment
700 University Avenue, Room H10, E4
Toronto, ON M5G 1X6
CANADA
Ph: (416) 592-8596
FAX: (416) 978-0156
USEPA
401 M Street, S.W.
Washington, DC 20460
U.S.A.
Ph: (202) 382-5775
FAX: (202) 252-0929
A-6
-------
APPENDIX A, continued
FOURTH ANNUAL ECOLOGICAL QUALITY ASSURANCE WORKSHOP
Cincinnati, Ohio
February 26-28, 1991
List of Participants
Name
Group
Lawrence, Dr. John
Aquatic
Lazorchak, Dr. James
Atmospheric
Leovey, Dr. Elizabeth
Terrestrial
Li, Dr. Wan
Aquatic
Loehle, Dr. Craig
Terrestrial
Affiliation and Address
Environment Canada
867 Lakeshore Road, P.O. Box 5050
Burlington, ON L7R 4A6
CANADA
Ph: (416)336-4927
FAX: (416) 336-4989
USEPA
3411 Church Street
Newtown Facility
Cincinnati, OH 45244
U.S.A.
USEPA
401 M Street, S.W.
Washington, DC 20460
U.S.A.
Ph: (703) 557-2162
FAX: (703) 557-9309
Environment Canada
867 Lakeshore Road, P.O. Box 5050
Burlington, ON L7R 4A6
CANADA
Ph: (416) 336-4926
FAX: (416) 336-4989
Westinghouse
205 Longleaf Ct.
Aiken, SC 29803
U.S.A.
Ph: (803) 725-5462
FAX: (803) 725-4704
A-7
-------
APPENDIX A, continued
FOURTH ANNUAL ECOLOGICAL QUALITY ASSURANCE WORKSHOP
Cincinnati, Ohio
February 26-28, 1991
List of Participants
Name
Group
MacDonald, Dr. Peter
Aquatic
Maciorowski, Dr. Tony
Terrestrial
Mangis, Dr. Deborah
Terrestrial
McMullen, Mr. Dennis
Aquatic
Mickler, Mr. Robert A.
Terrestrial
Affiliation and Address
McMaster University
Hamilton, ON L8S 4K1
CANADA
Ph: (416) 525-9140
FAX: (416) 528-5030
Battelle
505 King Avenue
Columbus, OH 43201-2693
U.S.A.
Ph: (614) 424-6424
National Park Service
P.O. Box 25287
Denver, CO 80225
U.S.A.
Ph: (303) 969-2809
FAX: (303) 969-2822
TAI
3411 Church Street
Cincinnati, OH 45244
U.S.A.
Ph: (513) 533-8114
FAX: (513) 533-8181
Southern Global Change Program
1509 Varsity Drive
Raleigh, NC 27606
U.S.A.
Ph: (919) 737-3311
FAX: (919) 737-3593
A-8
-------
APPENDIX A, continued
FOURTH ANNUAL ECOLOGICAL QUALITY ASSURANCE WORKSHOP
Cincinnati, Ohio
February 26-28, 1991
List of Participants
Name
Group
Moench, Mr. Charles
Aquatic
Morrison, Ms. Marilyn
Aquatic
Neary, Ms. Anne
Terrestrial
Neptune, Dr. Dean
Aquatic
Newhouse, Mr. Steven
Aquatic
Affiliation and Address
USEPA
26 W. Martin Luther King Dr.
Cincinnati, OH 45268
U.S.A.
Ph: (513) 569-7325
FAX: (513) 569-7115
Management Technology
200 S.W. 35th Street
Corvallis, OR 97333
U.S.A.
Ph: (503) 757-4666
FAX: (503) 757-4335
Ontario Ministry of the Environment
P.O. Box 39
Dorset, ON POA 1EO
CANADA
USEPA
401 M Street, S.W.
Washington, DC 20460
U.S.A.
Ph: (202)475-9464
FAX: (202) 252-0929
Indiana Department of Environmental Management
5500 W. Bradbury Avenue
Indianapolis, IN 46421
U.S.A.
Ph: (317)243-5114
A-9
-------
APPENDIX A, continued
FOURTH ANNUAL ECOLOGICAL QUALITY ASSURANCE WORKSHOP
Cincinnati, Ohio
February 26-28, 1991
List of Participants
Name
Group
Nolan, Mr. Melvin
Aquatic
Palmer, Dr. Craig
Terrestrial
Peck, Mr. Dave
Aquatic
Piehler, Mr. Michael
Terrestrial
Pollock, Dr. Thomas
Aquatic
Affiliation and Address
USEPA
401 M Street, S.W.
Washington, DC 20460
U.S.A.
Ph: (202) 382-5975
FAX: (202) 382-6370
University of Nevada
4505 S. Maryland Parkway
Las Vegas, NV 89154
U.S.A.
Ph: (702) 798-2186
FAX: (702) 798-2454
Lockheed
1050 Flamingo Road, Suite 209
Las Vegas, NV 89119
U.S.A.
Ph: (702) 734-3254
FAX: (702) 796-1084
AScI
1365 Beverly Road
McLean, VA 22101
U.S.A.
Ph: (703) 847-0001
Environment Canada
Federal Building, Main Street, P.O. Box 861
Moncton, NB E1C 8N6
CANADA
Ph: (506) 857-6606
FAX: (506) 857-6608
A-10
-------
APPENDIX A, continued
FOURTH ANNUAL ECOLOGICAL QUALITY ASSURANCE WORKSHOP
Cincinnati, Ohio
February 26-28, 1991
List of Participants
Name
Group
Provoncna, Mr. Brian
Terrestrial
Ralph, Ms. Karen
Aquatic
Reid, Dr. Neville
Atmospheric
Schulte, Mr. Jerry
Aquatic
Schumacher, Dr. Brian
Terrestrial
Affiliation and Address
ENSECO
445 Pineda Court
Melbourne, FL 32940
U.S.A.
Ph: (407) 254-4122
FAX: (407) 254-3293
Department of Fisheries and Oceans
867 Lakeshore Road
Burlington, ON L7R 4A6
CANADA
Ph: (416)336-6425
FAX: (416) 336-4819
Ontario Ministry of the Environment
880 Bay Street, 4th Floor
Toronto, ON M5S 1Z8
CANADA
Ph: (416) 326-1691
FAX: (416) 326-1733
ORSANCO
49 E. 4th Street, Suite 300
Cincinnati, OH 45202
U.S.A.
Lockheed
1050 E. Flamingo Road
Las Vegas, NV 89119
U.S.A.
Ph: (702) 734-3229
FAX: (702) 796-1084
A-11
-------
APPENDIX A, continued
FOURTH ANNUAL ECOLOGICAL QUALITY ASSURANCE WORKSHOP
Cincinnati, Ohio
February 26-28, 1991
List of Participants
Name
Group
Simes, Mr. Guy
Aquatic
Simmons, Ms. Carol L
Terrestrial
Squires, Ms. Louisa
Aquatic
Stainken, Dr. Dennis
Aquatic
Stribling, Mr. Sam
Aquatic
Affiliation and Address
USEPA
26 W. Martin Luther King Dr.
Cincinnati, OH 45268
U.S.A.
Ph: (513) 569-7635
FAX: (513) 569-7276
NREL, Colorado State University
Fort Collins, CO 80523
U.S.A.
Ph: (303) 491-5580
FAX: (303) 491-1965
Management Technology
200 S.W. 35th Street
Corvallis, OR 97333
U.S.A.
Ph: (503) 757-4666
FAX: (503) 757-4335
Malcolm Pirnie, Inc.
2 Corporate Park Dr., Box 751
White Plains, NY 10602
U.S.A.
EA
15 Loveton Circle
Sparks, MD 21152
U.S.A.
Ph: (301) 584-7000
FAX: (301) 771-9148
A-12
-------
APPENDIX A, continued
FOURTH ANNUAL ECOLOGICAL QUALITY ASSURANCE WORKSHOP
Cincinnati, Ohio
February 26-28, 1991
List of Participants
Name
Group
Tilley, Mr. Larry J.
Aquatic
Vet, Mr. Robert
Atmospheric
Wentworth, Ms. Nancy
None
Willauer, Mr. John
Atmospheric
Affiliation and Address
U.S. Geological Survey
345 Middlefield Rd.
Menlo Park, CA 94025
U.S.A.
Ph: (415)329-4549
FAX: (415) 329-4463
Atmospheric Environment Service
4905 Dufferin Street
Downsview, ON M3H 5T4
CANADA
Ph: (416) 739-4853
FAX: (416) 739-4224
USEPA
401 M Street, S.W.
Washington, DC 20460
U.S.A.
Ph: (202) 382-5763
FAX: (202) 252-0929
AScI
1365 Beverly Road
McLean, VA 22101
U.S.A.
Ph: (703)847-0001
*U.S GOVERNMENT PRINTING OFFICE: 1 9 9 2 .6 ". 8 - o o s/ >. i e 5 2
A-13
-------
United States
Environmental Protection
Agency
Center for Environmental Research
Information
Cincinnati, OH 45268
Official Business
Penalty for Private Use, $300
Please make all necessary changes on the above label,
detach or copy, and return to the address in the upper
left-hand corner.
If you do not wish to receive these reports CHECK HERE [] ;
detach, or copy this cover, and return to the address in the
upper left-hand corner.
EPA/600/R-92/097
------- |