DEVELOPMENT
OF
AGENCY-WIDE
QUALITY CONTROL
PROGRAM
February 1973
U.S. Environmental Protection Agency
Office of Monitoring Systems
Quality Assurance Division
-------
26804
DEVELOPMENT
OF
AGENCY-WIDE
QUALITY CONTROL
PROGRAM
February 1973
U.S. Environmental Protection Agency
Office of Monitoring Systems
Quality Assurance Division
-------
TABLE OF CONTENTS
Page
INTRODUCTION 1-3
ELEMENTS OF A QUALITY ASSURANCE PROGRAM 4-18
Development and Issuance of Procedures .... 4-11
Sampling-Procedures 6-7
Method Selection Procedures 8-10
Laboratory Procedures 10-11
Intra-laboratory Quality Control 11-15
Quality Measurement and Performance
Evaluation 11-14
Personnel Staffing and Training 14-15
Inter-laboratory Quality Control 15-16
Management 16-18
Program Evaluation 17
Certification 17-18
Communication 18
CURRENT STATUS AND NEEDS 19-30
Water 22-24
Pesticides 24-27
Air 27-28
Radiation 28-30
IMPLEMENTATION 31-44
Organizational Plan 32-39
Projects and Schedules 39-44
-------
Page
QUALITY CONTROL PROGRAM FOR STATE AND
LOCAL AGENCIES 45-70
COMMENTS ON PLAN BY HEADQUARTERS' STAFF,
REGIONS, AND NERC's 71-80
-------
1
Executive Summary
DEVELOPMENT OF AN AGENCY-WIDE
QUALITY CONTROL PROGRAM
Introduction
The primary goals of a quality control program are
to improve and document the accuracy and validity of
environmental measurements. In achieving these goals/
quality control must be imposed on nearly all segments of
monitoring activities and should cover personnel/ methods
selection/ equipment, and data handling procedures. A
total quality control program is composed of four major
elements:
- Development and issuance of procedures
- Intra-laboratory quality control
- Inter-laboratory quality control
Monitoring program evaluation and
certification.
All these elements are equally essential to a successful
quality control program and must be developed and carried
out simultaneously.
Current Status
The on-going EPA quality control activities are
quite varied for the different media and categorical
programs. Certain groups exist within EPA that presently
provide some support for quality control. These are:
Air/ NERC-Research Triangle Park; Water/ NERC-Cincinnati and
NERC-Corvallis; Pesticides, NERC-Research Triangle Park/
Perrine; and Radiation, Winchester and NERC-Lasi Vegas.
The programs for water, pesticides/ and radiation have been
operational for several years/ and we estimate that they
are providing approximately 50 percent of the support
necessary in their area of monitoring. The program in air
was just established in January of this year and is the
most critical area. A major deficiency of these programs
is that they are almost entirely voluntary, i.e. EPA
monitoring activities are not required to participate.
The Regional quality control programs are lacking in most
areas and are not properly supported.
-------
11
Implementation Plan
A quality control program can succeed only if it is
given authority and made mandatory for all areas supplying
environmental data. The plan calls for central authority
and for adherence to prescribed guidelines by the operating
monitoring programs. It recommends that a uniform
control is implemented over all EPA data and that it must
be carried out by the operating programs. Although the
program (resources) is limited to EPA, it must in the
future be extended to include State and local agencies
and contracting laboratories.
The organizational elements primarily involved in
this program are the Office of Monitoring, quality control
groups at the NERC's and the Surveillance and Analysis
Divisions in the Regional Offices. Other programs with
monitoring interests will primarily have review
responsibilities.
Nine specific project areas have been identified in
this plan which are necessary for achieving the goals of
the quality control program. These are:
Preparation and Issuance of Procedures
and Guideline Manuals
Repository of Standard Reference Material
(SRM) and Samples (SRS)
Operation of Intra-laboratory Quality Control
- Operation of Inter-laboratory Quality Control
Evaluation of Instruments both for
Laboratory and Field Use
Quality Control - Site and Laboratory
Evaluation
Development and Management of Certification
Program
- Communication and Coordination
- Quality Control - Training.
-------
Ill
The plan calls for immediate action, and projects that the
total quality control activity will be, to a large extent,
operational by July 1973. Completion dates for the various
tasks have been established and they are consistent with
this goal.
Resouj:ce_
The proposed quality control program will require al-
location of resources for three organizational elements:
- Office of Monitoring to direct and.
coordinate the program
- NERC's to implement and support the
program
- Regions to implement the program.
Not all of the resources needed for the quality control
program are new requirements.
-------
INTRODUCTION
Measurements are made daily of pollutants present in
the environment and being discharged from a multitude of
sources with the resultant data being used in all facets
of pollution control. These data are used in investigating
the effects of pollution on human health, in determining
the types of environmental quality standards that should be
promulgated, and in providing evidence for enforcement
actions. The measurements are made by numerous agencies
and private organizations at a large number of field stations
and laboratories. Much of the data are stored in the
Environmental Protection Agency's (EPA) data banks where
it is used to establish nation-wide appraisals of environ-
mental quality, and subsequently, to aid in specific
decision-making processes. Regardless of the end uses of
the data, it is necessary that the accuracy and validity of
the data be known in order that the actions taken can be
technically supported.
The primary goals of the quality control program are
to improve and document the accuracy and validity of environ-
mental monitoring measurements. In order to achieve these
goals, quality control must be imposed on nearly all phases
of monitoring activitiesfrom sampling site selection to
data formatting. Quality control should include personnel,
methods selection, equipment, and data handling procedures.
-------
2
Quality control applies not only to these components indi-
vidually, but also to their interrelationships and
functioning as a total system.
The benefits derived from an Agency-wide quality con-
trol program will amply justify the resources expended. In
addition to assuring that the accuracy and validity of the
data can be verified, an active quality control program,
through the use of uniform procedures, will enhance the
utilization of personnel, facilities, and equipment. The
data gathered by the various laboratories and field stations
will become comparable. Once accepted at the Federal and
hopefully the non-Federal level, the promulgation of instru-
ment specifications will undoubtedly improve the quality
of instruments, laboratory apparatus, and commercially pro-
duced reagents. Furthermore, the implementation of a qual-
ity control program will increase the overall credibility of
the Agency, thereby, strengthening its means for enforcement.
A quality control program is a continuing activity.
The program must be flexible in responding to changes in
monitoring requirements for the various media.
Quality control practices are inherent in any monitor-
ing or laboratory operation--personnel must be trained,
instruments must be calibrated, and uniform operating proce-
dures must be developed and followed. Thus, the structuring
-------
3
of an Agency-wide quality control program does not imply
that all of the activities to be undertaken are new and
require new resources. The program implies that activities
must be identified and expanded, and if necessary, developed
into an Agency-wide quality control framework which is
given emphasis and authority for supporting the Agency's
requirements.
This report presents an implementation plan for an
Agency-wide quality control program. Because of the urgent
need to upgrade the quality of EPA's monitoring activities,
the plan calls for immediate action and projects that the
total quality control activity will be, to a large extent,
operational by July 1974. In addition, sections are
included that describe the various elements of a quality
control program and summarize the existing quality control
activities.
-------
4
ELEMENTS OF A QUALITY ASSURANCE PROGRAM
A total quality control program is composed of four
major elements:
Development and issuance of procedures.
Intra-laboratory quality control program.
Inter-laboratory quality control program.
Monitoring program evaluation and
certification.
All these elements are equally essential to a successful
quality control program and must be developed and carried
out simultaneously. This section of the report describes
each element and its subelements (Table 1) necessary to
provide a basis for understanding the proposed plan for a
total quality assurance program.
Development and Issuance of Procedures
A basic requirement of a quality control program is a
series of manuals describing the procedures to be followed
during the course of sampling, analysis, and data handling.
It is the use of such prescribed procedures that provide a
uniform approach in the various monitoring programs and
which allow the evaluation of the validity of data produced.
Procedures are needed for all of the facets of a monitoring
programfrom criteria to be used in locating stations to
the formats to be used in reporting the data. The required
procedures may be grouped into three categoriessampling,
-------
TABLE I
Elements of Quality Control
DEVELOPMENT AND ISSUANCE OF PROCEDURES
Sampling Procedures
Site selection criteria
Station design criteria
Sampling criteria
Calibration procedures
Methods Selection Procedures
Methods selection criteria
Compendium of recommended methods
Equivalency determination
Laboratory Procedures
INTRA-LABORATORY QUALITY CONTROL
Staffing control
Training programs
Use of correct procedures
Performance evaluation
INTER-LABORATORY QUALITY CONTROL
Cross-check sample program
Methods evaluation studies
MONITORING PROGRAM EVALUATION AND CERTIFICATION
Periodic review/evaluation
Certification
-------
6
methods selection, and laboratory procedures. A brief
description of each of these follows:
Sampling Procedures
(a) Site selection The selection of sampling/
monitoring sites is the responsibility of each monitoring
program, however, guidelines must be established that govern
the specific placement of monitors or the exact location
where the sample is to be taken. Such rules or guidelines
are necessary to ensure that the measurements made or the
samples taken are representative and comparable. For example,
in the case of air pollution monitoring, criteria must be
established specifying the allowable nearness of inlet
probes to buildings or, in the case of water monitoring, the
depth at which samples are taken.
(b) Station or instrument Measurements may be
severely affected by the type and configuration of the facil-
ities used in collecting the sample. For example, in some
situations, variability in voltage, temperature, and humidity
can influence the measurements. Consequently, operational
parameters must be specified and controlled. Similarly,
design characteristics must specify the types of monitors
and special equipment which must be accommodated. Adherence
to station design criteria will provide optimum use of
-------
7
equipment, ease of operation, minimum maintenance, and
reduced data losses.
(c) Sampling and preservation criteria --
Procedures must be established that govern the manner in
which samples are collected and handled. These procedures
should include the following:
(1) Use of equipment and materials for
collecting, preserving, and transporting the samples.
(2) The length of sampling periods.
(3) The types of accompanying information
needed.
In order that meaningful data are obtained, a sound statis-
tical basis for determining the frequency and duration of
sampling/monitoring must be used in the program design.
Such procedures must be provided in the manuals. Specific
procedures must also be established that document the chain-
of-custody of samples needed for enforcement actions and
for samples taken at or near points of suspected violations.
(d) Calibration procedures To ensure that data
generated by automatic or integrating field sampler-analyzers
are valid, procedures for routine field calibration of
these instruments must be specified. The calibration fre-
quency, as well as the procedures used, should be identified
and described.
-------
8
Method Selection Procedures
(a) Compendium of recommended methods Many
different methods are available for measuring pollutants in
environmental media. Some methods have been promulgated
by groups within EPA. Other methods have been published
by groups such as the American Public Health Association
(water), the American Society for Testing and Materials
(air/water), and the Intersociety Committee (air). The
Environmental Protection Agency has the statutory responsi-
bility to promulgate environmental, source, and effluent
standards which includes a method of collection and
analysis. To ensure the use of standard, reference, or
equivalent methods throughout the Agency, an approved com-
pendium of standard, reference, or equivalent methods must
be developed for all pollutants in all media and published
for all monitoring activities.
Method selection procedures based
on standardization activities -
The cornerstone of any quality
control program is the uniform
use of acceptable methods and
procedures. The need for stand-
ard methods of sampling, analysis,
and data handling is readily
apparent, and the use of such
methods between cooperating lab-
oratories removes methodology as
a variable in the comparison or
joint use of data. The use of
standard or equivalent methods is
particularly important when
-------
9
laboratories are providing data
to central data banks such as
STORET and NADIS. The routine
use of non-standard or non-
approved methods within an
agency raises serious doubts as
to the validity of the data
reported.
Methods selected must be pre-
sented in a concise, easy-to-
follow format. The format will
indicate: principle, appli-
cability, range, sensitivity,
interferences, accuracy and
precision, apparatus, calibra-
tions , reagents, procedures for
sampling and analysis, calcula-
tions, references or bibli-
ographies, and special
precautions necessary for
handling and/or disposing of
hazardous materials and reagents.
Based on past experience,
generally applicable criteria
for the selection of acceptable
methods for environmental
monitoring should be recommended
by a panel of experts represent-
ing the ultimate users. The
measurement technique will
exhibit sufficient precision
and accuracy to meet the data
needs of EPA. Furthermore, the
necessary equipment should be
available at many laboratories
and be adequate for the routine
examination of a large number of
samples. In addition, the
validity of the technique should
be tested.
(b) Method selection criteria All EPA labora-
tories will be encouraged to utilize the same methods for
measuring pollutants. The methods which may be selected
-------
10
will be described in an EPA-approved compendium. Where
more than one method is available, the selection criteria
will be based on the applicability of the method, its
comparability with methods used in other laboratories, and
its ability to meet requirements of the data user.
Specific guidelines for selection of alternative methods
and procedures should be included in all manuals.
(c) Equivalency determination criteria
Individual monitoring activities may decide to utilize
methods that differ from the prescribed EPA reference
method. It is necessary to establish guidelines to ascertain
that the results of the particular monitoring method
employed is equivalent to EPA-approved methods and, hence,
valid data can be obtained.
Laboratory Procedures
Bias and variability in data may result from
variation in the general design and construction of facili-
ties. Differences in the equipment, apparatus, and reagents
used by laboratories can also contribute to both the bias
and variability of data which decreases the reliability
and comparability of results between laboratories. The
functional design of a laboratorythe use and maintenance
of equipment, standardization of reagents, temperature, and
humidityinfluences the reliability of data. Therefore,
-------
11
it is extremely important that quality control procedures
concerning standard laboratory requirements and practices
be developed, promulgated, and followed. These guide-
lines should cover all facets of routine laboratory
operations and maintenance of equipment and apparatus.
Intra-laboratory Quality Control
Quality Measurement and Performance Evaluation
To maintain a high level of competence in daily
activities, quality control must be implemented in the
field and at the bench using a system of checks to determine
the accuracy and precision of results and the performance of
monitors and analysts. Intra-laboratory quality control
is a continuing in-house activity to ensure the output of
valid data. The specific objectives of the program are
to devise and implement procedures that:
Measure and control the precision
of procedures and instruments.
Measure and control the accuracy
of analytical results.
Ensure data output is computer
compatible.
Present data in proper format.
Document performance of instru-
ments and analysts.
Document training needs.
Identify weak methodology and
consequently research needs.
-------
12
An intra-laboratory quality control program
employs several important tools/techniques:
Standard Reference Materials (SRM)
are substances which qualify as
absolute quantities against which
other like substances can be
calibrated. The SRM, typically
produced by organizations like
the National Bureau of Standards,
is used to prepare standard
reference samples (SRS) for lab-
oratory application.
Standard Reference Samples (SRS)
are preparations or known amounts
of standard reference materials
added to an actual environmental
sample which has been previously
analyzed. The amount of the sub-
stance found in the sample is a
"true" indication of the accuracy
of the method for a given measure-
ment. The use of the standard
reference samples measures the
extent of interferences which
cannot be obviated.
Quality control charts should be
an integral part of a quality
control program. Figure 1, which
shows the summation of the dif-
ferences squared for replicate
samples, is one form of a quality
control showing trends.
Additional control charts
are recommended where standard
deviations are (d = V«/«7£ ) for use
on a daily basis to establish
rapidly if an analysis is out
of control on a given day. Once
precision and accuracy data are
available on the method and the
analyst (from the use of SRM/SRS),
systematic daily checks are
necessary to ensure that valid
-------
13
QUALITY CONTROL CHARTS
= The summation of the difference squared for replicate samples,
NO APPARENT TREND
lid2)
SAMPLE SET NO.
ANALYSIS IN CONTROL
Kd2)
SAMPLE SET NO.
ANALYSIS OUT OF CONTROL
UPPER LIMIT
APPARENT TREND EVIDENT
ltd2)
SAMPLE SET NO.
ANALYSIS OUT OF CONTROL
LOWER LIMIT
I(d2)
SAMPLE SET NO.
ANALYSIS OUT OF CONTROL
UPPER LIMIT
FIG. 1
-------
14
data are being generated. From
these daily precision and accuracy
data, quality control charts can
be constructed and maintained
to determine when the method used
is producing valid data, when the
data are questionable, or when a
trend is detected which must be
investigated and corrected.
Several techniques are available
for constructing quality control
charts and plotting subsequent
data. The two techniques currently
used by EPA are the Shewhart
technique and the Cumulative-
Summation technique. For both
techniques, precision control
charts are constructed from
duplicate sample analyses, and
accuracy control charts are con-
structed from spiked sample
analysis.
Personnel Staffing and Training
An important part of intra-laboratory quality
control is proper staffing and the continuous training of
personnel. Guidelines need to be established and made
available to all laboratories concerning the qualifications
(training and experience) that are necessary to perform
the various tasks. For example, the operation of a gas
chromatograph will typically require a professional
scientist with considerable experience. On the other hand,
many routine functions can be adequately performed by
entrance grade technicians. Staffing guidelines will assist
the laboratories in selecting and deploying personnel.
-------
15
The rapidly changing methodology of environmental monitor-
ing requires that the personnel undergo periodic training
both formal and on-the-job. Training courses in method-
ology as well as quality control practices will need to
be incorporated into EPA's formal training program and some
requirements will need to be established for all monitoring
personnel to participate.
Inter-laboratory Quality Control
An inter-laboratory quality control program serves to
select and evaluate methods, characterize their precision
and accuracy, and provide data for evaluating both lab-
oratory and analyst performance. This aspect of quality
control is referred to as cross-check sample studies or
methods evaluation studies. Specific objectives of this
program are to:
Measure the precision of reproducibility
of methods of analysis within various
programs.
Identify interference in different
sampling environments.
Measure the precision and accuracy of
results between laboratories.
Provide a mechanism for evaluation and/or
certification of laboratories and
analysts.
Detect weak, improper, or impractical
methodology.
-------
16
Detect training needs and upgrade
laboratory performance.
Assist laboratories or programs in
obtaining new resources.
Participating laboratories should be provided with
standard reference samples, instructions, and data forms
necessary to test methods under certain prescribed condi-
tions. The results should be submitted to a coordinating
laboratory where they are statistically evaluated to
determine the accuracy and precision of the method. More-
over, they are also evaluated to determine the general
applicability of the method and to rate the performance
of the laboratories and analysts.
This type of activity involving laboratories in evalua-
ting methods is necessary to provide a sound statistical
model. Single laboratory tests introduce both method and
laboratory bias, and the results obtained may have little
relationship to the "true" reliability. All operating
quality control programs have used an inter-laboratory quality
control program as a mechanism for methods standardization
and selection.
Management
Certain aspects of implementing and maintaining a
quality control program require actions and decisions which
affect total operation or reflect position/policy of the
-------
17
Agency. These include program evaluation, certification,
and communications.
Program evaluation Involves periodic review
and assessment of all quality control activities as
necessary to determine their proper functioning, effective-
ness, and reliability. Evaluations are necessary at least
quarterly and should be comprehensive enough to include
all field and laboratory activities related to monitoring.
Special attention should be given to field procedures and
calibration, performance of laboratories and analysts, and
adequacy of manuals/methods, training, etc. This evaluation
should provide the basis for a quarterly quality control
status/needs report to Regional Administrators, Center
Directors, and Headquarter's elements.
Certification This is an important subelement
of the management activity which involves formal approval
and/or endorsement of acceptable performance by a laboratory
or analyst. Certification is directly relatable to the
inter-laboratory quality control. Certification implies
that performance of analysts and laboratories is acceptable
with respect to an established population average. It is
important to note that certification does, in both fact and
concept, imply endorsement by EPA and, as such, can have
serious impact/repercussions on our enforcement actions and
-------
18
our contracting systems. Since EPA's experience in con-
ducting certification programs is limited/ it is necessary
to review other programs existing outside EPA and select
the best available information to develop guidelines for
EPA. Certification programs may first be applied to EPA
laboratories and then expanded to include States and
others on a voluntary basis.
A reasonable and valid alternate approach to a
certification program would be to. require all laboratory/
field activities supplying monitoring data to EPA to oper-
ate within certain prescribed guidelines for intra-laboratory
quality control and participate in EPA's inter-laboratory
quality control program. This alternative will provide
documented evidence of laboratory/analyst performance and,
hence, the validity of the data.
Communication For a quality control program to
succeed, a necessary subelement is an adequate information
exchange program. The following types of activities are
included:
Reports from meetings of a
quality assurance committee.
Reports from central files
of information on request/
use of SRM and SRS.
Reports of quality control
conference.
Quality Control Newsletter.
-------
19
CURRENT STATUS AND NEEDS
Many elements of quality control are currently being
practiced to some degree at EPA and other environmental
monitoring programs. The quality control programs for the
different media and categorical programs are quite varied
ranging from rather comprehensive activities in some areas
to essentially non-existent activities in others. What-
ever the case, all of the on-going activities suffer from
the lack of established requirements resulting in essen-
tially voluntary and partial participation by individual
programs and laboratories. The problem is further com-
pounded by the decentralization of monitoring activities.
The effort thus far, then, has been uncoordinated,
fragmented, and operated with minimal resources. The results
have been unsatisfactory. Much of the data have been of
invalid or questionable quality. For example, more than 20
percent of the air quality data gathered from the State and
local agencies for insertion into the National Air Quality
Data Bank was not useable. In addition to loss of data,
this represents considerable waste of both manpower and
resources. Since different methods, equipment, or proce-
dures have been used to monitor separate locations, various
programs have produced data that are not comparable. In
many instances, analyses are being made without proper
-------
20
reagents, without calibration, and by unqualified analysts.
This has resulted, for the most part, in a data base that
is of questionable validity and, therefore, limited utility.
In developing a plan, it is necessary to assess the present
status of our quality control programs, evaluate their
needs, and provide solutions.
Each of the current programs in quality control has a
record of accomplishments and capabilities as well as cer-
tain shortcomings. Although quality control programs are
under way for air, water, radiation, and pesticide monitor-
ing, they all are notably lacking in several areas. The
most critical deficiencies are lack of established procedures
for sampling methodology, field calibration, equivalency
determination, and the lack of performance evaluation and
certification. Comparable quality control programs for noise
or solid waste monitoring are non-existent. The current
capabilities for each media's quality control program are
the subject of Figure 2. While the estimated degrees of
accomplishment are subjective, they indicate relative
strengths of the current programs. It is estimated that
the quality control programs for water, pesticide, and radia-
tion monitoring are about 50 percent complete. However,
the quality control activity for.air monitoring which has
just begun is inadequate. The major accomplishments and
-------
STATUS OF EXISTING QUALITY CONTROL ACTIVITIES JUNE'72
(AS PERCENT TOWARD FULL ACTIVITY)
MEDIA
0%
50%
100%
AIR
WATER
PESTICIDES
SOLID WASTES
RADIATION
NOISE
FIG. 2
-------
22
shortcomings of each quality control program are high-
lighted below.
Water The quality control activities for
monitoring water and wastewater are centered at the Analyt-
ical Quality Control Laboratory (AQCL) at the National
Environmental Research Center (NERC) in Cincinnati. This
program has been in existence for more than five years.
However, the limited resources available have not allowed
for the development of a comprehensive program. The major
accomplishments and deficiencies of the water quality
control program are:
Accomplishments
(1) Compendium of Methods (latest
issuance in 1971)
(2) Intra-laboratory Quality Control
Manual (1971)
(3) Inter-laboratory Quality Control
(maintenance of SRS repository
for cross-check sample studies
and methods evaluations)
(4) Laboratory procedures guide-
lines (1971)
(5) Quality Control Newsletter
(issued periodically)
Deficiencies
(1) The methods manual does not
include instrumental, bio-
logical, and microbiological
measurements (external
methods exist)
-------
23
(2) Technical assistance to field
laboratories is provided on
an ad hoc basis.
(3) The sources of SRM are not well
established.
(4) The production of SRS fall
short of demand.
(5) Reports of methods evaluated
are not timely.
(6) Laboratory manual needs to be
updated.
(7) The scope of the present
program is not geared to meet
the expanding needs of the
Regions.
(8) Participation by laboratories
is voluntary.
In addition to the activities of the Analytical
Quality Control Laboratory, the Water Supply Research
Laboratory, NERC-Cincinnati, conducts limited quality con-
trol services for State laboratories concerned with
drinking water standards. This laboratory, started in
1953 as part of a training program, supplies reference sam-
ples and uses and recommends use of the 13th Edition of
Standard Methods for Water/Wastewater Analyses. In
addition, it prepares a training manual which includes
some quality control procedures. Moreover, the Water Supply
Research Laboratory participates in a certification program
for State bacteriological and chemistry laboratories.
-------
24
Additionally, Consolidated Laboratory Services,
NERC-Corvallis, has conducted an intra-laboratory quality
control program for water quality data since its inception
and has improved to the point of being a near model
system. This program has recently extended this activity
to its satellite laboratories, and NERC-Corvallis requires
that quality control be practiced by all contractors
assisting it in acquisition of water quality data. This
Laboratory has also developed a Sampling Handling and
Verification System (SHAVE) which assures continued integ-
rity of the sample from collection to analysis to data
reporting. A simplified description is portrayed in Figure
3. It should be noted that NERC-Corvallis works closely
with AQCL-Cincinnati by using its methods, supporting
lake eutrophication studies, and participating in other
special projects.
Pesticides The Primate and Pesticide Effects
Laboratory under NERC-RTP conducts an inter-laboratory
quality control program for pesticides monitoring. This
program is almost totally dedicated to supporting the Com-
munity Pesticides Studies and the State Services Labora-
tories. Under this program, quality control services are
available to 18 laboratories which are under contract to
EPA.
-------
25
SAMPLE HANDLING AND VERIFICATION SYSTEM
SOURCE
COLLECTION
CODE
LOG
ANALYSIS
DATA
(VERIFICATION)
STORET
USERS
FIG. 3
-------
26
The complexity of pesticide analysis requires
extensive quality control activities. Hence, the scope of
the services provided by the Primate and Pesticide Effects
Laboratory is extensive. Standard reference materials
and samples and standardized materials for gas chroma-
tographic columns are provided as well as technical assist-
ance, training, repair, and maintenance of both gas
chromatographs and mass spectrometers. An inter-agency
agreement (December 22, 1971) established a repository of
standard reference materials from which the Food and Drug
Administration (FDA) and State and local enforcement
authorities may draw standards. The major accomplishments
and deficiencies of the program are highlighted below.
Accomplishments
(1) Compendium of Methods
(2) Inter-laboratory quality control
(3) Intra-laboratory quality control
(4) Methods evaluation program
(5) Technical assistance and training
(6) Standard Reference Materials
Deficiencies
(1) Methods are not recommended for
enforcement actions and are
not official EPA methods.
-------
27
(2) The cross-check sample program
is inadequate to meet needs
of the Regions.
(3) No quality control manuals are
available.
(4) The methods evaluation program
should be expanded to Regional
participation.
(5) Technical assistance and train-
ing must be expanded and
redirected to include Regional
needs.
(6) Standard reference materials
are not analyzed and
maintained.
Air A formal quality control program for air
monitoring activities was initiated in January 1972. It is
located in the Quality Assurance and Environmental Monitor-
ing Laboratory, NERC-RTP. The slow start in this area is
due primarily to lack of resources and from difficulties
encountered in developing standard reference materials for
gases as well as delivery systems to generate standard
reference samples. Although some of these initial diffi-
culties have now been resolved, and a quality control program
is under way, much remains to be done. The implementation
of quality control for air monitoring is the most critical
area. The current status of this program is briefly
described below.
-------
28
Accomplishments
(1) Reference methods have been
promulgated with Air Quality
and Source Performance
Standards.
(2) Standard reference materials
have been developed for
sulfur dioxide, nitrogen
dioxide, carbon monoxide,
and some hydrocarbons.
(3) Methods for suspended particulates,
sulfur dioxide, and carbon
monoxide have been evaluated
and tested.
Deficiencies
(1) No compendium of EPA-approved
methods has been developed.
(2) No delivery systems have been
developed to generate
standard reference samples.
(3) No quality control manuals or
guidelines have been
developed.
(4) No cross-check sample program
has been initiated.
(5) Lack of standard gas mixtures.
(6) Lack of source and emission test
procedures.
Radiation The Analytical Quality Control Service
of the Office of Radiation Programs located at Winchester,
Massachusetts, which had been providing an inter-laboratory
quality control program for radiation measurements since
-------
29
1962, has been relocated to NERC-Las Vegas. It is currently
known as the Office of Quality Assurance-Radiation. The
present program consists of conducting cross-check sample
studies for EPA contract laboratories and utility companies.
Standard reference samples for radionuclides are provided
to other Government agencies and private organizations as
requested. The Office of Quality Assurance-Radiation col-
laborates with groups such as the American Society for
Testing and Materials, the American Public Health Association,
and the Association of Official Analytical Chemists in test-
ing radiochemical methods of analysis. Technical assistance
in developing quality control programs is provided to EPA
and other laboratories on an ad hoc basis.
In addition to developing an operational intra-
laboratory quality control program, this program conducts
quality control activities with States, the International
Atomic Energy Agency, the World Health Organization, and the
U.S. Atomic Energy Commission laboratories. A functional
system for collecting, handling, and analyzing large numbers
of samples has been developed. Moreover, data processing
and reporting have been computerized. The computer program
generates quality control charts of measurement accuracy and
precision. The program also conducts a field sampling course
and provides for the certification of participants.
-------
30
The current status of this activity is briefly
summarized below.
Accomplishments
(1) A compendium of radiochemical
methods of analysis has been
published.
(2) A cross-check sample studies
program has been developed.
(3) A methods evaluation program
has been developed.
Deficiencies
(1) A compendium of EPA-approved
methods has not been
promulgated.
(2) A quality control manual has
not been developed.
(3) Training and technical assist-
ance to support Regional
needs is inadequate.
(4) Methods evaluation studies are
not comprehensive.
-------
31
IMPLEMENTATI ON
In the development of an Agency-wide quality control
strategy, two overriding issues must be considered and
resolved:
What should be the scope of coverage?
(EPA only or include State and local
and other programs)
What should be the extent of control?
Environmental monitoring data are produced by many
organizations, both governmental and private. The Environ-
mental Protection Agency, State and local agencies, and
other Federal agencies are involved in environmental moni-
toring. The State and local efforts are especially exten-
sive and the data are routinely placed into EPA's data
systems. In the private sector, contracting laboratories
and private industry are involved in the measurement and
analysis of pollution.
Because of EPA's reliance on outside organizations for
the production of environmental data, the quality assurance
program should extend outside EPA. Only in this manner
can EPA support the data it uses and, consequently, the
actions that are taken. Therefore, at a minimum, the qual-
ity assurance program should embrace EPA's own programs,
the State and local laboratories, and the contracting
laboratories conducting work for EPA. The quality control
-------
32
strategy presented herein is based upon the development of
the tools and procedures for a quality control program
which can easily be extended from a regional base to
State, local, and contract laboratories.
A quality control program is doomed to failure if it
is not given authority and is not adhered to by the organ-
izational elements that it encompasses. Usefulness of the
procedures, manuals, and training programs will be limited
if the operating programs are not required to incorporate
them into their monitoring activities. The successful
implementation of the quality assurance program has two
requisites. First, the program should be emphasized so
that it becomes widely accepted. Secondly, the program
should have sufficient authority to support the control
activity necessary to produce valid data.
This section of the report presents an organizational
plan and a listing of projects, tasks, and schedules.
Organizational Plan
(1) The program must be authoritative/
compulsory.
(2) The program must be centrally directed
and coordinated, yet allow for individual program
implementation.
-------
33
(3) At least initially, the program must
utilize existing facilities and areas of competence.
(4) The program must provide for centrali-
zation of all elements of the program that lend themselves
to such organization, as in provision of SRM and prepara-
tion of SRS.
(5) The program must provide for routine
exchange of quality control information.
(6) The program must be responsive to all
operating monitoring programs.
(7) The program must extend outside EPA,
both in terms of building the program and providing service.
Accordingly, four levels of activities and respon-
sibilities are necessary:
(1) Program Management, planning, and
coordination -- In order to ensure the development of a
uniform quality control program that meets the needs of all
monitoring activities, it is necessary to centralize
general management, planning, and coordination. In this way,
each program receives equitable emphasis and is evaluated
by a single standard.
(2) Program review and concurrence Since
monitoring activities within EPA are carried out independ-
ently by the various organizational elements (i.e., Regions,
-------
34
Program Offices, Enforcement, and Research), the quality
control program should be developed and operated in concert
with these offices. The responsibilities of each monitor-
ing activity, therefore, is to identify its needs and
review and concur on proposed Agency-wide quality control
programs.
(3) Technical procedures, development, and
technical support A sound quality control program can
only be developed and operated if the field programs are
supported by central quality control groups. It is these
groups, located at the NERC's, that will assist in program
development, develop and test the procedures before routine
field application, provide necessary materials, and render
assistance as needed.
(4) Program implementation and operation
Quality control must be implemented by each of the operating
monitoring programs within the guidelines and directives
provided. This implies that monitoring programs in each of
the Regions and those in Program Offices, Research, and
Enforcement will conduct their own quality control programs.
To ensure that these programs operate within central
directives, close liaison must be maintained between indi-
dual monitoring activities and the central quality
assurance activity.
-------
35
Specific program functions and responsibilities have
been determined. They are designed to provide the Agency
with an effective quality control program for all moni-
toring activities. The proposed program takes advantage
of currently available expertise in order to implement a
quality control program responsive to the Agency's needs.
Office of Monitoring; The Office of Monitoring
will assume overall responsibility for the direction and
coordination of both the development and implementation of
the quality control program. It will continually review and
assess the program's progress and efficacy. These respon-
sibilities will be carried out in concert with the appro-
priate elements of the NERC's. Specific tasks require the
Office of Monitoring to:
Issue directives and guidelines
for quality control to operating
monitoring programs.
Prepare and review program plans.
Ensure that .technical assistance
is provided to Regions and other
monitoring programs.
Develop and direct a laboratory
certification program.
Evaluate the effectiveness of
continuing quality control
programs and provide biannual
status reports.
Coordinate requisite training
programs.
-------
36
Maintain a rapid information
exchange system that will
notify appropriate organiza-
tions of important changes
and events relating to
quality control.
Establish an internal quality
assurance committee necessary
for program coordination.
Provide a central focal point
within EPA on all matters
pertaining to quality control
of monitoring activities.
National Environmental Research Centers; The
technical support elements of the quality control program
will be located at the National Environmental Research
Centers. The quality control groups at these Centers will
be responsible for providing the necessary tools for
operating monitoring programs as required. Responsibilities
for particular environmental media and categories are
specified as follows:
Air NERC-RTP
Pesticides NERC-RTP (Primate and
Pesticide Effects
Laboratory)
Radiation NERC-Las Vegas
Water NERC-Cincinnati
Solid Waste and Noise responsibilities
for the quality control activities
have not yet been assigned.
-------
37
Responsibilities of these quality control groups
will include the following tasks:
Develop and provide standard
reference materials/samples.
Direct and coordinate intra-
and inter-laboratory quality
control programs.
Develop quality control
procedures.
Provide technical assistance
in quality control.
Regions: Each Regional Office will implement the
quality control program for all appropriate monitoring
activities through an officially designated full-time
quality control coordinator. Specific Regional tasks
include:
Centralization of Regional
authority for quality control
activities.
Operation of intra-laboratory
quality control programs to
assure adherence to EPA
procedures and guidelines for
all monitoring activities.
Provision of technical assist-
ance to State and local agencies.
« Participation in methods
selection and in cross-check
sample studies.
The review of each program to
provide a quarterly status/
needs report to the Office of
Monitoring.
-------
38
Participation in the develop-
ment and implementation of
quality control training
activities.
Provision of technical support
and assistance as required for
certification programs.
Office of Categorical Programs; Develop require-
ments and review quality control plans for the monitoring
of hazardous materials, radiation, pesticides, solid waste,
and noise.
Office of Air and Water Programs: Develop require-
ments and review plans for quality control activities of
air and water monitoring, both ambient and source. The
Office of Air and Water Programs will be responsible for the
general management of data acquisition and storage and will
review and concur in this aspect of the quality control
program.
Office of Enforcement and General Counsel; Review
and concur in all quality control activities which support
enforcement and case preparation. The Office of Enforcement
and General Counsel will establish case preparation monitor-
ing guidelines and provide manuals to appropriate program
managers and operational elements.
Office for Regional Liaison; Review and concur in
all quality control activities which relate to Regional
responsibilities. The Office for Regional Liaison will also
-------
39
actively participate in all communication and coordination
activities.
Projects and Schedules
The implementation of an Agency-wide quality
control program will require the accomplishment of a series
of projects. In total, nine separate projects with numerous
tasks have been identified. The accomplishment of these
projects and tasks will require the efforts of many EPA
organizational elementsmost notably the Regional Surveil-
lance and Analysis Divisions, the quality control groups at
the NERC's, and the Office of Monitoring. It must be
remembered, however, that each monitoring program will need
to expend resources for quality control.
A brief description of each project, including
the objectives they fulfill, is included below-
Project 1: Preparation and Issuance of
Procedures and Guidelines Manuals The result of this pro-
ject will be "loose-leaf" manuals that will include uniform
procedures for the entire monitoring activity. The proce-
dures are to be used by all environmental monitoring activi-
ties within and outside EPA. The project calls for continual
expansion and updating of these manuals as more and better
information becomes available. Primarily, the manuals and
inserts will be developed by the quality control groups at
-------
40
the NERC's. In some instances, the issuance of procedures
and guidelines may involve only review and compilation of
existing materials. For others, it may require extensive
surveys of actual field and laboratory investigations.
Project 2: Provide a Repository of Standard
Reference Materials and Samples The result of this pro-
ject will be stockpiles of standard reference materials and
samples for testing and evaluating all aspects of monitor-
ing. This will include ambient, emission, and effluent
monitoring as well as fuel additive and pesticide registra-
tion. The tasks to be accomplished include identification
of sources of standard reference materials and development
of standard reference samples including appropriate delivery
systems. In some cases, such as hazardous materials, special
procedures will be developed for handling, storage, and use.
Whenever possible, standard reference materials will be
obtained from outside organizations, such as the National
Bureau of Standards, and will be produced internally only
when not available elsewhere.
Project 3: Operation of Intra-laboratory
Quality Control This is a continuing project which will
be carried out by all on-going monitoring programs. This
effort includes resources spent calibrating instrumentation
and equipment, conducting laboratory performance checks,
-------
41
maintaining daily control charts, etc. This project also
includes technical assistance rendered to other organiza-
tions to improve or upgrade their programs.
Project 4; Operation of Inter-laboratory
Quality Control This project is a continuing activity
involving the quality control centers and the operating
monitoring programs. Its results will be documentation of
laboratory, method, and analyst performance. Performance
will be continually evaluated by use of computerized control
charts which will show performance levels and data quality.
Project 5; Laboratory and Field Evaluation
of Instruments -- The result of this project will be a series
of reports and papers that will provide a scientific basis
for selection of pollution monitors. For the most part,
this involves review of operational data from monitors which
are currently in the field or are used in laboratories. The
data will be evaluated and organized into a user performance
report which will be made available to all prospective users.
This activity will be primarily carried on by the NERC's.
Project 6; Quality Control - Site and
Laboratory Evaluation -- This project provides external checks
on the performance of on-going monitoring activities. One
task is the periodic evaluation of EPA's monitoring programs
by the quality control groups of the NERC's and the Office of
-------
42
Monitoring personnel. A second task is the evaluation of
State and local monitoring programs by the quality control
elements of the Surveillance and Analysis Divisions. These
checks may range from site inspections and review to actual
field calibrations using mobile units.
Project 7; Development and Management of
a Certification Program _The result of this project will be
to establish and maintain qualified laboratories and analysts
for EPA and other monitoring activities. Certification
programs will be the culmination of all the quality control
activities. Performance levels will be established against
which the performance of each laboratory will be judged, and
upon approval, certificates will be issued and renewed
periodically. The actual mechanisms to be followed and the
extent of the programs are yet to be investigated. Alterna-
tives must be weighed as to whether the program should be
mandatory and whether it should be extended to external lab-
oratories. The frequency with which the program should be
updated must also be considered.
Project 8: Communication and Coordination
This project provides the overall management of the quality
control program and ensures continuity of all operations.
Some of the tasks under this project are concerned with
information exchange. The newsletter, the information system,
-------
43
and the conferences will provide a mechanism whereby the
people engaged in quality control can share their experiences
and resolve problems. Quality control activities within EPA
must depend upon the expertise of outside organizations. A
task under this project will ensure that the necessary con-
tacts are maintained. The development of a successful
quality control program involves numerous policy and major
technical decisions. It is proposed under this project that
an internal quality assurance committee be established.
Project 9; Quality Control - Training This
project will serve to coordinate and develop training courses
and lectures concerning the use of approved methods and
quality control practices. The intent of this project will
be to work with the formal EPA training programs to ensure
that the proper courses are developed and given. No sepa-
rate training program is to be developed. In order that all
monitoring professionals and technicians may be accommodated,
an attempt will be made to make the requisite courses
available through selected colleges or junior colleges.
Courses on methodology and quality control practices might
be incorporated into the chemistry programs of selected
schools (primarily evening schools) via the grant mechanism.
An alternative is to provide on-the-job training at the
Regional or NERC laboratories. An important aspect of this
-------
44
project will be to develop mechanisms for ensuring that all
appropriate monitoring personnel take advantage of these
courses.
-------
45
QUALITY CONTROL PROGRAM FOR
STATE AND LOCAL AGENCIES
INTRODUCTION
In June of this year, the Office of Monitoring pre-
pared and distributed for review a proposal for an Agency-
wide quality control program. Comments have now been
received from most of the Program and Regional Offices and
their suggestions have been incorporated into the plan.
The plan discusses and sets forth the organizational strat-
egy to be employed and delineates the projects and tasks
that must be done. A further analysis is required to focus
on the scope of coverage and the extent of control that the
EPA quality control program should have. This paper
addresses these points and makes recommendations regarding
these issues. Thus, the "plan," together with this issue
paper, constitutes the strategy for an Agency-wide quality
control program.
The primary issue is whether the quality control pro-
gram should be limited to the internal EPA monitoring
activities or be extended to cover State and local agencies,
contracting laboratories, EPA grant recipients, and the
manner in which it should be implemented. What are the
alternatives and their costs? The other major consideration
is the extent of control needed to ensure reliable and
defensible data. What are the costs associated with the
various alternative "control levels?"
-------
46
Should EPA's quality assurance program be extended
to external monitoring programs?
Although stated as an unresolved issue, the question
of extending the EPA quality assurance program to external
laboratories, namely State and local agencies, is rather
academic. The very nature in which the monitoring
functions and responsibilities are carried out by the
State and local agencies and the Federal pollution control
agencies dictate the need for a broadly based quality
assurance program encompassing all these activities. This
conclusion is based on the following:
(1) EPA depends upon State and local data.
(2) EPA has and will continue stipulating
requirements for State and local
monitoring programs^
(3) EPA as well as State and local data
are routinely entered into joint
information systems for common useage.
The resources available for monitoring activities have
always been limited, requiring that the maximum use be made
of the "monitoring dollar." Wherever possible, duplication
of effort must be avoided and the available data must be
used to serve a multitude of purposes and by the various
agencies or organizations requiring the data. The data
needs by EPA are many and varied ranging from evaluating
compliance with standards to determining the health effects
that occur at different levels of pollution. The current
-------
47
or projected EPA monitoring capabilities fall far short of
the level of effort that is needed. In the future, as in
the past, EPA will need to depend to a large extent upon
monitoring data produced by the State and local agencies.
In fact, the current level of effort by the State and local
agencies for monitoring the more common pollutants surpass
those of EPA by a factor of nearly 10 in the case of air
monitoring and by a factor of more than five for water quality
monitoring. Full use should be made of these data.
Before EPA can place full reliance upon these data,
their accuracy and precision must be known and fall within
acceptable limits. The use of data generated by numerous
agencies and laboratories for a common purpose (e.g., delin-
eation of nation-wide trends) required that these data be
of acceptable quality and be comparable. Only through an
active quality assurance program can EPA verify the State
and local data it uses and upgrade the data where necessary.
In many instances, EPA now places monitoring require-
ments upon State and local agencies by specifying the pol-
lutants that need to be measured, the number of stations
that must be installed, types of instrumentation, frequency
of sampling, etc. It follows that the requirements for data
quality should be made part of these requirements.
-------
48
Many agencies and organizations, governmental and
private, use environmental data from the EPA environmental
data systems, e.g., SAROAD and NEDS. It is extremely
desirable that all of the data entered into these systems
be, if not of the same quality, of acceptable and known
accuracy and precision. A good quality assurance program
can meet this goal.
RECOMMENDATION: Because of the above reasons, it
is recommended that State and local monitoring programs be
covered under the EPA quality assurance program.
How should the EPA quality assurance program be
operated with respect to the State and locaT
monitoring programs?
There are several basic options available to EPA in the
formulation of the strategy for a quality control program
for State and local monitoring activities. They differ in
the requirements that EPA wishes to place upon the State
and local programs. Each option must be very carefully con-
sidered in terms of the improvements in data that would
result and in the resources that EPA must expend. Since the
number of operating agencies and their laboratories is large,
a systematic and coordinated effort is required to implement
a successful program. Furthermore, the acceptance of the
program by the individual State and local agencies is a
necessity because without their willing participation the
-------
49
quality assurance program will fail. A well designed and
operated quality assurance program will greatly enhance
our future working relationships with the State and local
monitoring activities. A poorly operated program can have
the opposite effect and must be avoided.
In all, three approaches have been identified. They
are: (1) mandatory participation; (2) voluntary participation;
and, (3) voluntary participation with some requirements as
to the data.
The first option would require the participation of all
State and local monitoring activities that impinge on the
mandated EPA responsibilities. Under this option, EPA would
require the States to adhere to EPA monitoring guidelines,
use EPA designated standard reference materials and samples,
and participate in inter-laboratory testing and evaluation.
These requirements could be levied upon the States through
regulations concerning national environmental standards.
The major advantages and disadvantages of this approach are:
Advantages:
Full participation by all monitoring
programs is assured.
The "weak" programs can be readily
identified and corrective actions
can be instituted.
Data of better quality would ensue.
-------
50
Uniformity of data would be
increased through use of common
and approved techniques
nationwide.
Disadvantages:
Another EPA requirement for
States could "worsen"
relationships.
The required EPA resources for
immediate implementation of
the program are nearly prohib-
itive. (The average Region
would require about four to five
qualified individuals to admin-
ister the program. The volume
of standard reference materials
and samples required would
increase tenfold.)
The legal issues concerning
requirements will delay start
of the program.
The second option places no requirements on the external
programs. The Environmental Protection Agency would make
the various aspects of the quality control program available
to the State and local agencies. For example, EPA would
distribute methods and quality control guidelines and invite
the various laboratories to participate in the inter-
laboratory testing program. The Environmental Protection
Agency would provide standard reference materials and samples
as requested, if available. The advantages and disadvantages
of this approach are:
-------
51
Advantages;
Program can be started immediately
and expanded as resources and
expertise increase. (For water
quality, radiation, and pesticide
measurements, this service is now
being provided through the NERC's.)
An additional requirement is not
placed on the States. This may
result in a program that may be
more acceptable.
Disadvantages:
It is probable that the programs/
laboratories in most need of
quality control would not volun-
tarily participate.
Data of questionable quality would
still be submitted to EPA1s
information systems.
The implementation of the system
may never reach full coverage.
The third option would combine certain facets of the
previous two approaches. The Environmental Protection Agency
would require that the validity of all of the monitoring
data submitted to EPA be documented. The Environmental
Protection Agency would issue guidelines concerning the
information that must accompany the data. Such information
could include sample history, analytical methodology/
calibration, and performance checks. In addition, EPA would
continue to provide monitoring methods and guidelines,
-------
52
conduct inter-laboratory testing (on a voluntary basis),
and make standard reference materials and standard reference
samples available. Certification may or may not be part
of this program. The pros and cons of this option are:
Advantages:
The validity of all data submitted
to EPA is known.
The program can be implemented
on a more timely basis and
expanded as resources become
available.
The "weak" programs can be iden-
tified and assistance may be
offered.
The latitude for quality assurance
given to the States should make
the program more widely acceptable.
A less rigid approach to quality
control may better serve to
advance the state-of-the-art.
The quality assurance program will
be self regulating.
Disadvantages;
At least initially, the overall data
quality may not be substantially
improved.
EPA will need to be concerned with
a much greater number of methods
and procedures.
The preparation and issuance of data
validity requirements will delay
the start of the program.
-------
53
Ideally, the first option (i.e., mandatory State par-
ticipation) will provide EPA with the best data base and
the most complete coverage. Furthermore, since the States
would be required to adhere to EPA guidelines, the pro-
liferation of non-uniform methods and procedures would be
minimized. This approach, however, does not appear possible
at the existing resource levels. Once the requirements are
placed upon the States, EPA must be in a position to admin-
ister the program, which would include both technical
assistance and official evaluation functions. Both of these
tasks will require substantial resources by the Regions and
the quality control support groups at the NERC's.
The second option (voluntary participation) is the
easiest for EPA to undertake. In fact, the on-going programs
in water, radiation, and pesticides presently operate in
this manner. Past experience has shown that these programs
are well accepted by the more qualified external programs/
laboratories. However, the major weakness of this approach
is that the laboratories in most need of quality control do
not generally participate in a voluntary program. It is
these programs/laboratories that are of primary concern to
the EPA quality assurance program.
The third option eliminates the major disadvantages of
the other two approaches--the large resource requirements of
-------
54
the mandatory program and the limited coverage of the volun-
tary program. The submission of "data quality" information
by the State and local agencies will provide a basis upon
which to judge the validity of the data EPA receives. The
States, although invited to take part in the various ele-
ments of the EPA quality assurance program, are not required
to participate. They can implement and carry on their own
quality control. This approach does not place an over-
whelming resource requirement upon EPA immediately, yet pro-
vides for a way to review data validity. An important
aspect of this option is that it provides a self-regulating
mechanism. The need for the State and local agencies to
provide certain quality control statistics will encourage
them to upgrade their monitoring activities through increased
quality control practices.
Development of a quality assurance program for
State and local agencies
The proposed strategy calls for both the stipulation of
certain quality control information that must accompany all
the data submitted to EPA and the provision of quality con- >
trol services to the State and local agencies. Each of
these are briefly discussed here as are the various options
that are available with respect to implementation of the
program.
-------
55
"Quality Control" Information; The data cur-
rently submitted to EPA, for the most part, consist only of
the results with a minimum of supporting information.
This supporting information is limited generally to site
location, sampling time and duration, method of analysis
used (sometimes), and the agency reporting the data. On the
basis of this information, it is impossible to evaluate the
validity of the data.
With the help of additional statistics (that can
be compiled by each agency/laboratory and submitted with
the data), EPA could evaluate the data prior to acceptance
into the information systems used by EPA. The supporting
information needed would consist of:
(1) Sample handling and verification methods.
(2) Details of the analytical method used.
(3) The type and source of standard reference
materials and standard reference samples used.
(4) Frequency of use of quality control or
check samples.
(5) Last time the program participated in an
inter-laboratory testing program.
In addition, to aid in evaluating the validity
of data, "baseline" information on each laboratory's capabil-
ities would also be compiled and kept up to date. This
-------
56
would contain information on each physical facility, equip-
ment, personnel, and the like. This "baseline" information
together with the accompanying quality control statistics
would provide the basis for evaluating the validity of the
data submitted to EPA.
There does not appear to be any alternatives to
this approach. Therefore, the basic decision is how best to
develop and implement a program to gather this information.
The implementation strategy must consider the following:
(1) The quality control data (including "base-
line" facts) should be compatible with EPA's data systems.
(The checking of data validity is to be computerized.)
(2) The information required must be kept at
an absolute minimum.
(3) The submission of the accompanying infor-
mation must be made an integral part of the data flow from
State/local agencies to EPA.
Implementing this portion of the program will require
the accomplishment of a series of tasks. These tasks together
with suggested organizational responsibilities are given
below:
(1) Compilation of "baseline" information.
This information will serve to provide an initial rating of
the capability of an individual monitoring program/laboratory
-------
57
that submits data. (In most likelihood, the capabilities
will be pollutant-oriented.) It is proposed that a question-
naire be developed by the Office of Monitoring (and the
NERC's) in concert with the Regional Offices and other
appropriate EPA elements (including the information systems).
The information would be collected, compiled, evaluated by
the Regional Offices, and forwarded to the Office of Monitor-
ing through the appropriate information systems. The
evaluation or "rating" scheme is to be devised by the pre-
viously named EPA organizations. An annual update of this
information is envisioned. The following sequence is
proposed:
Develop a questionnaire and
rating scheme.
Compile baseline information.
Baseline information included
in data systems (to be
determined).
(2) Submission of "quality control" information.
This information, to be routinely supplied with all data,
will provide a statistically sound basis for evaluating the
validity of submitted data. It is proposed that the list of
required "quality control" information be developed by the
Office of Monitoring (with the NERC's), the Regional Offices,
and other appropriate EPA elements. The participation of
-------
58
information systems (SAROAD, NEDS, etc.) is essential,
since the routine evaluation or "flagging" of invalid or
questionable data is to be computerized. Once developed,
the quality control information will become an integral
part of all data submitted to EPA. The manner in which
the quality control information is to be collected and
incorporated into the system will need to be investigated.
Quality Control Services; According to the
recommended option, EPA will provide certain quality con-
trol services to the State and local monitoring programs/
laboratories. The alternative approaches of the EPA involve-
ment are best discussed through the consideration of the
nine project areas considered in the "plan." The organiza-
tional responsibilities for these tasks are given at the
end of this discussion.
(1) Preparation and issuance of procedures
and guidelines. The manuals prepared and issued for EPA
monitoring activities (see plan, page 4) will be applicable
to State and local agencies. Whether or not included under
EPA quality control, the State/local agencies would take
advantage of these manuals. The only extra cost (minimal)
to EPA would be on publication and distribution. There are,
however, two points that should be considered:
-------
59
(a) State/local agency involvement in
their preparation and review.
(b) Whether reordering of priorities
is necessary.
In order that these manuals reflect the needs
of the States, take advantage of their expertise (in some
areas, certain State/local laboratories are very advanced),
and receive wider acceptance, a mechanism for their partic-
ipation should be developed. This may be either in the form
of scheduled advisory group meetings or purely on an ad hoc
basis. The preparation of some guidelines may, in fact,
be "contracted" to a given State or States. Since the pri-
orities for control are dictated by common legislation, the
priorities for methods or procedures should not vary between
EPA and the States.
(2) Provision of standard reference materials
and samples. Standard reference materials and samples are
needed to calibrate monitoring and laboratory instrumentation,
to evaluate proposed new methods, and to maintain a con-
tinuous check on the validity of the data that are produced.
The success of an inter-laboratory quality assurance program
depends on the availability and use of high quality standard
reference materials and standard reference samples. Cur-
rently, there is little or no control over the use of standard
-------
60
reference materials and standard reference samples (type,
quality, frequency of use, etc.). The State/local programs,
as well as other monitoring activities, obtain these mate-
rials from a wide variety of sourcesEPA laboratories,
National Bureau of Standards, private laboratories, and in
some cases, the reference samples are being prepared by the
user laboratory itself. Increased control on the production
and use of standard reference materials and standard refer-
ence samples is necessary if the State/local data are to
be upgraded and their validity assessed. There are several
options available to EPA in making these materials available
to State/local programs. These are:
(a) EPA purchases the standard refer-
ence materials from appropriate sources such as the National
Bureau of Standards and private organizations, prepares
standard reference samples, and makes these materials avail-
able to State/local agencies at no cost. (These repositories
could be maintained by the quality control groups at the
NERC's, Regional Offices, or both.)
(b) EPA identifies and supports (and
approves) an external organization(s), such as the National
Bureau of Standards and private suppliers, as a central
repository for standard reference materials and standard
reference samples. (The actual production cost of the
-------
61
materials would be borne by the users. EPA would continue
the support of the development of these materials. Although
the National Bureau of Standards would be a major source,
private suppliers could be designated as approved sources
of given materials.)
(c) Combination of options 1 and 2.
EPA provides priority standard reference materials and stand-
ard reference samples as resources allow and identifies com-
mercial sources where these and other materials may be
obtained.
(d) EPA requires that all agencies sub-
mitting data to EPA document the source(s) and quality of
the standard reference materials/standard reference samples
they use. (The agencies would be free to select their own
source of these materials.)
The first option assures the maximum EPA con-
trol over all monitoring data which it receives from the
State/local programs. It also assures a much wider use of
these materials. However, with present funding restrictions,
the logistics and the cost of providing this service may
seriously limit the scope of coverage. Option 2 above would
reduce the cost to EPA. This alternative requires that
external organizations have the necessary expertise and
willingness to develop and maintain such a repository.
-------
62
Moreover, a service "charge to the user" could reduce volun-
tary participation and delay the response to State/local
needs. The third option (e.g., the provision of all standard
reference materials and standard reference samples to State
and local monitoring programs) would not place an over-
whelming requirement upon EPA. It would allow EPA to develop
and provide reference materials on a priority basis. The
identification of private and other governmental suppliers
of reference materials will allow EPA to maintain some con-
trol over their use. This option, although requiring a
minimum of EPA resources, will not provide performance up-
grading in the various laboratories. If it results in
proliferation of standard reference materials/standard
reference samples of unknown purity, the overall quality of
data will not be substantially improved.
RECOMMENDATION: For the reasons cited above,
option 3 is recommended to provide standard reference
materials and standard reference samples.
(3) Operation of intra-laboratory quality
control. This activity is the responsibility of each program
and laboratory. The need to submit quality control infor-
mation with the data will encourage the States and local
agencies to maintain an adequate quality control program
-------
63
within their laboratories. Guidelines would be provided
through Project 1. There would be no extra cost to EPA.
(4) Operation of inter-laboratory quality
control. An inter-laboratory quality control program provides
an evaluation of the performance of individual laboratories
in conducting a given analysis or series of analyses. This
activity also serves as the mechanism for evaluating new
methods. An inter-laboratory quality control program (cross-
check sample program) serves the interests of participating
laboratories by identification of their shortcomings. It
also serves the interests of the data recipients by
documenting the validity of the data.
As part of the recommended strategy, inter-
laboratory quality control would be provided to the State/
local agencies on a voluntary or service basis. This proce-
dure, in fact, is currently being followed in the water,
pesticide, and radiation monitoring activities. Under this
approach, the quality control groups at the NERC's would
conduct inter-laboratory tests (issue test procedures and
samples and evaluate the results). The State/local agencies
would be informed of these tests and invited to participate
(by the Regional Offices). This approach raises the question
of extent of State participation in a voluntary program.
Experience has shown that the more qualified laboratories
-------
64
participate in a voluntary program of this type whereas
laboratories/programs most in need of performance improve-
ment do not participate. Under the proposed program,
this should not happen. The requirement for the State and
local monitoring programs to provide quality control infor-
mation as part of their data submission to EPA should serve
as a self-regulating mechanism. Furthermore, it encourages
all programs to participate in inter-laboratory quality
control.
(5) Laboratory and field evaluation of
instruments. This project area is intended to provide guide-
lines for selecting and evaluating instruments. The extension
of the quality control program to the State/local agencies
would not require an additional effort on the part of EPA.
The experience of State/local agencies with methods and
instruments can provide valuable inputs to guideline
formulations.
(6) Quality control - Site and laboratory
evaluation. This project provides external checks on the per-
formance of on-going monitoring activities. These checks
may range from site inspections and review to actual field
calibration using mobile units. This activity is a necessary
consequence of an on-going inter-laboratory program uncover-
ing weak or marginal performance of State/local monitoring
-------
65
programs. Also, this project provides assistance, if
requested, by State/local programs. This project area will
require considerable activity on the part of EPA which
would be carried out by the Regional Offices with the
assistance of the NERC's as required.
(7) Development and management of laboratory
certification program. A most important issue in the develop-
ment of a strategy for extending the EPA quality control
activity to State/local monitoring programs is the question
of certification of State and local laboratories, analysts,
etc. A certification program requires the establishment of
performance levels against which the performance of each
laboratory will be judged in terms of facilities, equipment,
personnel, participation in inter-laboratory quality control
programs, etc. and, upon approval, the issuance and periodic
updating of certificates. "Certification" is the culmination
of an inter-laboratory quality control program and carries
with it an official approval label. The advantages and dis-
advantages of a formal certification program are discussed
below:
Advantages;
The evaluation of data validity
is improved.
Laboratory performance is upgraded.
(The need for increased performance
-------
66
levels can be used as a tool for
obtaining resources, high level
of personnel, etc.)
Disadvantages:
The current state-of-the-art is
not sufficient to provide a strong
scientific basis for setting
performance levels. (The setting
of performance levels assumes the
existence of model laboratories
against which all others would be
judged. Statistical models may
or may not be a sufficient basis.)
Official certification may
"worsen" relationships with the
States. (Another EPA require-
ment on the States.)
The operation of a certification
program will require substantial
resources that can best be used
elsewhere.
A certification program may be a
deterrent in continual development
and improvement for the "certified"
programs/laboratories.
RECOMMENDATION: For the present, a formal
certification program is not proposed. It must be understood,
however, that many of the activities proposed as part of the
quality control program for States, i.e., data requirements,
inter-laboratory tests, etc., provide a mechanism for
continuous evaluation of program/laboratory performance.
In effect, the "grading" of laboratories and their data
-------
67
(which will be carried out) constitutes a form of certifica-
tion, lacking only the formal approval of laboratory
performance.
Once the quality control program is under
way, the question of certification should be reexamined. In
the interim, EPA should continue to provide assistance to
those States that are developing their own certification
programs.
(8) Communication and coordination. The
extension of quality control services to State/local programs-
will require a substantial coordination and management effort
by EPA. Information on laboratory activities/capabilities
will need to be collected and maintained. Problem areas will
need to be identified and addressed. Continual updating of
the program will also be necessary. These tasks will primarily
fall upon the Regional Offices.
(9) Training. Training of quality control
specialists and monitoring personnel is an important part of
the quality control program. The number of State/local
personnel that will need training is large and requires sub-
stantial resources. The following approach is recommended:
(a) Workshops on quality control manage-
ment aspects, proposed for EPA programs, should also include
the appropriate State personnel.
-------
68
(2) Specific courses on quality control
techniques, i.e., use of quality control charts, split
samples, etc., should be provided to State/local personnel
through the Regions.
(3X Courses in the use of analytical
methods should continue to be part of EPA's training
program (see plan).
RESOURCE REQUIREMENTS
The extension of the quality control program to the
State and local agencies will require EPA to provide resources
for:
(1) The initial development and the routine main-
tenance and evaluation of quality control information which
will accompany all data submitted to EPA; and,
(2) the provision of quality control services to
the State and local monitoring programs.
Whereas the first of these activities is independent of the
development of the EPA quality control program, the provision
of quality control services is closely allied to it. Any
limitations placed upon the EPA program will reduce the
services accordingly. Thus, for example, the lack of support
for the development and production of standard reference
materials and standard reference samples will reduce the
-------
69
number of these materials available for distribution to State
and local programs.
The development and maintenance of quality control
information will require resources for: (1) expansion of
the information systems (NADIS, STORET, etc.) to accommodate
this information; and, (2) the routine handling and evalua-
tion of the information. The cost of reprogramming will
need to be determined. The routine evaluation will require
one to two individuals to be assigned to each information
system for dealing with quality control problems.
The provision of quality control services to the State/
local agencies should be primarily the responsibility of
the Regional Offices with technical support provided by the
quality control groups at the NERC's. The main thrust of
this program, as pointed out earlier, will be to supply
standard reference materials and standard reference samples,
conduct inter-laboratory tests, provide technical assistance,
conduct periodic quality control checks, and make training
courses available to the appropriate State and local
personnel.
IMPLEMENTATION SUMMARY
The increased dependence by EPA on the State/local
monitoring data requires that the quality control program
be implemented as soon as possible. In most likelihood,
-------
70
the provision of quality control services will be insuf-
ficient at firstbecause of funding limitations and the
need for EPA to develop the basic quality control tools.
The following major milestones are proposed:
Determine the mechanisms to be used
for collecting and evaluating
quality control information.
Issue a list of quality control
information needed.
Initial "grading" of monitoring
programs.
States informed of quality control
services.
Program under way.
-------
71
Comments on "Development of an
Agency-wide Quality Control Program"
INTRODUCTION
Earlier this year, the Office of Monitoring distri-
buted for review a proposal for an Agency-wide quality
control program. Comments were requested from nearly 30
EPA organizational segments including the NERC's, Regions,
and Program Offices. In all, some 20 responses were
received covering all of the Regional and many of the
other offices. This paper summarizes their comments and
the major changes made in the document. First, the
major comments and the resulting changes are discussed.
This is followed by a summary listing the individual
comments submitted.
DISCUSSION
There was a general consensus throughout EPA that a
strong Agency-wide quality control program is urgently
needed. Although most of the reviewers agreed with the
implementation strategy, there were some different points
of view regarding the manner in which the program should
be carried out. The technical aspects of the program,
i.e., the description of the elements of a quality control
program were well received by the reviewers. Some
objections were raised with respect to the appraisal of
current status and capabilities of the on-going programs.
Several reviewers felt that the report was overly
critical of some programs. The major issues raised by the
review are given below.
(1) Varying degrees of quality control; The
proposal included a recommendation that the degree of
quality control should be varied on the basis of the end
uses of data, e.g., data to be used for enforcement would
be produced under stricter quality control than, say, data
for trend assessment. Several reviewers addressed this
issue and strongly recommended that all EPA data be
subjected to same control. Their recommendations were
based on the argument that EPA shou-ld only produce data
of high quality, that the end uses of data are not
frequently known, and that strict quality control measures
are needed for trend and background monitoring where the
concentrations measured are much lower than for enforce-
ment. On the basis of their arguments, the proposal was
-------
72
changed and now recommends a uniform degree of quality
control for all data.
(2) Regional role in the development of guidelines;
Comments submitted by the Regional Offices addressed the
need for Regional input in the development of the proce-
dures and guidelines. In their view a much greater
Regional input was called for than outlined in the proposal.
It was not intended to exclude the Regions from this very
important task. The proposal was modified accordingly.
(3) Quality control over sampling: Some reviewers
felt that the sampling portion of monitoring was not given
sufficient attention in the proposed quality control
program. The general consensus was that it is extremely
important. Changes were made in the document to give
increased emphasis on quality control relating to sampling.
(4) Compulsory quality control program; The
proposal calls for a very strong and compulsory program.
Comments regarding this issue were quite varied. Although
several reviewers supported a, compulsory program, some
felt that the proposed program was too authoritative and
under too much "strict Washington control." If EPA is to
establish and implement an adequate program, it must be
adhered to by all monitoring programs. No basic change
was made.
In addition to the above, numerous changes were made
in the document in response to the comments.
SUMMARY OF COMMENTS
REGION I (1) Does not agree that the degree of
quality control should be varied
according to uses of data.
(2) Emphasizes that program must be
compulsory-
(3) Emphasizes that program must cover
all aspects of monitoring. Feels
that program is too much laboratory
oriented.
(4) Does not think that the program
can be operational by July 1973.
-------
73
REGION II
REGION III
REGION IV
(1) Feels that the question of how,
who, and how many will direct and
operate the program is not suf-
ficiently described.
(2) Comments that changes in design and
construction are not applicable to
existing facilities.
(3) Recommends the use of additional
control charts.
(4) Recommends that a full-time quality
control coordinator be designated
in each Region.
(5) Recommends a certification program
for State and local programs/
laboratories.
(6) Recommends that a separate quality
control group be set up for marine
related monitoring.
(1) Agrees with the proposed approach.
(2) Recommends extending the program
to State and local agencies.
(3) The resources programmed for train-
ing are underestimated. Should
be about two man years and $60,000
per Region.
(4) Raises questions concerning the
detailed implementation of the
program.
(1) Agrees that a strong quality con-
trol program is needed.
(2) Suggests that resources be made
available to the Regions for prep-
aration of methods manuals and
the provision of standard reference
materials and standard reference
samples.
-------
74
REGION V
REGION VI
REGION VII
(3) Recommends the allocation of 10
man years to Region IV for
carrying out a certification
program.
(4) Does not agree with the statement
that much of the existing data
are questionable.
(5) Recommends that every EPA
reference method be interlab-
oratory tested.
(6) Recommends that EPA do not depend
upon external organization for
quality control.
(1) Agrees with the approach, particu-
larly with the recommendation that
the program be compulsory.
(2) Does not agree that the degree of
quality control can be varied.
All data should be of same quality.
(3) Raises a question on how many
laboratories can one quality con-
trol office handle.
(1) Questions the need for a certifica-
tion program.
(2) Thinks the schedule of projects and
tasks are well laid out.
(1) Agrees with the concept of an
Agency-wide quality control
program.
(2) Thinks the program is under too
much "strict Washington control."
(3) All data must be subjected to same
level of quality control.
(4) Instructions on sampling site
selection and calibration
frequency.
-------
75
REGION VIII
REGION IX
REGION X
(5) Laboratory certification is a
must.
(6) Resources and identity of the
program are needed now.
(1) Agrees with the concept of the
program.
(2) Does not agree with the estimated
resource requirements for the
Regions. Recommends that such
additional resources must be made
available to the Regions.
(3) Agrees that the program should
cover State and local efforts, but
that its scope should not be
disproportionate to the available
resources.
(1) Regions should be involved in the
development of guidelines, train-
ing programs, etc.
(1) Agrees with the proposed program.
(2) Thinks guidelines for sampling
may result in being too restrictive,
(3) Feels that a formal laboratory
certification program is several
years away in Region X.
(4) The issuance of standard reference
materials and standard reference
samples should be through Regional
coordinators.
(5) Suggests that Regions be involved
in procedure and guideline
development.
(6) Recommends the inclusion of
Regional coordinators in any
Quality Control Intra-Agency
Committee.
-------
76
Office of
Radiation
Programs
Office of
Categorical
Programs
(1) Agrees to the need to establish
an Agency-wide quality control
program.
(2) Recommends that the existing
radiation quality control group
(activity) at Winchester be more
recognized and that EPA quality
control programs for radiation
measurements be under the Office
of Radiation Program's control
at Winchester.
(3) Suggests that quality control
for the sampling end of monitor-
ing be emphasized more.
(4) Thinks that the report is too
critical of on-going quality
control activities.
(5) Suggests an inconsistency between
recommendations regarding "an
authoritative/compulsory program"
and the statement that quality
control is the responsibility of
the monitoring program which it
serves.
(6) Suggests that the proposed mile-
stones are far too ambitious.
(1) States that the report is unclear
regarding the development of
standard reference materials to
support the pesticide registration
program.
(2) Recommends that guidelines on
sampling consider statistical
basis for design of sampling
programs and schedules.
(3) States that there are a number of
unresolved issues within EPA to
adequately evaluate the proposal
with respect to inter-laboratory
-------
77
Office of
Water
Programs
NERC-RTP
NERC-LV
NERC-Cin
quality control, coordination,
certification, and training.
(4) States that the proposal does
not consider the quality control
activities of Program laboratories.
Suggests that these operate
independently with the Agency-
wide program as a point of
reference.
(5) States that the proposed resources
are excessive.
(1) Comments that the program is
realistic.
(2) Recommends that quality assurance
must also include data storage
and validation.
(1) Agrees with the proposal.
(2) Does not favor a certification
program at this time.
(1) Agrees with the proposal.
(2) Suggests that too much emphasis
is placed on standard methods.
(3) Suggests the use of duplicate
samples for intfa-laboratory quality
control.
(4) Recommends the shifting of radia-
tion quality control from
Winchester to NERC-LV.
(1) States that the program cannot be
centrally imposed, but must be
accepted by all elements.
(2) The evaluation of current status
does not give sufficient credit
to existing quality control
programs in water and pesticide
monitoring.
-------
78
NERC-Cin (3) The suggested sampling guidelines
(Cont) are too restrictive.
(4) The specific roles of the Office
of Monitoring and the NERC's are
not adequately defined.
(5) The project/task descriptions
for water are not accurately
described.
-------
79
Development and Review of the Plan
Quality Control Workshop
An Agency-wide quality control workshop was held on
February 16 and 17 to lay the groundwork for the
Agency's quality control program. The workshop was
attended by approximately 50 individuals represent-
ing all Regions, NERC's, and most of the Program
Offices.
Review of the Quality Control Plan
Comments on the plan were submitted by the
following individuals:
Region I
Region II
Region III
Region IV
Region V
Region VI
Region VII
Region VIII
Region IX
Region X
Office of
Radiation
Programs
- E.V. Fitzpatrick,Director
Surveillance & Analysis Division
R.T. Dewling, Director
Surveillance & Analysis Division
J.G. Gardner, Deputy Director
Surveillance & Analysis Division
J.H. Finger, Chief
Chemical Services Branch
R.J. Bowden, Director
Surveillance & Analysis Division
E.R. Lozano, Deputy Director
Surveillance & Analysis Division
G.L. Fish, Director
Surveillance & Analysis Division
J.A. Green, Administrator
B.D. Clark, Director
Surveillance & Analysis Division
G.L. O'Neal, Director
Surveillance & Analysis Division
- W.D. Rowe, Deputy Assistant Administrator
-------
Office of
Categorical
Programs
Office of
Water
Programs
NERC-RTP
NERC-LV
NERC-Cin
NERC-Corv
Office of
General
Counsel
80
G.R. Cornstock, Program and
Management Operations
K.M. Mackenthun, Director
Applied Technology Division
S. Hochheiser, Chief
Quality Control Branch
M.W. Carter, Director
D.G. Ballinger, Director
Analytical Quality Control Laboratory
D.F. Krawcyzk, Director
Consolidated Laboratories
B. Anderson, Staff Chemist
. GOVERNMENT PRINTING OFFICE: 1974 546-317/297
-------
POINTS OF CONTACT FOR QUALITY CONTROL PROGRAM
Media
Program
Management
Contact
Mr. Guntis Ozolins
Director
or
Mr. Thomas W. Stanley
Chief
Location
Quality Assurance Division
EPA Headquarters
Quality Control Branch
EPA Headquarters
Phone No.
202/755-0646
202/426-2382
Water
Mr. Dwight G. Ballinger
Director
Methods Development and Quality
Assurance Research Laboratory
NERC-Cincinnati
513/684-2925
Air
Dr. S. David Shearer
Director
Quality Assurance and Environmental
Monitoring Laboratory
NERC-RTP
919/549-2106
Pesticides
Dr. William F. Durham
Director
Pesticides and Toxic Substances
Effects Laboratory
NERC-RTP
919/549-2655
Radiation
Mr. Richard E. Jaquish
Chief
Technical Support Laboratory
NERC-Las Vegas
702/736-2969
Attachment
------- |