DRAFT
GUIDE TO THE PREPARATION OF QUALITY
ASSURANCE PROJECT PLANS
for the
OFFICE OF TOXIC SUBSTANCES
OFFICE OF PESTICIDES AND TOXIC SUBSTANCES
U.S. Environmental Protection Agency
Washington, D.C. 20460
September 28, 1984
-------
ufS QM project Plan
Revision No. 2
Date: 09/28/84
Page i of v
ABSTRACT
The purpose of the quality assurance policy of the Office of Toxic
Substances is to ensure that (1) the quality of all reported data is deter-
mined so that all data is of known quality and (2) all environmental data
generated within or for OTS meets quality standards consistent with the
intended uses of the data.
A requirement of OTS QA policy is that each project which generates
environmental data must develop a Quality Assurance Project Plan covering its
data, must ensure that adequate resources (both monetary and staff) are pro-
vided to support the quality assurance effort. The QA Project Plan includes
all phases of the monitoring program: sample planning and collection,
laboratory analysis, data processing, and the analysis of the final results.
The purpose of this document is to provide a detailed guide to the
preparation of QA Project Plans for OTS programs.
-------
O'lii QA Project Plan
Revision No. 2
Date: 09/28/84
page iv of v
TABLE OF CONTENTS
(Continued)
ITEM " PAM
APPENDICES
A. Glossary A~1
B. Systems Audits B-l
C. Performance Audits C-1
D. Laboratory Inspections D-l
-------
LIST OF FIGURES
OTS QA P, _j.:;t ' _;i
Revision No. 2
Date: 09/28/84
page v of v
FIGURE RAGE
1 Suggested Table of Contents for a QA Project Plan 4
2 Example of Sample Label 16
3 Example of Field-Tracking Report Form 17
4 Example of Lab-Tracking Report Form , 19
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page 1 of 33
1.0 INTRODUCTION
In order to assure the quality of environmental data used by the
EPA, the Administrator issued two memoranda: "EPA Quality Assurance Policy
Statement", issued on May 30, 1979(1), and "Quality Assurance Requirements for
All EPA Extramural Projects Involving Environmental Measurements", issued on
June 14, 1979(2). More recent Administrators continue to emphasize quality
assurance (QA) in environmental measurement data and require that such data be
of known quality. In these policy memoranda, the Administrators have stated
that the Agency-wide QA effort is mandatory, and that the Quality Assurance
Management Staff (QAMS) of the Office of Research and Development (ORD) is
responsible for the development of the Agency-wide QA Program.
The purpose of the QA policy of the Office of Toxic Substances
(OTS), as described in the OTS QA Program Plan(3), is to ensure that:
( i) the quality of all reported data is determined so that all data is
of known quality,
(ii) all environmental data generated within or for OTS meets quality
standards consistent with the intended uses of or that data.
Quality assurance requirements cover all environmental measure-
ment data, and apply not only to in-house efforts but also to all grants,
contracts, cooperative agreements, and interagency agreements.
An important requirement of OTS QA policy is that each project which
generates environmental data must develop a Quality Assurance Project Plan
covering its data, must ensure that adequate resources (both monetary and
staff) are provided to support the quality assurance effort, and must be
responsible for conducting the quality assurance effort. The QA Project Plan
specifies the detailed procedures to be followed to assure quality data.
QA Project Plans describe the specific QA-related requirements and
procedures to be followed to assure that all environmental data which is used
is of known quality and of appropriate quality for the intended uses. The QA
Project Plan includes all phases of the monitoring program: sample planning
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page 2 of 33
and collection, laboratory analysis, data processing, and the analysis of the
final results. Each project manager should work with the QA Officer for OTS
during the development, approval, and implementation of these project plans.
The purpose of this document is to provide a detailed guide to the
preparation of QA Project Plans for OTS programs.
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page 5 of 33
2.2 Document Control
All QA Project Plans must be prepared using a document control
format consisting of information placed in the upper right hand corner of each
document page:
Section Number
Revision Number
Date (of revision)
2.3 Responsibilities
2.3.1 Intramural Projects
Each Project Officer working in close coordination with the QA
Officer is responsible for the preparation of a written QA Project Plan for
each intramural project that involves environmental measurements. This
written plan must be separate from any general plan normally prepared for the
project. The Project Officer and the QA Officer must ensure that each
intramural project plan contains procedures to document and report precision,
accuracy, and completeness of all data generated.
2.3.2 Extramural Projects
Each Project Officer working in close coordination with the QA
Officer has the responsibility to see that a written QA Project Plan is
prepared by the extramural organization for each project involving
environmental measurements. The elements of the QA Project Plan must be
separately identified from any general plan normally prepared for the project.
The Project Officer and the QA Officer must ensure that each extramural
project plan contains procedures to document and report precision, accuracy
and completeness of all data generated.
-------
'.:> Project Plan
Revision No. 2
Date: 09/28/84
Page 6 of 33
3.0 CONTENTS OF A QA PROJECT PLAN
The following is a brief discussion of each item listed in the
suggested Table of Contents (see Figure 1).
3.1 Title Page
At the bottom of the title page, provisions must be made for the
signatures of the approving personnel. As a minimum, the QA Project Plan must
be approved by the following:
A. For intramural projects
1. Project Officer's immediate supervisor
2. QA Officer
B. For extramural projects
1. Organization's Project Manager
2. Organization's responsible QA Official
3. Funding organization's Project Officer
4. Funding organizations QA Officer
3.2 Table of Contents
See Figure 1 for a suggested Table of Contents.
3.3 Project Description
Provide a general description of the project, including the experi-
mental design. This description may be brief but must be clearly written and
have sufficient detail to allow those individuals resonsible for review and
approval of the QA Project Plan to perform their task. Items to be addressed
in the project description include:
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page 7 of 33
t A statement of objectives and hypotheses to be tested
t A description of the experimental design including the variables
to be measured, sample sizes, experimental materials, conditions,
and instruments
t An outline of the method of data analysis to be used
Anticipated duration of the project
Intended use(s) of the acquired data.
3.4 Project Organization and Management
In order for a monitoring study to proceed smoothly and yield valid
and usuable data, it is essential that all individuals are clearly informed of
their responsibilities. The Project Organization and Management Section of
the QA Project Plan should, at a minimum, identify key individuals responsible
for:
Sampling operations
Sampling QC
t Laboratory analyses
Laboratory QC
Data processing activities
Data quality review
§ Performance auditing
Systems auditing (on-site evaluations)
0 Overall QA and
Overall project coordination.
It is often useful on a project to indicate how these individuals relate in
the organization(s). An organizational chart is a convenient way of
illustrating this.
For each key individual named, provide a brief sentence or two
explaining that individual's responsibility. Telephone numbers should be
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page 8 of 33
listed with the key individuals in order to facilitate communications. The
name and number of an "on-call" person or project coordinator should be
available to field personnel at all times.
Where there are several different monitoring institutions or
subcontractors involved, complete addresses should be provided.
3.5 Personnel Qualifications
Describe the qualifications of all project personnel for their
assigned tasks, emphasizing quality assurance aspects. Resumes of key task
personnel should be included as an appendix.
3.6 Facilities, Equipment, Consumables and Services
The QA Project Plan must document the conformance of all facilities,
equipment, consumables and.services to OTS QA require ments. These
requirements are described below.
3.6.1 Facilities and Equipment
3.6.1.1 Evaluation
All OTS-supported facilities shall be documented prior to use as
capable of producing acceptable quality data in an efficient manner with
minimum risk to personnel.
The suitability of a facility for the execution of both the
technical and QA aspects of a task may be assessed prior to its use through a
systems audit (Appendix B) by qualified QA personnel. These audits shall
ascertain whether facilities are of adequate size, with satisfactory lighting,
ventilation, temperature, noise levels, and humidity and are operationally
consistent with their designed purpose. Satisfactory safety and health
maintenance features must also be present. OTS shall require the facilities
used to meet acceptable safety and health standards.
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page 9 of 33
Utility services must be operationally consistent with their
designed purposes and adequately provide for the generation and processing of
environmental data having the quality and integrity established by the QA
Project Plan.
General laboratory equipment must be present in sufficient quantity
and condition, operationally consistent with its intended use, to provide for
the generation and processing of environmental data having the quality and
integrity established by the QA Project Plan.
Personnel must be provided with adequate protective equipment to
ensure their health and safety. In addition, all personnel exposed to envi-
ronmental samples as a part of their duties shall be provided the opportunity
for health and safety medical monitoring services.
Similarly, all equipment is evaluated for its applicability to the
OTS task prior to use. Under the OTS QA Program, the relationships of all
measurement methods and the variables to be monitored must be well character-
ized and documented before being approved for use. Similarly, the subleties
of design and performance of different manufacturer's equipment shall be
thoroughly evaluated with the aid of a professional who has both a theoretical
and a practical understanding of the specific instrument operation. In addi-
tion, acceptance testing for new equipment is performed on an item-by-item
basis and is documented for comparison with future testing. All testing
programs are designed in such a way that operation of an instrument at its
extreme limits (i.e., worst case), as well as at routine settings, will be
thoroughly characterized before the instrument is made available for routine
use. Ongoing evaluation of equipment and facilities is provided by periodic
systems and performance audits (see Appendices B and C, respectively).
3.6.1.2 Inspection and Maintenance
In order to ensure consistently high data quality in the OTS
program, a plan for routine inspection and preventive maintenance (PM) must be
developed and followed for all facilities and equipment. Laboratory
inspections are conducted according to the methods described in Appendix D.
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page 10 of 33
Scheduling of a particular PM program is based on the identification
of critical components that are most likely to fail without PM and the overall
effect of facility or equipment failures on data quality.
All maintenance activities shall be performed by suitably qualified
technical personnel using accepted, documented procedures according to the QA
plan. The desirability of full or part time equipment operator and/or main-
tenance support is an important consideration. Frequently, sophisticated
instrumentation performs poorly or not at all when many occasional users have
access to it. On the other hand, minor but frequent maintenance often keeps
an instrument operating at peak performance. In such cases, the cost of a
full-time dedicated operator is justified.
Documentation of all maintenancescheduled or notis essential to
monitoring and documenting data quality. Permanent records of the maintenance
histories of all facilities and equipment, including detailed descriptions of
all adjustments made, parts replaced, etc., shall be kept in individual bound
notebooks, dated, and signed by the proper authority.
3.6.1.3 Calibration Procedures and Reference Materials
Calibration is the process of establishing the relationship between
the output of a measurement system and that of a known input; it allows
different instruments to be correlated with each other and with a specified
reference standard.
Calibration standards should be of the highest quality available and
fully characterized. In the United States, the National Bureau of Standards
(NBS) holds the position of final authority in the preparation of many refer-
ence materials and the NBS Standard Reference Materials (NBS-SRMs) are gener-
ally regarded as the best standards of each type available. Unfortunately,
for many common measurement processes routinely used at OTS, there are no NBS-
SRMs available. In these cases, the investigator must use the best available
calibration standard or devise a standard. Such standards must also meet the
requirements of high quality and complete characterization applicable to NBS-
SRMs.
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
.Page 11 of 33
Calibration procedures will:
Reference the applicable standard operating procedure (SOP) or
provide a written description of the calibration procedure(s) to
be used. For each major measurement parameter, including all
pollutant measurement systems.
List the frequency planned for recalibration.
List the calibration standards to be used and their source(s),
including traceability procedures.
3.6.2 Consumables
A well-documented acceptance testing program for all incoming
expendable supplies (e.g.,' chemicals and biological materials) shall be
applied prior to (and judiciously during) use. This acceptance screening
assures that supplies not meeting task specifications are not integrated into
the task's supply stream. The results of a successful acceptance test confirm
(a) that the substance fully corresponds to the label specifications, and
(b) that known or suspected interferents are absent.
Acceptance screening under the OTS QA Program involves two classes
of consumables: biological materials and chemicals. The screening of b-iolog-
ical materials presents special problems. Even though they may be regarded as
a commodity, they are susceptible to all the random variations applicable to
living organisms. Biological material used throughout .a particular experiment
should be obtained from the same source. Biological material should be
examined, upon receipt, for gross defects and a random sample should be
examined for parasites or other internal defects. During the quarantine
period organisms showing unusual behavior or appearance shall be destroyed.
On a regular basis, a random sample of the biological material shall be
examined for parasites, pathogens, lesions, etc. Standard operating
procedures on caring for the organism (handling, cleaning, feeding, etc.)
shall be documented and adhered to.
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page 12 of 33
The screening of chemical or reagent commodities involves verifica-
tion of assay and examination for impurities. Such screening is performed on
a batch basis using accepted, documented analytical methods. Special emphasis
is placed on the need to characterize all incoming cylinder gases containing
pollutants in specified concentrations. Following successful completion of
the acceptance test, an expiration date is permanently marked on each con-
tainer, and it is stored on a first-in first-out basis. Storage conditions
are maintained which will protect the integrity of the material and protect
personnel from harmful exposure. In particular, parameters such as
temperature, light, and humidity are considered.
In addition, recertification is performed routinely to characterize
changes in concentration, formation of new species, or loss of original
species to prevent them from degrading task data quality. Where possible,
the integrity of the substance is checked prior to each use.
A permanent record of all certification procedures, dated and signed
by the appropriate authority, should be kept in a bound .laboratory notebook.
3.6.3 Services
The reliability and quality of all service (e.g., analytical
services, audit services, etc.) provided shall be assessed both prior to and
during use, both in terms of personnel and service provided.
3.7 Data Generation
3.7.1 Experimental Design
During the planning and design phase of monitoring programs and
special studies, the primary QA consideration is to ensure the completeness
and representativeness of the samples to be collected. The level of quality
required should be explicitly specified in the QA Project Plan (e.g., data
acceptance limits, etc.).
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page 13 of 33
3.7.1.1 Quantification and Control of Sampling Errors
The sample design used by the monitoring program must assure a
statistically valid representation of the characteristics being assessed. The
sample design should be described in the QA Proje ct Plan and its statistical
validity should be addressed. The elements listed below must be adequately
considered:
The selection of appropriate sampling units (e.g., persons,
sites, etc.), based upon a clearly defined probability framework
which minimizes bias, so that results can be generalized to a
specifically defined population and the sampling errors can be
measured,
t Provision for a statistically sufficient number of samples and
sampling sites, including provision for lost or damaged samples,
etc.,
t Measurement of all necessary ancillary data (e.g., demographic
data),
t Determination of the climatic conditions, flow conditions, etc.
under which samples should be collected,
0 Determination of the frequency of sampling and length of. the
sampling period,
Determination of the types of samples (e.g., composites versus
grabs) to be collected.
3.7.1.2 Quantification and Control of Measurement Errors
The inclusion of QA procedures, reference standards, and QC samples
must be explicitly described in the QA Project Plan. They are the manager's
primary tools for assuring that the level of quality of his data will be
sufficient to meet the requirements of the intended use.
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page 14 of 33
3.7.2 Sample Collection
The QA factors relevant to sample collection must be explicity
considered in the QA Project Plan and provision should be made for keeping
records to substantiate that the procedures described in the QA Project Plan
actually were followed. These QA factors are:
Use of EPA acceptable sample collection and field measurement
methods
. Use of EPA acceptable field equipment and instruments.
Calibration of field instruments to within acceptable limits (EPA
or manufacturer's specification). Planned periodic inspection
and/or re-calibration of equipment should be performed as
necessary before, during, and after use in the field. Reference
standards should be used when appropriate. Permanent records
must be kept of all calibrations and inspections.
Planned periodic inspection, maintenance, and servicing of
equipment and instruments. This activity must be planned and
explicitly described in the QA Project Plan. Inspection must be
frequent enough to assure that instruments are not producing
faulty data due to some malfunction. Permanent records must be
kept so that "suspect" data (collected during a period when
instruments may have malfunctioned) can be flagged and excluded
from the study results.
t Use of EPA acceptable sample containers in order to prevent
contamination and assure an adequate sample size.
Use of EPA acceptable sample preservation methods and adherence
to recommended sample holding times.
t Collection of all important associated environmental and site
data (e.g., flow measurements, climatic data, media effects) by
EPA acceptable methods and instruments.
Use of EPA acceptable sample packaging and shipping procedures.
Use of EPA Acceptable sample identification and, as necessary,
formal chain-of-custody procedures in the field and during
shipment.
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page 15 of 33
o Collection of QC samples (e.g., field blanks, duplicate samples,
etc.) as needed for the laboratory analysis QA program.
o Recordkeeping sufficient to assure that QC procedures were
adhered to and that QC activities were conducted as planned in
the QA Project Plan.
3.7.3 Sample Custody
Sample custody is a part of any good laboratory or field operation. Where
samples may be needed for legal purposes, "chain-of-custody" procedures, as
defined by the Office of Enforcement, will be used. However, as a minimum,
the following sample custody procedures will be addressed in the QA Project
Plan:
A. Field Sampling Operations:
Documentation of procedures for preparation of
reagents or supplies which become an integral part
of the sample (e.g., filters, and absorbing
reagents).
Procedures and forms for recording the exact
location and specific considerations associated
with sample acquisition.
Documentation of specific sample preservation
methods.
t
Pre-prepared sample labels containing all
information necessary for effective sample
tracking. Figure 2 illustrates a typical sample
label applicable to this purpose.
Standardized field tracking reporting forms to
establish sample custody in the field prior to
shipment. Figure 3 presents a typical example
of a field-tracking report form.
B. Laboratory Operations:
Identification of responsible party to act as
sample custodian at the laboratory facility
authorized to sign for in coming field samples,
-------
Olii QA Project Plan
Revision No. 2
Date: 09/28/84
Page 16 of 33
(NAME OF SAMPLING ORGANIZATION)
SAMPLE DESCRIPTION
PLANT:
DATE:
MEDIA:
SAMPLE TYPE:
SAMPLED BY:
SAMPLE ID NO.:.
LAB NO.
LOCATlOtL.
TIME:
STATION;
PRESERVATIVE:
5
IU
FIGURE 2. EXAMPLE OF SAMPLE LABEL
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page 17 of 33
W/O No.
FIELD SAMPLE CODE
(FSC)
FIELD TRACKING REPORT:
PAGF
(LOC-SN)
DESCRIPTION
DATE
TIME
SAMPLER
FIGURE 3. EXAMPLE OF FIELD-TRACKING REPORT FORM
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page 18 of 33
obtain documents of shipment (E.G., bill of lading
number or mail receipt), and verify the data
entered onto the sample custody records.
Provision for a laboratory sample custody log
consisting of serially numbered standard lab-
tracking report sheets. A typical example of a
standardized lab-tracking report form is shown in
- Figure 4.
3.7.4 Laboratory Analysis Procedures
For each measurement parameter, including all pollutant measurement
systems, reference the applicable standard operating procedure (SOP) or pro-
vide a written description of the analytical procedure(s) to be used.
Officially approved EPA procedures will be used when available.
3.7.5 Internal Quality Control Checks
Describe and/or reference all specific internal quality control
("internal" refers to both laboratory and field activities) methods to be
followed. Examples of items to be considered include: :
Replicates
Spiked samples
Split samples
Control charts
Blanks
Internal standards
t Zero and span gases
Quality control samples
Surrogate samples
Calibration standards and devices
Reagent checks.
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page 19 of 33
W/O No.
PAGE
LAB TRACKING REPORT: - -
FRACTION
CODE
X
PREP. /ANAL.
REQUIRED
(LOC-SN-FSC)
RESPONSIBLE
INDIVIDUAL
DATE
DELIVERED
DATE
COMPLETED
FIGURE 4. EXAMPLE OF LAB-TRACKING REPORT FORM
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page 20 of 33
3.7.6 Performance and System Audits
Each QA Project Plan must describe the internal and external perfor-
mance and systems audits which will be required to monitor the capability and
performance of the total measurement system(s).
The systems audit consists of evaluation of all components of the
measurement systems to determine their proper selection and use. This audit
includes a careful evaluation of both field and laboratory quality control
procedures. Systems audits are normally performed prior to or shortly after
systems are operational; however, such audits should be performed on a
regularly scheduled basis during the lifetime of the project or continuing
operations. The on-site systems audit may be a requirement for formal
laboratory certification programs such as apply to laboratories analyzing
public drinking water systems. Instructions and forms for use in conducting
systems audits are found in Appendix B.
After systems are operational and generating data, performance
audits are conducted periodically to determine the accuracy of the total
measurement system(s) or component parts thereof. The plan should include a
schedule for conducting performance audits for each measurement parameter,
including a performance audit for all measurement systems. As part of the
performance audit process, laboratories may be required to participate in
analysis of performance evaluation samples related to specific projects.
Project plans should also indicate, where applicable, scheduled participation
1n all other inter-laboratory performance evaluation studies. Instructions
and forms for use in conducting performance audits are found in Appendix C.
3.8 Data Processing
Data processing encompasses all manipulations performed on raw ("as
collected") information to change its form of expression, its location, its
quantity, or its dimensionality. This includes data collection, validation,
storage, transfer, reduction, and analysis. The goal of QA in data processing
is to prevent errors and loss of data. Quality control of the data processing
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page 21 of 33
activity strives to faithfully.reproduce the information contained in the
original source data.
3.8.1 Collection
The QA Project Plan will address both manually collected and com-
puterized data acquisition systems. Manually collected data are frequently
monitored by the person recording the data. However, computerized data acqui-
sition systems do not have the potential for this treatment and are known to
pick up false voltage transmissions, which may introduce error. The internal
checks that must be used to ensure suitable quality in the data collection
process will be identified. Validation of raw data will also be addressed.
3.8.2 Validation
Data validation has been defined as "the process whereby data are
filtered and accepted or rejected based upon a set of criteria". In the
processing aspects of data validation, the QA Project Plan will clearly indi-
cate that raw data are not altered, and how a reduced data set is generated,
along with a clearly defined audit trail.
The validation process may include many forms of manual or computer-
ized checks, but it clearly involves specified criteria. Validation criteria
may include evaluation of the data with respect to physically determined
checks (e.g., a record indicating a negative weight is not reasonable).
Similarly, as the sophistication of the model increases, relational checks may
also be used. Built-in redundancy and cross-checks (e.g., the "check bit"
used widely in computer systems) are among the most powerful tools for con-
trolling data accuracy, and for allowing good validation later on. These
elements must be built into, or designed into, the system from the beginning.
Redundancy and cross-checks should be identified and addressed in the QA
Project Plan.
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page 22 of 33
3.8.3 Storage
Data storage involves keeping the data in such a way that they are
not degraded or compromised, and that any datum (value) desired may be found
(uniquely identified). For computerized raw data, there must always be at
least one copy that is off-line and not machine mounted. At every stage of
data processing in which a "permanent" collection of data is stored, there
will be a physically separate copy for purposes of integrity and security.
Project data must be securely stored (archived) in a suitable manner. Such
aspects as storage media, conditions, and location must be addressed. Access
by authorized personnel and retention time must also be addressed in the QA
Project Plan. Selected QA "labels", which indicate the level of quality of
the data point, should be stored along with each data point. For example,
when samples are analyzed by several different chemical methods, then the
particular method used for a specific sample should be part of the data record
which is stored for that sample.
Another aspect of data storage to be addressed is data inviola-
bility. Raw data must never be altered. If an error in data can be
demonstrated and the correct value(s) determined, then of course the data
should be corrected. However, such changes must be documented fully,
including date and reason for correction. For computerized data files, it is
advisable to keep a separate file containing notes fully documenting data
changes.
3.8.4 Transfer
As a rule data transfer should be kept to a minimum to prevent
errors. Each QA Project Plan will describe procedures which will be used to
characterize data transfer error rates and how information loss is minized in
the transfer. All data transfers, from raw data through final interpretation,
must be diagramrnatically indicated. Examples of data transfers are: copying
raw data from a notebook onto a data form for keypunching; converting a
written data set to punched cards; copying from computer tapes to disks; and
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page 23 of 33
telemetering. Human engineering principles will be used to reduce data
transfer errors. A well designed form for recording data reduces the chance
of errors. Well designed (i.e., human engineered), simplified procedures for
coding data reduce
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page 24 of 33
would result makes it impractical to save each datum. In such cases data
subgroups are sometimes summarized by statistics such as averages, standard
deviations, and sample sizes. In these cases the notion of raw data is
broadened so that the summary statistics are regarded as the raw data. A
preliminary study should address the adequacy of the candidate summary
statistics, in terms (of the end uses of the data. For instance, a variable
that can assume only positive values may have a distribution that is skewed to
the left. In some cases a logarithmic or square root transformation might
bring the distribution closer to normality. In this case the average of the
transformed data would come closer to satisfying standard statistical
assumptions than would the average of the raw data.
3.8.5.2 Software
The objective of software QA is to ensure that calculator and
computer programs perform accurately. Such operations should introduce no
more than negligible error (e.g., 1 percent or less) relative to the intrinsic
variation in the measured processes. For manual calculations, an example
should be given in which actual raw data are transformed and can be checked by
reviewers. If a programmable calculator is used in this process, a copy of
the programs used should be provided.
Computer programs should be designed to expedite validation.
Programs should be modular, sructured, well documented, logical, and should
liberally employ comment statements. The use of widely available statistical
analysis packages such as SAS, BMD, SPSS, and MINI-TAB is recommended, as
opposed to writing analysis programs in FORTRAN, BASIC, or PL/I code. Such
packages are heavily used; therefore errors have been largely eliminated, and
standard documentation is widely available.
The_following minimal documentation is sufficient for computerized
data manipulation or analysis:
MATERIAL BELONGS TO:11
US EPA TOXICS U3RARY
401 MST3W/TS-733
WASHINGTON, DC 20460
(202) 260-3944
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page 25 of 33
1. Reference to system documentation (some software
packages supply this automatically);
2. A copy of the calling program and resulting output;
3. A concise, clearly written description of the operand
data set and how it derives from the raw data and the
operation or analysis to be performed; these may be
embedded in the beginning of the program as a comment
statement;
4. Several examples of hand calculated data reduction
using the procedures from the computer program; and
5. A data dictionary defining the variables as they
pertain to the operation or analysis as described in
item 3 above; the data dictionary may be embedded
in 3.
Compliance with items 1 through 5 has no implications for validity
of the analyzed data or appropriateness of the statistical methodology
employed; they must each be addressed separately.
At any point in the data, one should be able to check for the
validity of the reduction from the raw data to the reduced data set. A
2-person, independent, spot check on the calculations should be performed.
Any corrective action taken during data reduction should be
documented.
3.8.5.3 Parameter Estimation and Hypothesis Testing
Statistical analysis usually involves the use of study data for
estimation of model parameters. These estimates may be put to a variety of
uses, depending on the study objectives required and the tastes of the data
analysts. If the objectives have been expressed as formal hypothesis (null
and alternative hypothesis), and the statistical tests may be based on
parameter estimates. If estimation rather than inference is the goal of the
study, then a confidence interval approach to summarization may be adopted.
In either case, the estimates are acknowledged to contain error, due to
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page 26 of 33
intrinsic variation and/or factors neglected by the model. Model assumptions
must be clearly stated and validated.
The approach of hypothesis testing is to ask whether the deviation
of an estimate from a given hypothesized value or range of values is plausibly
due to chance alone. If study data are sufficiently improbable in the light
of a hypothesis, then the hypothesis may not be further entertained.
Probabilities must also be dealt with, in an inverse sense, to construct a
confidence interval; i.e., a range which contains the true value of a
parameter with a certain probability. When tests of hypotheses are used, the
null and alternative hypotheses need to be listed. The exact alpha level for
the test should be stated when possible and if the null hypothesis is not
rejected, the power of .the test should be calculated when possible.
For the statistician/data analyst, questions of accuracy of mea-
surements become questions of accuracy of these probabilities. If the model
is correct, then so are the probabilities. This is an oversimplification,
since there is no such thing as a totally correct model. A model is a theo-
retical construction intended to represent reality. In most cases we have
only observations and lack the detailed schematic that would allow completed
evaluation of the model. This illustrates a common problem for the quality
assurance of scientific research: true reference values are not available,
and accuracy, in the sense of closeness to the correct value, cannot be
determined.
However, a well-defined model does have numeric consequences that
can be checked against the data. Such checks fall under the general category
of goodness-of-fit tests and are the statistician's main tool for the QC of
his own measurement process. As a general rule, if it can be seen from the
data that a model assumption is false, then corrective action is required.
Some specific techniques for checking agreement between the model
and data include inspection of plots observed and expected values or
residuals, goodness-of-fit tests for probability distributions overfitting,
and analysis of replicates. Goodness-of-fit tests include chi-squared tests
for probability densities or discrete distributions of specified form and the
Kolmogorov-Smirnov test for the cumulative distribution function. Overfitting
-------
OTS QA Project P.Ian
Revision No. 2
Date: 09/28/84
Page 27 of 33
involves viewing the tentative model as embedded in a larger family of models
and testing within the larger family whether or not there is significant
evidence against the restricted (tentative) model. Ideally the larger family
is defined on the basis of suspicion of how the tentative model might fail.
For instance, in calibration problems where a straight line through the origin
is sometimes assumed (y = ax + error), the model y = ax + b + error may be
used to represent certain types of departures. For regression problems,
repeated (replicate) observations at fixed conditions allow variance estima-
tion, independent of any model. Also, it is generally possible to estimate
variance as a consequence of the model. The two variance estimates should be
consistent if the model is approximately correct.
The statistical analysis and interpretation of results must be con-
sistent with the study design. QA Project Plans will address the reliability
of computations (software QA), appropriateness of the model(s) as a framework
for investigating the study questions, and robustness of statistical
procedures to model inaccuracies (methodological QA).
The QA Project Plan should address potential problems in the data
analysis scheme, and how it is anticipated that they will be resolved.
3.9 Data Quality Assessment
Each QA Project Plan must address the assessment of data quality.
The factors discussed below must be considered.
3.9.1 Precision
Each QA Project Plan will contain a mechanism for demonstrating the
reproducibility of each measurement process.
3.9.1.1 Replicate Samples
Replicate sample data shall be within predetermined acceptance
limits.
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page 28 of 33
3.9.1.1 Instrument Checks
Each measurement device shall have routine checks performed to demonstrate
that variables are within predetermined acceptance limits. Examples of checks
Include:
A. zero and span;
B. noise levels;
C. drift;
D. flow rate; and
E. linearity.
3.9.2 Accuracy
Each QA Project Plan will contain a mechanism for demonstrating the
relationship of the reported data compared to the "true" value(s).
3.9.2.1 Traceability of Instrumentation
Each measurement device will be assigned a unique identification
number. Documentation shall identify the specific measurement device, where
and when used, maintenance performed, and the equipment and standards used for
calibration.
3/9.2.2 Traceability of Standards
Each standard and each measurement device will be calibrated against
a standard of known and higher accuracy. All calibration standards will be
traceable to available National Bureau of Standards (NBS) Standards. If NBS
standards are not available, other validated (primary) standards will be used.
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page 29 of 33
3.9.2.3 Traceability of Samples
When samples are extracted from a test system, each sample will be
assigned a unique identification number. Documentation shall identify
sampling time, place, and action taken on each sample.
3.9.2.4 Traceability of Data
Data will be documented to allow complete reconstruction, from
initial field records through data storage and system retrieval.
3.9.2.5. Methodology
If available, Federal reference, equivalent, or approved alternate
test methods will be used.
3.9.2.6 Reference or Spiked Samples
Recoveries shall be within predetermined acceptance limits.
3.9.2.7 Performance Audits
Each measurement project will participate in the EPA Performance
Audit Programs. Each project will develop a system of internal performance
audits to demonstrate that all measurements are within acceptable, predefined
control limits.
3.9.3 Representativeness
Each QA Project Plan will contain procedures to ensure and document
that each sample collected represents the medium sampled insofar as is poss-
ible. Parameters relevant to this aspect of data quality will be specified
(e.g., storage temperature) and recorded as part of the raw data.
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page 30 of 33
3.9.4 Comparability
Each QA Project Plan will contain procedures to assure the
comparability of data. Examples are:
Consistency of reporting units
Standardized siting, sampling, and analysis
Standardized data format.
3.9.5 Completeness
For projects where it is relevant, the QA Project Plan will identify
the quantity of data needed to meet the needs of the project, and percent
recovery required.
3.10 Corrective Action
Each QA Project Plan will include provisions for written require-
ments establishing and maintaining QA reporting and feed back channels to the
appropriate management authority, in order to ensure that early and effective
corrective action can be taken when data quality falls below required, speci-
fied limits. Each QA Project Plan will also include provisions to keep
responsible management informed of the performance of all data collection
systems. Each QA Project Plan will describe the mechanism(s) to be used when
corrective actions are necessary. Corrective action shall relate to the over-
all QA management scheme: who is responsible for taking corrective actions;
when corrective actions are to be taken; and who ensures corrective actions
are taken and produce the desired results.
Corrective action shall be minimized through the development and
implementation of routine internal program controls prior to an adverse
program inpact. Examples of controls include the following:
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page 31 of 33
Each measurement system (e.g., instrument) shall have
redetermined limits to indicate when corrective action
is required, before data become unacceptable
A procedure shall be established for each measure ment
system to identify the corrective action which will be
taken when the warning or control limits are exceeded
0 For each measurement system, the level within the
organization responsible for taking corrective action,
and also the level within the organization responsible
for approving corrective action, shall be clearly
stated
The responsible quality assurance officer shall be
informed of any major corrective action taken, of any
changes in procedure, and of any loss of data which
results from a change in procedure.
Results of the following QA activities may also initiate the following
actions:
Performance audits
Systems audits
Inter!aboratory comparison studies.
3.11 Documentation and Reporting
3.11.1 Documentation
Describe how documentation of all project results will be achieved,
so that conclusions based on the results can be adequately supported.
3.11.2 Quality Assurance Reports to Management
QA Project Plans should provide a mechanism for periodic reporting
to management on the performance of measurement systems and data quality. As
a minimum, these reports should include:
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page 32 of 33
Periodic assessment of measurement data accuracy,
precision and completeness
t Results of performance audits
Results of system audits
Significant QA problems and recommended solutions.
It is the responsibility of the project officer and the QA officer
to establish QA reporting requirements for individual extramural projects
prior to project initiation. For intramural tasks, QA reporting requirements
must be included in each QA Project Plan. These requirements should be estab-
lished by the appropriate functional manager (i.e., division director, branch
or section chief, or group leader) in consultation with the QA officer.
Projects of short duration (1 year or less) may require only a final
QA report. Projects of longer-duration may require periodic (e.g., quarterly)
QA reports.
The individual(s) responsible for preparing the periodic reports
should be identified. The final report for each project just include a separ-
ate QA section which summarizes data quality information contained in the
periodic reports.
Reports should conform to the OTS Manual for Preparing Documents.
July 29, 1981, U.S. Environmental Protection Agency, Office of Toxic
Substances.
3.12 References
All documents cited in the QA Project Plan should be refer enced; reference
citations should conform to the OTS Manual for Preparing Documents. July 29,
1981, U.S. Environmental Protection Agency Office of Toxic Substances.
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page 33 of 33
REFERENCES
1. U.S. Environmental Protection Agency, Environmental Protection Agency
(EPA) Quality Assurance Policy Statement. Administrator's Memorandum,
May 30, 1979.
2. U.S. Environmental Protection Agency, Quality Assurance Requirements for
All EPA Extramural Projects Involving Environmental Measurements,
Administrator's Memorandum, June 14, 1979.
3. U.S. Environmental Protection Agency, Quality Assurance Program Plan for
the Office of Toxic Substances. Office of Pesticide and Toxic Substances,
Washington, DC, September 30, 1983.
4. U.S. Environmental Protection Agency, Interim Guidelines and Specifica-
tions for Preparing Quality Assurance Project Plans, QAMS-005/80, Office
of Research and Development, Washington, DC, December 1980.
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page A-l of A-4
APPENDIX A
GLOSSARY OF TERMS
Accuracy; The degree of agreement of a measurement (or an average of measure-
ments of the same thing), X, with an accepted reference or true value, T, usu-
ally expressed as the difference between the two values, X-T, or the
difference as a percentage of the reference or true value, 100 (X-T)/T, and
sometimes expressed as a ratio, X/T. Accuracy is a measure of the bias in a
system.
Audit; A systematic check to determine the quality of operation of some func-
tion or activity. Audits may be of two basic types: (1) performance audits
in which quantitative data are independently obtained for comparison with
routinely obtained data in a measurement system; or (2) system audits of a
qualitative nature that consist of an on-site review of a laboratory's quality
assurance system and physical facilities for sampling, calibration, and
measurement.
Batch; A specific quantity or lot of a test or control substance.
Blanks; In chemical analysis, the blank is often a pure sample component
(e.g., distilled water) that does not give a positive measured response.
Chemical blanks can be classified as reagent blanks or total method blanks
(includes all reagents in the quantity required by the method). In biological
experiments, the term control is analogous to the method or analytical blank
described for chemical analysis. The control is an organism or part of an
organism that is handled in a manner identical to the experimental group,
except that it does not receive the treatment hypothesized to cause the
response of interest.
Comparability; The confidence with which one data set can be compared to
another.
Completeness; A measure of the amount of valid data obtain ed from a measure-
ment system compared to the amount that was expected to be obtained under
correct normal conditions.
Cooperative Analyses; Round-robin type analyses, which are useful for
estimating the precision of a measurement among several different operators
and/or laboratories. Accuracy of the measurement can only be assessed if the
analyte is a reference material.
Data Quality; The totality of features and characteristics of data that bears
on its ability to satisfy a given purpose. The characteristics of major
importance are accuracy, precision, completeness, representativeness, and
comparability.
-------
OTS QA rYoject Plan
Revision No. 2
Date: 09/28/84
Page A-2 of A-4
Data Validation; A systematic process for reviewing a body of data against a
set of criteria to provide assurance that the data are adequate for their
intended use. Data validation consists of data editing, screening, checking,
auditing, verification, certification, and review.
Good Laboratory Practice (GLP): The organizational process and the conditions
under which studies are planned, performed, monitored, recorded, and reported.
Environmental or Environmentally Related Measurement: The term "environmental
(or environmentally related) measurement" applies to all field and laboratory
investigations that generate data involving the measurement of chemical,
physical, or biological parameters in the environment, such as determining the
presence or absence of priority pollutants in waste streams; health and
ecological effects studies; clinical and epidemiological investigations;
engineering and process evaluations; studies involving laboratory simulation
of environmental events; studies involving laboratory simulation of environ-
mental events; and studies or measurements on pollutant transport and fate,
including diffusion models.
Internal Standard; A species different from, but very similar in analytical
response to, the analyte of interest. The method of internal standards is
often employed in the analysis of environmental or biological samples to com-
pensate for possible matrix effects. In this method, the analyte of interest
is added in a fixed and known amount to all samples and standards analyzed
(including calibration standards). The analytical response-of both the ana-
lyte of interest (R/\) and of the internal st andard (Rjs) is recorded for each
sample or standard and the ratio of these responses (i.e., RA/RIS) is used to
obta in quantitative estimates of the amount of analyte present. The basic
assumption required is that matrix effects and minor experimental variations
will influence the analytical responses of the analyte and the internal
standard in a very similar, if not identical, manner.
Inspections: Generally, visits to the sponsor or laboratory involved. A
laboratory inspection involves an actual, physical view of current laboratory .
facilities and operations and past record keeping operations.
Performance Audits; Procedures used to determine quantitatively the accuracy
of the total measurement system or component parts thereof.
Precision; A measure of mutual agreement among individual measurements of the
same property, usually under prescribed similar conditions. Precision is test
expressed in terms of the standard deviation. Various measures of precision
exist depending upon the "prescribed similar conditions."
Protocol; A detailed description of the study design and conduct to be
followed.
Quality Assurance: The total integrated program for assuring the reliability
of monitoring and measurement data. A system for integrating the planning,
-------
UTS WM Project Plan
Revision No. 2
Date: 09/28/84
Page A-3 of A-4
assessment, record keeping, and improvement efforts to meet user requirements.
Quality assurance procedures are often conducted by an external source on a
random sample of the data.
Quality Assurance Program Plan; A QA Program Plan is a written document that
presents in general terms the overall policies, organization, objectives, and
functional responsibilities (within the organization) designed to achieve
specified data quality goals of a particular organization (e.g., EPA
Laboratory, Program Office, Regional Office, contracting organization).
Quality Assurance Project Plan; A QA Project Plan is a written document that
details the policies, organization, objectives, functional activities, and
specific QA and QC activities designed to achieve data quality goals or
requirements of a specific measurement program. The QA Project Plan must
address procedures used to routinely assess precision, accuracy, completeness,
representativeness, and comparability of the data produced.
Quality Assurance Officer; The QA officer is that individual who is assigned
the responsibility for overview and guidance of the QA Program for an organiz-
ation or for a specific project. Organizationally, the QA officer should be
in a position to provide independent and objective evaluation and assessment
of the effectiveness of the QA Program and to provide timely feedback and
recommendations.
Quality Control; The routine application of procedures for obtaining
prescribed standards of performance in the monitoring and measurement process.
Quality Control Samples; Samples containing known and verified concentrations
of the analyte of interest that are prepared independent of calibration
standards and are analyzed at frequent intervals throughout routine analysis.
Raw Data; Any laboratory worksheets, records, memoranda, notes, or exact
copies thereof, that are the result of original observations and activities of
a study and are necessary for the reconstruction and evaluation of the report
of that study. In the event that exact transcripts of raw data have been pre-
pared (e.g., tapes which have been transcribed verbatim, dated and verified by
signature), the exact copy of the exact transcript may be substituted for the
original source of raw data. "Raw data" may include photographs, microfilm or
microfiche copies, computer printouts, magnetic media, including dictated
observations, recorded data from automated instruments, and correspondence
relating to the planning, conduct, and interpretation of the study (44 FR
27370, 1979).
Replicates; Repeated but independent measurements of the same sample by the
same analyst at essentially the same time and under the same conditions.
Representativeness; The degree to which data accurately and precisely
represent a characteristic of a population, parameter variations at a sampling
point, a process condition, or an environmental condition.
-------
u ^ QA iYwjcCC t1 Idii
Revision No. 2
Date: 09/28/84
Page A-4 of A-4
Specimen; Any material derived from a test system for examination, analysis
or storage.
Spiked Samples; Environmental samples to which a known quantity of a given
analyte has been added in order to evaluate matrix effects. When spiked sam-
ples are analyzed concurrently with correspondingly unaltered samples, it is
possible to determine if there are components in a sample that bias measure-
ment values. Results of such analyses are often expressed as percent recovery
and may be used to correct unaltered sample results to obtain "true" or
correct values.
Split Samples; A quantity of homogeneous material that is split into two or
more portions which are analyzed independently.
Standard Operating Procedure (SOP); A written document which details an oper-
ation, analysis or action whose mechanisms are thoroughly prescribed and which
is commonly accepted as the method for performing certain routine or repeti-
tive tasks. SOP's can be written for test protocols (procedures) or equipment
operation methods.
Study Plan; A document which defines the entire scope of the study including
a detailed description of the methodology (protocol) to be used in the study.
Surrogate Sample; A second variable that is easier to measure than the vari-
able of interest, where a relationship (ratio) can be established between the
two variables. The second variable is measured as a surrogate sample where it
is impossible or prohibitively expensive to measure the variable of interest.
Systems Audit; A systematic, on-site, qualitative review of facilities,
equipment, training, procedures, record keeping, data validation, data manage-
ment, and reporting aspects of t he total measurement system. A QA systems
audit may be required: (1) to assess, prior to project initiation, the capa-
bility of a measurement system to generate data of the required quality; or
(2) to determine compliance of an ongoing project with specified QA
requirements.
Text Mixture; A combination which results from mixing a test substance with
another substance or substances, including vehicle, dust-suppressant, feed,
water, etc., for the purpose of exposing the test system to the test
substance.
Test Substance; The specific chemical substance or mixture that is used to
develop data by exposure to the test system.
Test System; The apparatus, equipment, and any animal, plant, microorganism,
or subparts thereof, to which the test or control substance or mixture is
administered or added for study. "Test System" also includes appropriate
groups or components of the system not treated with the test or control
substance or mixture.
-------
OTS QA Project rian
Revision No. 2
Date: 09/28/84
Page B-l of B-17
APPENDIX B
SYSTEMS AUDITS
A systems audit consists of a qualitative evaluation of all
components of the measurement system to assure their proper selection and use.
The objective of the on-site qualitative systems audit is to assess and
document: facilities; equipment; personnel; record keeping; data validation
and management; operation, maintenance and calibration procedures; and
reporting aspects of the total QC program for a project. The review should:
Identify existing system documentation, i.e.,
maintenance manuals, organizational structure,
operating procedures.
0 Evaluate the adequacy of the procedures as documented.
Evaluate the degree of use of and adherence to the
documented procedures in day-to-day operations, based
on observed conditions and a review of applicable
records on file.
From a qualitative review of the measurement system, an auditor
independent of the task organization can assess the suitability of the
facilities and operations to meet project goals and identify specific areas
where corrective actions may be implemented.
Accordingly, the following forms are designed to address all the
major components of the measurement system, including personnel qualifica-
tions; facilities and equipment; data generation; data management; reporting;
and quality control. Completion of the forms by an auditor should provide an
objective and unbiased evaluation of the total measurement system, from the
Initiation of the project to the reporting of the results. These forms also
provide a means to standardize a systems audit so that interlaboratory com-
parisons or comparisons among studies performed in the same laboratory should
be possible. It is recommended that a systems audit be conducted- at all
testing laboratories on an annual basis.
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
. Page B-2 of B-17
The forms are meant to be general enough so that the questions are
applicable to any laboratory study. It will not be possible to ask extremely
specific questions concerning a specialized test without the use of test-
specific audit forms. For example, there are questions important to the
proper completion of a Salmonella reverse mutagenesij assay that will not
pertain to any other laboratory study. These issues can be addressed most
effectively by the use of test specific forms. Nevertheless, the general
systems audit forms provided here do address the use of laboratory standard
operating procedures (SOPs) for individual studies, and in this way introduce
some specificity to the general systems audit.
It is important to recognize that a systems audit is distinctly
different from a performance, data, or a good laboratory practices (GLP)
audit. A performance audit is designed to quantitatively evaluate the data
generation process, while the purpose of a data audit is to ascertain that all
data are arithmetically correct and have been handled in a mathematically
acceptable manner. Finally, a GLP audit (or laboratory inspection) is a check
to be certain that GLP regulations promulgated by EPA have been followed
throughout the laboratory study.
Audit Preparation
In preparing to conduct a systems audit, the auditor(s) should at a
minimum:
1. Read all submitted laboratory reports on the study
being audited
2. Contact appropriate EPA personnel responsible for the
test substance and its activity (e.g., carcinogenic,
teratogenic, toxic, etc.) for which the study is being
performed
3. Access and study results of any other audit reports
concerning the study (e.g., GLP audits, data audits,
etc.)
4. Read methods papers concerning the study.
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page 8-3 of B-17
5. Become familiar with Office of Toxic Substances
guidelines concerning testing laboratory operations.
6. Become familiar with the test system being used in the
study.
Information obtained prior to the audit should be confirmed during the on-site
systems audit.
Audit Report
The final audit report should contain, in addition to the completed
attached forms, a summary of audit findings identified in the audit. This
summary should be attached as the final page of the audit report. The summary
should contain names and titles of technical staff, Quality Assurance (QA)
representatives and other laboratory personnel involved in the study being
audited. The auditor may also identify information that was not available
during the audit and any detected irregularities that are not necessarily
within the scope of the audit being performed.
Debriefing
Immediately following the audit, the auditor should debrief the
laboratory on the overall results of the audit. It is recommended that the
auditor point out those aspects of the laboratory operations that are com-
mendable, as well as deficiencies uncovered during the audit. If the auditor
identifies irregularities that are beyond the scope of the particular audit
being performed, he or she should inform the laboratory of these irregu-
larities and that the appropriate authority will be notified.
The audit form resulting from the audit should be reviewed by the
laboratory, and the auditor should request the laboratory contact to sign and
date the form under a statement that they have found the report to be true and
accurate. The auditor should also sign the report.
-------
program:
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page B-4 of B-17
Training
The following are recommended components of an auditor training
1. A short training course addressing at a minimum:
a. Audit preparation and scheduling
b. Safety during the audit
c. Handling proprietary/confidential information
d. Laboratory accessis it necessary or not?
e. General audit procedures
f. Legalities involved in conducting an audit
g. How the auditor may be perceived by the auditee
h. Use and content of audit forms
i. Discretionary responsibilities of the auditor
j. Performance of necessary calculations
k. Test procedure being audited.
2. Participation in a minimum of two audits accompanied
by experienced auditors is recommended. It is also
recommended that personnel trained for conducting
audits have at a minimum a Bachelor's Degree in a
scientific area, and 2 years of experience in general
laboratory procedures.
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page B-5 of B-17
Audit Forms for a System Audit
Section I. Basic Study Information
When possible, the following information should be obtained in advance of
the audit being schedule.
A. Auditor Information:
1. Name(s)/Affiliation:
2. Date of Audit: , 19_
B. Testing Lab Information:
1. Laboratory Name:
2. Laboratory Address:
3. Laboratory Phone No.: ( ) -_
4. Laboratory Contact: Dr./Mr./Ms._
5. Principal Investigators:
6. Compound Code or Chemical Tested:
7. Physical Description of Compound or Chemical Tested:
8. Type of Study:
9. Date of Initiation:
10. Projected Date of Completion:
Sponsor Information:
1. Sponsor Name:
2. Sponsor Address:
3. Sponsor Contact:
4. Sponsor Phone:
-------
II. Personnel Qualifications
A. Principal Investigator (PI) Comments/Responses
1. What 1s the specific training and/or experience of the PI?
2. What percentage of time has the PI allocated to this particular study?
B. Technical Staff
1. What training and/or experience qualifies the technical staff to do
this study?
C. Other Staff
1. Briefly describe the personnel qualifications (training/experience)
for any others involved in the completion of this study, such as
laboratory supervisors, animal care technicians, staticians, etc?
D. Quality Assurance Unit (QAU)
1. Briefly describe the personnel qualifications (training/experience)
of the QAU Staff.
E. Technical Review Staff
1. If there is an Internal review staff apart from the PI or QAU, briefly
describe the personnel qualifications of the internal reviewers.
-------
Comments/Responses
F. Management Structure
1. Is there a management struture chart of the performing laboratory
available for inspection?
Yes No
2. How often does the PI interact with the technical staff to review
study progress?
3. How often does the PI interact with immediately higher management
to review study progress?
-------
III. Facilities and Equipment
The following questions pertain to adequacy and maintenance of facilities and equipment. Several of these
questions may have been addressed previously during a laboratory inspection. Because of their importance to
the proper functioning of a laboratory, they have been included here to cover those situations where a
laboratory inspection has not been conducted.
A. Facilities Operations Comments/Responses
1. Have the facilities operations methods been subjected to a laboratory
inspection?
Yes No
2. Are facillatles Inspected regularly for cleanliness, safety,
maintenance, etc.?
Yes No
3. Are the results of these inspections documented in writing?
Yes No N/A
4. Are special facilities or areas designated for the handling of
hazardous or radioactive materials?
Yes No N/A
5. Do facilities appear adequate with respect to lighting, space, well
equipped work areas, storage areas, safety equipment, etc. to perform
the study 1n a safe and efficient manner?
Yes No
-------
Comments/Responses
6. Describe any slgnlflcnt defects In facilities which may Impair the
successful completion of the project.
7. Are there written and approved standard operating procedures available
for performing routine laboratory functions?
Yes No
8. Are laboratory operations conducted so as to comply with GLP
regulations?
Yes No
B. Equipment Operations
1. Has a laboratory Inspection been conducted with respect to equipment
operations?
Yes No
2. Are instruments calibrated and serviced periodically?
Yes No
3. Is there a written calibration and service record for laboratory
instruments?
Yes No
-------
Comments/Responses
4. Are there written standard operating procedures available for each
piece of equipment which describe operation, maintenance schedule,
cleaning procedures, etc.?
Yes No
5. Have standard operation procedures been approved by the responsible
investigator as well as QA and management personnel?
Yes No
6. Is there a procedure for Insuring that laboratory technicians are
trained to operate equipment?
Yes No
-------
of the Laboratory Study
A. Protocols Employed Comments/Responses
1. What protocol will be/1s used in this study?
2. Was the protocol agreed upon by the sponsor, EPA, PI, and QAU?
Yes No
3. Briefly describe any modification or amendments to the protocols that
were implemented during the study.
N/A
4. Were amendments approved by EPA prior to their use?
Yes No N/A
If "yes" attach a copy of the documentation.
5. If amendments were incorporated, were they documented in the
laboratory records at the time the modifications were made?
Yes No N/A
B. General Qualitative Performance of the Study
1. Are/were the objectives of the study clearly documented from the start?
Yes No
-------
Comments/Responses
2. Was the study Initiated on schedule?
Yes No
3. Was the study completed on schedule?
Yes No
4. Was the targeted completion date extended during the study?
Yes No
5. If yes, was the extension necessary because of problems with the
protocol?
Yes No N/A
6. Were these problems corrected?
Yes No N/A
7. Is there a written record of these problems and the corrective
actions taken?
Yes No N/A
8. Are there field/laboratory SOPs for handling the test system?
Yes No
-------
Comments/Responses
9. Have field/laboratory SOPs been approved by the responsible
investigator as well as QA and management personnel?
Yes No N/A
10. Were these followed and documented?
Yes No N/A
11. Were specimens from the test system collected according to an SOP?
Yes No N/A .
12. Were all specimens from the test system labelled?
Yes No N/A
13. Is there documentation that traces the test system from its arrival
at the laboratory to the completion of the test?
Yes No N/A
14. Are the sources of the test system documented?
Yes No N/A
15 Are the dates of arrival documented?
Yes No N/A
-------
Comments/Responses
16. Will there be/was there a pre-test examination of the test system?
Yes No
17. What was the duration of the acclimation period for the test system?
18. Are there duly approved SOPs for handling the test compounds?
Yes No
19. If carriers are required during the handling of the test compound:
Are carrier/test compound mixtures handled accroding to approved
SOPs?
Yes No N/A
Are all compounds and carriers labelled?
Yes No N/A
20 Are concentration of materials and reagents verified by chemical
analysis?
Yes No
Attach a written documenation of the verification.
21. Is analytical chemistry performed according to duly approved
laboratory SOPS?
Yes No N/A
-------
Comments/Responses
22. Are the chemical characteristics of the test compound described 1n
the laboratory records?
Yes No
23. , Is the stability of the test compound monitored throughout the study?
Yes No N/A
24. If a carrier Is used, 1s the stability of the mixture verified?
Yes No N/A
25. If there were deviations from GLP procedures:
Attach a written record of the deviations from GLP.
Were the deviations approved by the PI, QAU, and the sponsor?
Yes No N/A
26. Are all raw data recorded Immediately 1n the permanent laboratory
records?
Yes No N/A
27. If any raw data were omitted, are omitted >«< data available for
inspection?
Yes No N/A
-------
Comments/Responses
28. Are data traceable through the course of the study?
Yes No N/A
29. At the completion of the study, are clean-up/decontamination
procedures employed according to GLP?
Yes No
C. Data Management
1. If raw data are/were subject to any form of data reduction:
Will/did a qualified statistician reduce the data or recommend a
data reduction method?
Yes No N/A
Do the laboratory records contain copies of the reduced data, as
well as citations for the methods employed?
Yes No N/A
Are all the reduced data clearly and directly traceable to the raw
data in the laboratory record books?
Yes J No N/A
2. Is there a permanent facility for storing all data forms, including
raw and reduced data?
Yes No
-------
Comments/Responses
3. Is there an Individual responsible for maintaining the data archives?
Yes No
0. Quality Assurance/Quality Control
1. Is the study conducted under QAU supervision?
Yes No
2. How often is the study monitored by QAU?
3. If there were any violations or deviations cited by the QAU:
Were corrective actions recommended? Yes No N/A
Were corrective actions initiatied? Yes No N/A
Were corrective actions approved by the QAU? Yes No N/A
-------
QM 1 ,'Ojcul HI an
Revision No. 2
Date: 09/28/84
Page C-l of C-13
APPENDIX C
PERFORMANCE AUDITS
Performance audits consist of a quantitative evaluation of the
laboratory data generation activities in order to determine the accuracy of
the total measurement system or its component parts. Quantitative measure-
ments and comparisons can provide objective estimates of data quality. In a
performance audit, laboratory operations are evaluated by the use of several
possible checks, including:
1. Reference materials, for accuracy determinations,
which are available from several sources, most notably
the National Bureau of Standards. These may be
included for analysis in various types of measurement
systems at relatively low cost with little inter-
ference to the normal laboratory routine and with the
highest possible degree of confidence.
2. Reference devices, for which the critical parameters
are known to the auditor but not the analyst. These
may be more disruptive of laboratory operations and
there is no possibility of anonymity of the sample;
however, the final result is still a measure of the
performance of the total analytical system, including
the operator.
3. Cooperative analyses, such as round-robin analyses,
which are useful for estimating the precision of a
measurement among several different operators and/or
laboratories. Accuracy of the measurement can be
assessed only if the analyte is a reference material.
4. Side-by-side analyses, or collaborative analyses,
which may be used if important variables are not
controllable in the sample*.
In addition, there are specific internal QC checks that may be employed within
the sampling or analytical process to determine the bias and imprecision of
specific project methodology. These are described below.
-------
UTS QA Prujevit Plan
Revision No. 2
Date: 09/28/84
Page C-2 of C-13
ReplicatesRepeated but independent measurements of
the same sample by the same analyst at essentially the
same time and under the same conditions.
t Spiked samplesEnvironmental samples to which a known
quantity of a given analyte has been added in order to
evaluate matrix effects. When spiked samples are
analyzed concurrently with correspondingly unaltered
samples, it is possible to determine if there are
components in a sample that bias measurement values.
Results of such analyses often are expressed as percent
recovery and may be used to correct unaltered sample
results, to obtain "true" or correct values.
Split samplesTwo or more portions of a homogeneous
material and which are analyzed independently. The
split-sample technique may be used to determine the
variance between observations (replicability), between
analysts or instruments in a laboratory (intralabora-
tory variance), between laboratories (interlaboratory
variance), or to determine comparability of different
analytical methods. This technique is employed often
when there are no reference materials available; in
such cases, the mean of reported results, recalculated
after exclusion of outliers and technically flawed
data, can be represented as the "target" or reference
value, which is. then used to determine the bias of
individual reported values.
BlanksA pure sample component that does not give a
positive measured response (e.g., distilled water), and
thus indicates the baseline or nonperturbed state.
Chemical blanks can be classified as reagent blanks or
total method blanks.
Reagent blanksThe background of each of the reagents
used in a given method of analysis must be determined.
The conditions for determining the background must be
identical to those used throughout the analysis,
including the detection system. If the reagents are
found to contain substances that interfere with a
particular analysis, they should be treated to remove
interferences or other satisfactory reagents must be
found.
Method or analytical blanksAfter determining the
individual reagent blanks, it must be determined if the
cumulative blank interferes with the analyses. Deter-
mination of a method blank is accomplished by following
-------
OTS QA HrojtCA i-;»;,,
Revision No. 2
Date: 09/28/84
Page C-3 of C-13
the normal analytical procedure, step by step, includ-
ing all of the reagents in the quantity required by the
method. If the cumulative blank intereferes with the
determination, steps must be taken to eliminate or
reduce the interference to a level that will permit
this combination of reagents to be used. If the
interferent cannot be eliminated, the magnitude of the
interference must be corrected for when calculating the
concentration of specific constituents in the samples
being analyzed.
Internal standardsA material different from but
similar in analytical response to the material of
interest, and added to all samples and standards
analyzed, including calibration standards. The method
compensates for possible matrix effects and minimizes
errors due to minor fluctuations or variations in
experimental conditions between individual analyses
(e.g., instrumental or electronic instability, changes
in environmental conditions, minor variations in
instrumental settings or configuration). Examples of
appropriate internal standard materials include chemi-
cal or biological samples that have been thoroughly
characterized by repeated, independent analyses;
chemical or biological materials validated against
known reference materials; and validated reference
materials such as those available from the National
Bureau of Standards.
Quality control samplesSamples containing known and
verified concentrations of the analyte of interest that
are prepared independent of calibration standards and
are analyzed at frequent intervals throughout routine
analysis. Such analyses provide continuous internal
evaluation of the measurement process. If practical
considerations do not prohibit the practice, QC samples
should be inserted by laboratory management into the
usual routine analytical stream of samples without the
knowledge of the analyst(s). In this way, more objec-
tive evaluation of everyday performance can be assured.
t Surrogate samplesA more easily measured variable than
the variable of interest, yet having an established
relationship (e.g., ratio) to the variable of interest.
The use of surrogate samples may permit more cost-
effective measurements of analytical performance com-
pared to the analysis of the variable of interest.
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page C-4 of C-13
Calibration standardStandard reference materials,
such as those available from the National Bureau of
Standards, that can be used to determine the detection
capabilities of an instrument; and to establish the
relationship between the output of the measurement
system and that of a known input.
Control chartsRepeated measurements of the same mate-
rial, such as reference materials or replicate measure-
ments of similar samples, presented for QC purposes as
charts of mean values and dispersion (e.g., ranges of
values) plotted against time. Deviations far from an
established .target valve indicate deficiencies in
analytical performance.
Reagent checksThe establishment that all chemicals,
equipment, and organisms are of a consistent and high
quality to meet program objectives.
The quantitative nature of a performance audit requires a more
rigorous effort than that of a systems audit. To facilitate the performance
audit, a set of forms has been developed containing questions germane to high
quality data generation. These questions were designed to be general enough
to apply to nearly any laboratory study and thus may not address specific
needs of a particular study. However, the forms do focus on analytical
techniques in general and address questions concerning the various indicators
of performance outlined above. In addition to the specific quantisation
checks for audit purposes, some of the questions deal with routine data
generation and management activities in order to better assess the
effectiveness of the measurement system.
Audit Preparation
Performance audits should be conducted by personnel knowledgeable in
the analytical and measurement techniques being utilized in the study. In
preparing to conduct a performance audit, the auditor(s) should at a minimum:
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page C-5 of C-13
1. Read any submitted laboratory reports on the study
being audited.
2. Contact appropriate EPA personnel responsible for the
chemical and its activity (e.g., carcinogenic, terato-
togenic, toxicity, etc.) for which the study is being
performed.
3. Obtain all required standard reference materials to be
submitted to the audited laboratory for performance
evaluations.
4. Access and study results of other audit reports
concerning the study, such as systems, data, or GLP
audits.
5. Read methods papers concerning the measurement system
involved in the study.
6. Become familiar with Office of Toxic Substance
Guidelines concerning testing laboratory operations.
Any information obtained prior to the audit should be confirmed during the on-
site performance audit.
Audit Report
The final performance audit report should contain, in addition to the
completed attached forms, a summary of audit findings identified 1n the audit.
This summary should be attached as the final page of the audit report. The
summary should contain names and titles of technical staff, Quality Assurance
(QA) representatives and other laboratory personnel involved in the study
being audited. The auditor may also identify information that was not
available during the audit and any detected irregularities that are not
necessarily within the scope of the audit being performed.
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page C-6 of C-13
Debriefing
Immediately following the audit, the auditor should debrief the
laboratory on the overall results of the performance audit. It is recommended
that the auditor point out those aspects of the laboratory operations that are
commendable, as well as deficiencies uncovered during the audit. If the audi-
tor identifies irregularities that are beyond the scope of the performance
audit, he or she should inform the laboratory of these irregularities and that
the appropriate authority will be notified.
The audit form resulting from the audit should be reviewed by the
laboratory, and the auditor should request the laboratory contact to sign and
date the form under a statement that they have found the report to be true and
accurate. The auditor should also sign the report.
Training
program:
The following are recommended components of an auditor training
1. A short training course addressing at a minimum:
a. Audit preparation and scheduling
b. Safety during the audit
c. Handling proprietary/confidential information
d. Laboratory, accessis it necessary or not?
e. General audit procedures
f. Legalities involved in conducting an audit
g. How the auditor may be perceived by the auditee
h. Use and content of audit forms
*
i. Discretionary responsibilities of the auditor
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page C-7 of C-13
2.
j. Performance of necessary calculations -
k. Test procedure being audited.
Participation in a minimum of two audits accompanied
by experienced auditors is recommended. It is also
recommended th at personnel trained for conducting
audits have at a minimum a Bachelor's
scientific area, and preferably 2 years of
ence in general laboratory procedures and
measurement system(s) being audited.
Degree in a
experi-
in the
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page C-8 of C-13
Audit Forms for a Performance Audit
Section I. Basic Audit Information
When possible, the following information should be obtained in advance of
the audit being schedule.
A. Auditor Information:
1. Name(s)/Affiliation:
2. Date of Audit: , 19_
B. Testing Lab Information:
1. Laboratory Name:
2. Laboratory Address:
3. Laboratory Phone No.: ( ) -_
4. Laboratory Contact: Dr./Mr./Ms._
5. Principal Investigators:
6. Compound Code or Chemical Tested:
7. Physical Description of Compound or Chemical Tested:
8. Type of Measurement System:
9. Date of Initiation:
10. Projected Date of Completion:
-------
C. Sponsor Informaton:
1. Sponsor Name:
Section II. Protocols Employed
A. Protocols
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page C-9 of C-13
2.
3.
4.
Sponsor
Sponsor
Sponsor
Address:
Contact:
Phone No.: ( )
1. What protocols are used in this study?
2. Was the protocol agreed upon by EPA, the principal investigator
(PI), and the quality Assurance Unit (QAU)?
Yes
No
3. Describe any amendments or modifications to the protocols that
have been implemented during the study. Pay particular attention
to those changes that may have affected quantitative output.
4. Were amendments approved by EPA prior to their use?
Yes No N/A
(Attach documentation is applicable)
5. Were the amendments employed documented in the laboratory records
at the time the modifications were made?
Yes
No
N/A
-------
UlS QA Project Plan
Revision No. 2
Date: 09/28/84
Page C-10 of C-13
6. If there are deviations from GLP procedures:
Attach a written record of the deviations from GLP.
Were the deviations approved by the PI, QAU, and the sponsor?
Yes No N/A
B. Laboratory Performance
1. Are there laboratory SOPs for handling the test system?
Yes No
2. Have laboratory SOPs been approved by the resonsible
investigator, the QAU, and management?
Yes No N/A
3. Are these followed and documented?
Yes No N/A
4. Are specimens from the test system collected according to an
approved SOP?
Yes No
5. Describe briefly the steps taken to assure that specimen sampling
was unbiased, representative, and statistically valid.
6. Describe briefly the method of sampling, including frequency and
length of sampling period.
7. Are all specimens from the test system labelled?
Yes No
-------
UM Project Plan
Revision No. 2
Date: 09/28/84
Page C-ll of C-13
8. Is there documentation that traces the test system from its
arrival at the laboratory to the completion of the test?
Yes No
9. Are the sources of the test system documented?
Yes No
10. Are the dates of arrival documented?
Yes No
11. Was there a pre-test examination of the test system?
Yes No
12. How long was the acclimation period for the test system?
13. Are there approved SOPs for handling the test compounds?
Yes No
14. If carriers are required during the handling of the test
compound:
Are carrier/test compound mixtures handled according to
approved SOPs?
Yes No N/A
Are all compounds and carriers labelled?
Yes No N/A
15. Are concentrations verified by chemical analysis?
Yes No N/A
Attach a written documentation of the verification.
-------
OTS ^ ;VOjCCu H iun
Revision No. 2
Date: 09/28/84
Page C-12 of C-13
16. Is analytical chemistry performed according to approved
laboratory SOPs?
Yes No N/A
17. Is analytical equipment regularly inspected and calibrated
according to approved laboratory SOPs?
Yes No N/A
Attach documentation of periodic inspection and maintenance
activities.
18. Are the chemical characteristics of the test compound described
in the laboratory records?
Yes No N/A
19. Is the stability of the test compound monitored throughout the
study?
Yes No N/A
20. If a carrier is used, is the stability of the mixture verified?
Yes No N/A
21. Identify (by a check mark) which, if any, of the following
internal/external checks of QC are performed to assure accuracy
of the measurement system:
a. Replicate
b. Spiked samples
c. Split samples
d. Blanks
Reagent
Method
e. Internal standards
f. QC samples
g. Surrogate samples
h. Calibration standards
i. Control charts
j. Reagent checks
k. Reference materials
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page C-13 of C-13
1. Reference devices
m. Cooperative analyses
n. Side-by-side analyses
o. Others (describe)
22. Are the results of these QC checks of peformance clearly
documented and traceable in the permanent laboratory records?
Yes
No
N/A
23. If the results of the QC checks were not recorded immediately in
the permanent laboratory records, are the data available for
inspection?
Yes
No
N/A
24. When the laboratory is presented with a QC reference specimen for
analysis, supplied by the auditor, describe the results of the
analysis in a quantitative fashion.
25. Are the results of the QC reference analysis within tolerance
limits associated with the analytical equipment, based on the
laboratory SOPs and the manufacturer's operations manuals?
Yes
No
N/A
26. In a second or third analysis of the auditor-supplied specimen,
are the results repeatable, within the tolerance limits as
described in laboratory SOPs and manufacturer's operations
manuals?
Yes
No
N/A
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page D-l of D-15
APPENDIX D
LABORATORY INSPECTIONS
Laboratory inspections, or audits, are designed to insure that the
standards set forth in Good Laboratory Practices (GLP) regulations are being
met by testing laboratories. Whereas aspects of a laboratory study are the
concerns of systems and performance audits, a laboratory inspection focuses on
facilities, equipment use and maintenance, personnel safety, etc.
A set of forms for conducting a laboratory inspection is attached.
These .forms are meant to serve as an SOP for performing laboratory inspections
and are based on nonclinical laboratory inspection forms used by the Food and
Drug Administration. Their completion should provide a comprehensive, objec-
tive evaluation of the testing laboratory, while minimizing the time and
effort required for the inspection.
Inspection Preparation
In preparing for a laboratory inspection, the inspector(s) should at
a minimum:
1. Become thoroughly familiar with GLP regulations
promulgated by EPA (FR 48: 53922-53944).
2. Become familiar with the contents and use of the
attached forms.
3. Access and review results of any earlier inspections
or audits concerning the laboratory.
Information obtained prior to the inspection should be confirmed during the
on-site laboratory inspection.
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page D-2 of D-15
Inspection Report
In addition to the completed attached forms, the final inspection
report should contain a summary of the findings of the inspection. The
summary may identify information that was not available during the inspection
and any detected irregularities that are not necessarily within the scope of
the laboratory inspection.
Debriefing
Immediately following the inspection, the inspector should debrief
the laboratory on the overall inspection results. It is recommended that
those aspects of the laboratory operations that are commendable, as well as
deficiencies uncovered during the inspection, be pointed out. If the
inspector identifies irregularities that are beyond the scope of the parti-
cular inspection being performed, the laboratory should be informed of these
irregularities and that the appropriate authority will be notified.
The forms resulting from the inspection should be reviewed by the
laboratory, and the laboratory contact should be requested to sign and date
the form under a statement that they have found the report to be true and
accurate. The inspector should also sign the report.
Training
The following are recommended components of an inspector training
program:
1. A short training course addressing at a minimum:
a. Inspection preparation and scheduling
b. Safety during the inspection
c. Handling proprietary/confidential information
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page D-3 of 0-15
d. Laboratory access procedures
e. General inspection procedures
f. Legalities involved in conducting a laboratory
inspection
g. How the inspector may be perceived by the
laboratory
h. Use and content of inspection forms
i. Discretionary responsibilities of the inspector
Participation in a minimum of two inspections accom-
panied by experienced inspectors is recommended. It
is also recommend ed that personnel trained for con-
ducting laboratory inspections have at a minimum a
Bachelor's Degree in a scientific area, and 2 years of
experience in general laboratory procedures.
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page D-4 of D-15
Laboratory Inspection Forms
Section I. Basic Information
When possible, the following information should be obtained in
advance of the audit being schedule.
A. Inspector Information:
1. Name(s)/Affiliation:
2. Date of Inspection: , 19_
B. Testing Lab Information:
1. Laboratory Name:
2. Laboratory Address:
3. Laboratory Phone No.: ( ) -_
4. Laboratory Contact: Dr./Mr./Ms,
5. Type of Laboratory:
Sponsor Information:
1. Sponsor Name:
2. Sponsor Address:
3. Sponsor Contact:
4. Sponsor Phone: ( ) -_
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page D-5 of D-15
Section II. Testing Facilities
1.
2.
The testing facility in general is of suitable size,
adequate construction, and properly located to perform
TSCA studies. Defined and, if necessary, separate
areas are provided.
Adequate space is provided for administration, super-
vision, and direction of the testing facility as well
as satisfactory facilities for toilets, lockers,
showers, with hot and cold water, and air driers or
single use towels plus all necessary accoutrements in
accordance with regulations set forth by the OSHA in
29 CFR.
Yes
No
N/A
Narrative
Section III. Personnel
1.
2.
3.
4.
Obtain organizational chart and list of personnel.
Employees practice good sanitation and health habits.
Employees follow standard operating procedures for
health and safety and have adequate laboratory
clothing appropriate for their duties to and to
prevent microbiological, radiological, or chemical
contamination of test substances and test materials.
All employees are instructed to report to supervisory
personnel any and all health or medical conditions
that may be considered to adversely effect the study.
Yes
No
N/A
Narrative
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page D-6 of 0-15
Section IV. Quality Assurance Unit
1.
2.
3.
4.
5.
6.
7.
8.
9.
There is a quality assurance unit (QAU).
A master schedule sheet of all laboratory studies is
maintained by the QAU.
Copies of all protocols and standard operating proced-
ures are maintained by the QAU.
Critical reviews of final reports are made to assure
accuracy of description with respect to methods; and
Standard operating procedures; and,
Observations; and,
Raw data; and,
Results (assuring that all adverse findings are
indeed included in the final report).
Procedures are written that describe the responsibili-
ties of the QAU and the records it maintains.
Yes
No
N/A
Narrative
V. Equipment
1. Equipment or
is available
appropriate design and adequate capacity
to obtain values reported.
Yes
No
N/A
Narrative
-------
Revision No. 2
Date: 09/28/84
Page D-7 of D-15
2.
3.
4.
5.
6.
7.
Location of equipment permits easy operation, clean-
ing, and maintenance; and,
Is cleaned, inspected, and maintained regularly.
There are written standard operating procedures which
describe in detail the methods, materials, and sched-
ules to be used in the routine inspection, cleaning,
maintenance, testing, and calibration of equipment;
and,
The specific remedial actions to be taken in the
event of failure or malfunction or equipment; and,
Designates the individual responsible for each of
the operations.
Copies of the standard operating procedures are
available to laboratory personnel.
Yes
No
N/A
Narrative
VI. Testing Facility Operation
1.
2.
Separate laboratory space is provided for the
performance of routine procedures or categories
of procedures; and,
Separate laboratory space is provided for the per-
formance of specialized activities auch as aseptic
surgery, intensive care, necropsy, radiography.
Yes
No
N/A
Narrative
-------
0- -
Revision No. 2
Date: 09./28/84
Page D-8 of 0-15
3.
4.
5.
6.
7.
8.
9.
10.
11.
Spaces for cleaning, sterilizing, and maintaining
equipment and supplies used during the course of
the studies are separate from the areas housing
test systems.
Studies involving radioactive or other biohazardous
materials are carried out in special facilities or
areas which provide protection to personnel, test
systems, and the external environment against contami-
nation or unnecessary radiation exposure, or infection.
Persons possessing and using radioactive materials are
licensed in accordance with the Nuclear Regulatory
Commission regulations or meet the requirements of an
agreement state.
Special procedures are employed for the handling of
other biohazardous materials.
Written standard operating procedures (which at least
meet GLP requirements) are maintained detailing the
methods to be used in performing laboratory studies.
Standard operating procedures are established for
animal room preparation; and
Animal Care, and
Test and control substances; receipt, identifica-
tion, strength, quality, purity, stability, storage,
handling, mixing, sampling, and administration, and
Test system observations, and
Yes
No
N/A
Narrative
-------
OTS QA Project '... .
Revision No. 2
Date: 09/28/84
Page D-9 of D-15
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
Laboratory tests, and
Handling of animals found moribund or dead during
study; and,
Necropsy of animals or post-mortem examination of
animals; and,
Collection and identification of specimens; and,
Histopathology; and,
Data handling, storage, and retrieval; and,
Preparation and validation of final study reports.
A historical file of standard operating procedures
annotating effective dates and dates of revisions is
maintained.
The relevant standard operating procedures are
available at all times in the immediate bench area of
personnel performing the procedures.
All reagents and solutions in the laboratory area are
labeled adequately.
Yes
No
N/A
Narrative
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page D-10 of D-15
VII. Animal Care
1.
2.
3.
4.
5.
6.
7.
Feed and water used for animals are analyzed period-
ically for the presence of known interfering contam-
inants.
A program for adequate veterinary care and humane
treatment has been established and is supervised by a
doctor of veterinary medicine (indicate name of DVM)
for studies involving cats, dogs, guinea pigs, ham-
sters, rabbits, or nonhuman primates; and,
For studies involving other animals by either a
doctor of veterinary medicine or by other qualified
persons (indicate name and qualifications).
Animals either known to be, or suspected of being dis-
eased, or carriers of a disease, are isolated in an
area contiguous with or near the animal housing area.
Animals are free of any naturally occurring diseases
or conditions that might interfere with the purpose or
conduct of the study.
The diagnosis, authorization for, and description of
the treatment (including dates of treatment of animals
involved) of test systems is adequately documented.
Methods for the unique and permanent identification of
all animals when needed have been developed and
applied to preclude mixup of animals, and
Yes
No
N/A
Narrative
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page D-ll of D-15
8.
9.
10.
11.
12.
Routine or specialized housing of animals of
different species, or of the same species used for
different studies is adequate to preclude inter-
species transmission of infection, mixup, or other
events that may affect the outcome of a study or
studies.
The proper placement of animals which are transferred
from one cage to another in the same location is
checked by the transferrer and verified by a respons-
ible person, appropriately documented, and a record
of the procedure maintained.
Animal waste and refuse is collected, stored, and dis-
posed of in a safe and sanitary manner so as to
preclude vermin infestation, odors, and disease
hazards.
Animal cages, racks, and accessory equipment are
cleaned and sanitized at appropriate intervals.
Storage areas for feed, bedding, supplies, clean
cages, and equipment are separate from areas housing
the test systems as well as the quarantine and isola-
tion area, and these materials are protected against
spoilage, infestation or contamination.
Yes
/
i
No
N/A
Narrative
VIII. Test and Control Substances
1.
Each container for a test and control substance is
appropriately labeled and adequately stored to main-
tain the identify, strength, and purity of said
substance.
Yes
No
N/A
Narrative
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page D-12 of 0-15
2.
3.
4.
5.
6.
7.
An appropriately identified reserve sample selected at
random from each batch of test and control substance
used in a study of more than 4 weeks duration, is
taken, stored in an identical immediate container
under appropriate storage conditions, and analyzed at
the time the batch is depleted, at the termination of
the study, or at the expriation date (whichever occurs
first) to assure that the identify, quality, strength,
purity, and stability conform to established
specifications.
If test or control substances are mixed with a carrier
prior to administration each batch of such mixture is
tested periodically for the adequacy of the mix to
assure uniformity and to determine the concentration
of the substance in the mixture. Describe procedures
used.
Enough samples of each batch of the mixture are
returned to the sponsor for such analysis if the study
is a blind study.
Each batch of the test and control substance-carrier
mix is tested for stability for at least the length of
time between mixing and use and to establish storage
conditions and an expiration date.
For each batch of the test and control substance,
tests are performed to determine the release from the
carrier mix and the results documented.
For each batch of test and control substance mixed
with a carrier an appropriately identified reserve
sample of each batch of the substance-mixture is taken
and retained for the required length of time.
Yes
No
N/A
Narrative
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page D-13 of D-15
8. All handing, storage, and disposal of known or
suspected chemical carcinogens used as a test
substance in a study are treated in accordance
with the safety principles set forth in the
"National Cancer Institute Safety Standards for
Research Involving Chemical Carcinogens",
Pub. No. (NIH) 75-900.
Yes
No
N/A
Narrative
IX. Storage and Retrieval of Records and Data
1.
2.
3.
4.
5.
The testing facility maintains and retains all raw
data, documentation, and other information, proto-
cols, specimens, and final reports generated during
and as the result of a laboratory study and they
are retained in an archive of adequate space and
design and are indexed to facilitate their orderly
and expedient storage and retrieval.
The archive provides the proper conditions to
minimize deterioration of all stored material
for as long as they are required to be retained.
The archive contains specific reference to other
locations in which documents and specimens may
be stored.
Documents and specimens required to be maintained
in the archive and not physically present there
have appropriate and specific reference to their
location filed in the archive.
An individual responsible for the archive is
identified.
Yes
No
N/A
Narrative
-------
OTS QA Project Plan
Revision No. 2
Date: 09/28/84
Page 0-14 of D-15
6.
7.
8.
9.
Only authorized personnel enter the archive and
whenever a custodian of the archive is not present,
the suitable repositories for the document and
specimens are locked.
Whenever the origina-1 material is transferred to the
sponsor's archive at the sponsor's request at the
completion of the study, duplicates of all material
required to be in the archive are retained there,
when the nature of the material permits.
All material required to be retainedin the archive
is available for inspection to authorized employees
of the Environmental Protection Agency or its
designee.
If the archive has been contracted out to a commer-
cial archive not belonging to the research facility
or sponsor, then the name and address of the commer-
cial archive has been provided to the sponsor in the
submission of the final study report.
Yes
-
No
N/A
Narrative
X. Retention of Records
1.
All protocols, raw data, specimens, final reports,
and other required documents pertinent to the conduct
of the study, including records and reports of main-
tenance, cleaning, calibration, and inspection of
equipment, are stored in an archive, and retained for
the specified time.
Yes
No
N/A
Narrative
-------
lS v ^'"^jticL Flan
ktv ision No. 2
Date: 09/28/84
Page D-15 of D-15
2.
3.
Curriculum vltae and job descriptions of all person-
nel engaged in conducting the study are retained for
the specified period of time, either in the facility
employment records, or the archive, and are available
for inspection.
The master schedule sheet, records of inspection
or evaluation, and status reports of the quality
assurance unit are retained for a specified period
of time.
Yes
No
N/A
Narrative
-------
PACKAGE CONTENTS
1. Slides and Stat. Reference
2. Part III of the OTS Quality Assurance Program Plan
3. Criteria for DQO Requirement
4. Review of DQO Information
5. QC Laboratory Audit of PCB's
°Control Chart
6. Principle of Environmental Analysis
7. Quality Assurance Program Plan for OTS
8. A Guide to Preparing Quality Assurance
Project Plains
-------
> Y ft-
HISTORY OF EPA OA PROGRAM
1979 Administrator Costle Emphasizes OA and Establishes a
Central Agency QA Unit and OA Policy
- OA unit is OAMS/OMSQA/ORD
Designation of Quality Assurance Officers (QAOs)
QA Addresses Generation of Environmental Measurements
1981 Administrator Gorsuch Supports the Agency OA Program
1983 OAMS Unsuccessfully Proposes Major Changes in OA Policy
Through EPA Order 5360.1
- OAMS/ORD Is Responsible for Adequacy of Agency Data
(Rejected by Program Offices)
- OAMS Leadership Is Replaced
1984 New QAMS Leadership Revises Order 5360.1 Which Al Aim
Signs. Implementation includes:
- . Development of DOO/DOI Guidance
Development of a Compehensive Audit System
- Development of OA Procedures for EPA Standard Chemical
Analysis Methodology
Revision of Agency Semi-Annual QA Meetings
Outline ORD OA Requirements
-------
£ (P/-1
GOAL OF EPA QA PROGRAM
The primary goal of the EPA QA program is to ensure that all
environmentally related measurements supported by the EPA
[generated by intramural and extramural EPA funds] produce data
of known quality.
o The quality of data is known when it is thoroughly
.documented, such documentation being verifiable and
defensible.
o It shall be the policy of all EPA organizational
units to ensure that data representing environmentally
related measurements are of known quality.
o Environmentally related measurements are defined as
any laboratory or field data gathering activity or
investigation involving the determination of chemical,
physical, or biological factors related to the
environment.
-------
OPTS QA ACTIVITIES AND SPECIAL RESPONSIBILITIES
(Based on the Implementation of
EPA Order 5360.1)'
o OPTS Has the Reponsibility to See That QA Is Implemented
- Al Aim Approved Recent QAMS Guidance in Order 5360.1
Designating Responsibility
- Each AA Has a QA Representative
+ Attends QAMS Administrative and Planning Sessions
+ Coordinates AA Response to QAMS (Based on QAO
Reviews)
o Data Quality Objectives and Data Quality Indicators (DQO/DQI)
- QAMS Provides DQO/DQI Guidance
- From EPA Monitoring Strategy QAMS Selects Major
Databases for Which DQO/DQI Will Be Prepared
- Program Offices Negotiate DQO/DQI Deliverables with QAMS
- OTS, OPP, CCS Prepare DQO/DQI '
- Following DQO/DQI Preparation, OTS, OPP, CCS, Provide
Regions with Instruction on Generating Data in
Accordance with DQO/DQI
-------
EXISTING PROGRAM OFFICE QA PROGRAMS
o Quality Assurance Officers (QAOs) Are Supposed to Report to
Office Directors
- Implement Agency QA Policy in Office
- Carry Out Audits
+ Systems Audits
+ Performance Audits
+ Data Audits
o QAO Signature of Approval Required by Regulation and Policy
- 40 CFR 30 (Grants and Cooperative Agreements)
- PIN (Contracts)
o QAO Undergoes Agency QA Audits of OTS QA Program
- QAMS Reviews OTS QA Activities
- QAMS Reviews OTS QA Documentation
- QAO Debriefs OTS/OD on Results of QAMS Audit
o QAO Represents Office Director in QAMS Planning and Programing
Activities
o OPTS QA Programs Well Regarded by QAMS
-------
OTS QUALITY ASSURANCE PROGRAM
o OTS QAO Reports to OD/OTS
o OTS QAO Is Chief of DDE
o OTS QAO Maintains Detailed Document Files of OTS QA Policy and
Activities
o OTS QAO Has Carried Out Systems, Performance, Follow-up, and
Data Audits on:
- PLM Analysis of Bulk Asbestos
- National Human Adipose Tissue Survey (NHATS)
- Analysis by Commercial PCB Disposal Demonstrations
- Analytical Method for Incidentally Generated PCBs
o OTS QAO Coordinates with ORD Preparation of Performance Audit
Specimens for NHATS and the Human Monitoring Initiative
o OTS QAO Maintains Quality Asssurance Files Containing
- Coorespondence
- Activities
- Quality Assurance Project Plans
- OTS QA Policy Documents
o OTS QAO Wrote QA Sections for Regulations
- Closed and Controlled PCB Rule
- Uncontrolled PCB Rule
o OTS QAO Was One of the First to Have a Agency-Approved QA
Program Plan
o OTS QAO IS Very Active in Agency QA Functions, Currently Leads
DQO/DQI V7orkgroup
-------
QTS QUALITY ASSURANCE PROGRAM
0 OTS SUBMITS A PROGRAM PLAN TO QAMS FOR APPROVAL
0 STATUS;
0 OTS QUALITY ASSURANCE PROGRAM PLAN PART I - APPROVED
0 PART 11 -- NOT REQUIRED
0 PART III -- ANNUAL REPORT
-- WORKPLAN
0 OTS HAS BEEN USED AS MODEL FOR THE AGENCY
° ACTIVE PARTICIPATION IN THE AGENCY QA PROGRAM
-------
WHO IS RESPONSIBLE FOR QUALITY ASSURANCE ?
You
Your Supervisor
All Decision Makers
-------
WHEN DO I USE QA AND HOW MUCH DOES IT COST ?
QA Is an EPA Requirement for Environmentally Related Measurements
(1) Pollutant Concentrations from Sources and in the
Ambient Environment, Pollutant Transport and Fate
(2) Response to Organisms to Pollutants
(3) The Effects of Pollutants onHuman Health and on the
Environment
(4) Risk/Benefit Analysis
(5) Environmental or Economic Impact
(6) The Environmental Impact of Cultural or Natural
Processes
(7) Pollutant Levels, Exposure Levels, ETC. Used in
Modelling
Everyone Uses QA to Some Degree, Sometimes Not By Name
The amount of QA employed must be sufficient to give
you confidence that your results are of acceptable quality
to meet your objectives within your resources.
From Your Total Resources, How Much Are You Willing to Pay Reduce
Your Risk of Your Supervisor's Making a Wrong Decision Based
on Your Information?
Too Much QA Stifles or Impedes Information Generation.
Too Little QA Increases the Probability of Making
a Wrong Decision from Information Generation
-------
RESOURCES FOR THE QA RULE OF THUMB
The "rule of thumb" is that ten to fifteen percent of a
project°s resources should be devoted to QA/QC. For data
where there is a large variability in the population or
parameter of interest (or a suspected large variability),
the "rule of thumb" may be too conservative. For very
routine or well characterized populations of interest, the
"rule of thumb" may be too liberal.
-------
GOALS OF THIS WORKSHOP
0 To SHOW THAT QUALITY ASSURANCE CAN MAKE YOUR LIFE EASIER
WHAT IS QUALITY ASSSURANCE/QUAI ITY COMTROI ?
QA -r ACCORDING TO WEBSTER is "A SYSTEM FOR VERIFYING AND
MAINTAINING A DESIRED LEVEL OF QUALITY IN A PRODUCT OR
PROCESS."
° QC -- IS THE SPECIFIC STEP-BY-STEP DESCRIPTION OF HOW A
QUALITY ASSURANCE PROGRAM MAY BE IMPLEMENTED TO ASSURE
DATA OF KNOWN QUALITY.
-------
OTS QUALITY ASSURANCE OFFICER
° QAO REPORTS DIRECTLY TO THE OFFICE DIRECTOR OF OTS
° QAO IS RESPONSIBLE FOR AND WILL OVERSEE ALL ASPECTS OF
QUALITY ASSURANCE ACTIVITIES WITHIN OTS
0 QAO HAS A STAFF RESPONSIBLE FOR COORDINATING THE DAY-
TO-DAY ACTIVITIES OF THE QA PROGRAM
0 STAFF;
0 EILEEN REILLY-WIEDOW -- MANAGER OF OTS QA PROGRAM
0 JOAN BLAKE -- BIOLOGIST, PCB AUDITS
0 C.J. NELSON -- STATISTICIAN
0 OTS QA MANAGER is RESPONSIBLE FOR:
-- IMPLEMENTING OTS QA PROGRAM
-- PROVIDING TECHNICAL ASSISTANCE/GUIDANCE
-- REVIEWING QA PROJECT PLANS AND QA-RELATED SECTIONS
OF PROCUREMENT PACKAGES
0 QAO AND MANAGER WILL BRIEF OD ON QA ISSUES
-------
TASK MANAGER RESPONSIBILITIES
RESPONSIBLE FOR DATA QUALITY OBJECTIVES
RESPONSIBLE FOR QUALITY ASSURANCE ON THEIR TASK
RESPONSIBLE FOR GIVING CONTRACTORS DIRECTION AND/OR
TECHNICAL GUIDANCE FOR:
° FULFILLING OBJECTIVES OF THE STUDY
° STATISTICAL DESIGN
° ANALYTICAL DESIGN
0 QUALITY ASSURANCE PROJECT PLANS (QAPJP)
° OVERSITE OF DAILY QUALITY ASSURANCE/QUALITY CONTROL ACTIVITIES'
° DOCUMENTATION OF QA/QC ACTIVITIES
-------
DATA QUALITY OBJECTIVES
QAMS
0 DQOS ARE STATEMENTS OF THE QUALITY OF DATA A DECISION
MAKER NEEDS SO THAT HIS/HER DECISION, THAT RELIES ON
THIS DATA IS DEFENSIBLE-
OR
° GOOD PLANNNG IS AN ESSENTIAL PRINCIPLE OF ENVIRONMENTAL
ANALYSIS- THE OBJECTIVE IS TO DEFINE THE PROBLEM AND
ANALYTICAL PROGRAM WELL ENOUGH SO THAT THE INTENDED
RESULTS CAN BE ACHIEVED EFFICIENTLY AND RELIABLY.
OR
0 WHAT DO 1 HOPE 1 CAN SAY WITH THIS STUDY?
-------
QAMS THREE STAGE DQO PROCESS
STAGE 1; DEFINE THE DECISION
0 RESPONSIBILITY OF THE DECISION-MAKER
STAGE 2: CLARIFY THE INFORMATION NEEDED FOR THE DECISION
0 RESPONSIBILITY OF THE SENIOR PROGRAM STAFF WITH
INPUT FROM TECHNICAL STAFF
STAGE 5: DESIGN THE DATA COLLECTION PROGRAM
0 RESPONSIBILITY OF TECHNICAL STAFF (TASK MANAGER
AND CONTRACTORJ
-------
DATA QUALITY OBJECTIVES (DQOs)
HOW DO I START?
-- DEFINE THE STUDY AS YOU SEE IT
r
-- BRIEFLY OUTLINE HOW YOU WOULD DO THE STUDY. WHAT DO
YOU THINK THE STUDY IS TRYING TO ACHIEVE?
-- PASS THIS BACK TO THE "DECISION-MAKER"
-- PREPARE DATA QUALITY OBJECTIVES
-- USE STEPS FOR DESIGNING AN EXPERIMENT AS A GUIDE
H
C
-------
DECISION
STUDY
PASSED TO TECHNICAL STAFF
PREPARE DQOsr
.TASK MANAGERS
CONTRACTORS
OBTAIN AGREEMENT ON DQOs
V
DESIGN STUDY (STATISTICAL AND ANALYTICAL)
!!! ALL AGREE !!!
TASK MANAGER
V
PREPARE QAPJP<
\
.STAT,
-CHEM,
QAPJP APPROVED BY QAO AND QA MANAGER
INITATE STUDY
OVERSEE STUDY
I
FINAL REPORT/RESULTS
-------
KEY TO GOOD QA/QC
0 DOCUMENTATION
TOOLS FOR REVIEWING QA:
LOGIC
RESPONSIBILITY
FOLLOW THRU
CLEAR, CONCISE STATISTICAL DOCUMENTATION
CLEAR, CONCISE ANALYTICAL DOCUMENTATION
/
CLEAR, CONCISE DATA TRANSFER/VALIDATION DOCUMENTATION
THE USE OF:
0 CONTROL CHARTS
° ACCEPTABLE CRITERIA
0 CORRECTIVE ACTIONS
0 FOLLOW-UP
0 THE ABOVE SHOULD BE USED FOR THE REVIEW OF THE OVERALL
STUDY
-------
STEPS FOR DESIGNING AN EXPERIMENT:
1. STATEMENT OF THE PROBLEM.
A. IDENTIFY THE NEW AND IMPORTANT PROBLEM AREA
B. OUTLINE THE SPECIFIC PROBLEM WITHIN CURRENT LIMITATIONS.
c. DEFINE THE SCOPE OF THE TEST PROGRAM
D- DETERMINE RELATIONSHIP OF THE PARTICULAR PROBLEM TO THE
WHOLE RESEARCH/DEVELOPMENT/MONITORING PROGRAM
2- FORMULATION OF HYPOTHESES-
3« DEVISING OF EXPERIMENTAL TECHNIQUE AND DESIGN.
4- EXAMINATION OF POSSIBLE OUTCOMES AND REFERENCE BACK TO THE
REASONS FOR THE INQUIRY TO BE SURE THE EXPERIMENT PROVIDES
THE REQUIRED INFORMATION TO AN ADEQUATE EXTENT.
5. CONSIDERATION OF THE POSSIBLE RESULTS FROM THE POINT OF
VIEW OF THE STATISTICAL PROCEDURES WHICH WILL BE APPLIED TO
THEM, TO ENSURE THAT THE CONDITIONS NECESSARY FOR THESE
PROCEDURES TO BE VALID ARE SATISFIED.
6. PERFORMANCE OF EXPERIMENT.
7- APPLICATION OF STATISTICAL TECHNIQUES TO THE EXPERIMENTAL
RESULTS
8. DRAWING CONCULSIONS WITH MEASURES OF THE RELIABILITY OF
ESTIMATES OF ANY QUANTITIES THAT ARE EVALUATED, CAREFUL
CONSIDERATION BEING GIVEN TO THE VALIDITY OF THE CONCLUSIONS
9. EVALUATION OF THE WHOLE INVESTIGATION, PARTICULARLY WITH
OTHER INVESTIGATIONS ON THE SAME OR SIMILAR PROBLEMS-
-------
Check List for Planning Test Programs
FT) A. Obtain'a clear statement of the problem
?. Identify the now and important problem area
2. Outline the specific problem within current limitations
3. Define exact-scope of the test program
4. Determine relationship of the particular problem to th
search or development program
B. Collect available background information
1. Investigate all available sources of information
2. Tabulate data pertinent to planning .new program
-------
Check List for Planning Tes.t Program (jpontl
C. Design the tc^t prop rum
1. Hold a conference of all parties concerned
^ a. State the propositions to be proved .
b. Agree on magnitude of differences considered worthwhile
. Outline the' possible alternative outcomes
d. Choose the factor? to be studied *
e. Determine the practical range of these factors and the specific
levels at which tests will be made
f. Choose the end measurements which arc to be made «
g. Consider the effect of sampling variability and of precision of test/
methods
h. Consider possible inter-relationships (or "interactions") of the '
factors
i. Determine limitations of time, cost, materials, manpower, instru-
mentation and other facilities and. of extraneous conditions, such----'
as weather .
}. Consider human relation angles of the program
2. Design the program in preliminary form
a. "Prepare a systematic and inclusive schedule *
b. Provide for step-wise performance or adaptation of schedule if
necessary
c. Eliminate effect of variables not under study by controlling,
balancing, or randomizing them
d. Minimize the number of experimental runs
e. Choose the method of statistical analysis
f. Arrange for orderly accumulation of data
3. Review the design with all concerned
a. Adjust the program in line with comments
b. Spell out the steps to be followed in unmistakal 3 terms
-------
Check List For PLannipg. .Teat
(.contl
Plan and carry out the experimental work
1. Develop methods, materials, and equipment '
2. Apply the methods or techniques
3. Attend to and check details; modify methods if necessary
4. Record any modifications of program design .
5. Take precautions in collection of data .
6. Record progress of the program
.,
-------
Check. Lis.t £or
TteSfc E^ggpTfTO facrttl
7 £. Analyse the data
1. Reduce recorded data, if necessary, to numerical form
2. Apply proper mathematical statistical techniques
/ F. Interpret the results
1. Consider all the observed data
2. Confine conclusions to strict deductions from the evidence at hand
3. Test questions suggested by the data by independent experiments
4. Arrive at conclusions as to the technical meaning of results as well
as their statistical significance
5. Point out implications of the findings for application and for further
work
6. Account for any limitations imposed by the methods used
7. State results in terms of verifiable probabilities
* G. Prepare the report j
* f
1. Describe \vork clearly giving background, pertinence of the problems
and meaning of results
2. Use tabular and graphic methods of presenting data in good form for
future use
3. Supply sufficient information to permit reader to verify results and
draw his own conclusions
4. Limit conclusions to objective summary of evidence so that the work
recommends itself for prompt consideration and decisive action-*
" »V-'^-»*
I
*.***
-------
TASK MANAGER Qu*J&4
CONTRACTOR
CHEMISTRY.
STATISTICS
OTS QA MANAGER (2 COPIES)
(BRIEF NOTE. ON PROJECT)-
\k
REVIEW
APPROVE,
V
OTS QAO SIGNS
DISAPPROVE
COMMENTS
TASK MANAGER RESPONDS
PRIOR TO PROJECT START
-------
AUDITS
TYPES
0 PERFORMANCE
0 SYSTEMS AUDITS
BY WHOM
0 TASK MANAGER
0 QA MANAGER
AUDIT REPORT REQUIRED
0 AVAILABLE TO QAMS
-------
FIELD VISITS
0 $$ ALLOWING ~- DO IT
0 INVALUABLE EXPERIENCE
FINAL REPORTS
0 QA INCLUDED AS PART OF THE REPORT
° GIVE EXPLANATIONS FOR UNUSUAL RESULTS
FUTURE
0 QA TRACKING SYSTEM (JUNE/JULY)
0 TASK MANAGER AUDITS
-------
SECOND EDITION
STATISTICS IN RESEARCH
BASIC CONCEPTS AND TECHNIQUES FOR RESEARCH WORKERS
BERNARD OSTLE
PROFESSOR OF ENGINEERING. ARIZONA STATE UNIVERSITY
THE IOWA STATE UNIVERSITY PRESS
_xvm«3, IOWA, U.S.A.
-------
164 CHAPTER 10, DESIGN OF EXPERIMENTAL INVESTIGATIONS
only those contrasts which are meaningful to the researcher should be
analyzed.
10.16 STEPS IN DESIGNING AN EXPERIMENT
Each statistician has his own list of steps M'hich he follows when
designing an experiment. However, a comparison of various lists
reveals that they all cover essentially the same points.
According to Kempthorne (28), a statistically designed experi-
ment consists of the following steps:
(1) Statement of the problem.
(2) Formulation of hypotheses.
(3) Devising of experimental technique and design.
(4) Examination of possible outcomes and reference back to the reasons
for the inquiry to be sure the experiment provides the required in-
formation to an adequate extent.
(5) Consideration of the possible results from the point of view of the
statistical procedures xvliirh will be applied to them, to ensure that
the conditions necessary for these procedures to be valid are satis-
fied.
(6) Performance of experiment.
(7) Application of statistical techniques to the experimental results.
(S) Drawing conclusions with measures of the reliability of estimates of
any quantities that are evaluated, careful consideration being
given to the validity of the conclusions for the population of objects
or events to which they are to apply.
(9) Evaluation of the whole investigation, particularly with other in-
vestigations on the same or similar problems.4
In a later section, these steps will be illustrated through the considera-
tion of some design problems.
Since the designing of an experiment or the planning of a test pro-
gram is such an important part of any investigation, the statistician
must make every effort to obtain all the relevant information. This
will usually require one or more conferences with the researcher, and
the asking of many questions. It has been my experience that the
amount of time consumed in this phase can be materially reduced if, at
the preliminary meeting between the researcher (e.g., a development
engineer) and the statistician, time is taken to explore the relationship
between research and/or development experimentation and. the sta-
tistical design of experiments. (NOTE: Frequentljr, there is a formid-
able communications barrier which must be overcome.) One of the
best ways to convince the researcher of the need for the multitude of
questions posed by the statistician is to give him (in the first meeting)
a "check list" which specifies various stages in the planning of a test-
program. (An even more efficient arrangement if you are the statisti-
cian in an industrial organization is to distribute copies of such a list to
all persons who may at some time have need of your services.) One
4 O. Kempthorne, The Design and Analysis of Ejperiments, John \Vilcv and
Sons, Inc., New York, 1952, p. 10.
-------
10.16 STEPS IN DESIGNING AN EXPERIMENT 365
such list, prepared by Bicking (3), is reproduced below for your con-
sideration.
Check List for Planning Test Programs
(l) A. Obtain a clear si-nU-mcnt of l/ic problem
1. Identify the new and important problem area
2. Outline the specific problem within current limitations
3. Define exact-scope of the test program
4. Determine relationship of the particular problem to the whole re-
search or development program
B. Collect available background information
\. Investigate all available sources of information
2. Tabulate data pertinent to planning .new program
C. Dtxign the list program
}. Hold a conference of all parties concerned
9 a. State the propositions to be proved".
b. Agree on magnitude of differences considered worthwhile
jlr. Outline the possible alternative outcomes
d. Choose the factors to be studied
e. Determine the practical range of these factors and the specific
levels at which tests will be made
f. Choose the end measurements which arc to be made
g. Consider the effect of sampling variability and of precision of test >
methods
h. Consider possible inter-relationships (or "interactions") of the
factors
i. Determine limitations of time, cost, materials, manpower, instru-
mentation and other facilities and-of extraneous conditions, such
as weather
j. Consider human relation angles of the program
2. Design the program in preliminary form
a. "Prepare a systematic and inclusive schedule *
b. Provide for step-wise performance or adaptation of schedule if
necessary
c. Eliminate effect of variables riot under study by controlling,
balancing, or randomizing them
d. Minimize the number of experimental runs
e. Choose the method of statistical analysis
f. Arrange for orderly accumulation of data
3. Review the design with all concerned
a. Adjust the program in line with comments
b. Spell out the steps to be followed in unmistakable terms
( (j D. Plan and carry out the experimental work
I. Develop methods, materials, and equipment
2. Apply the methods or techniques
3. Attend to and check details; modify methods if necessary -
4.. Record any modifications of program design .
5. Take precautions in collection of data .
6. Record progress of the program
-------
266
CHAPTEI 10, DESIGN OF EXPERIMENTAL INVESTIGATIONS
7 E. Analyze the data
1. Reduce recorded data, if necessary, to numerical form
2. Apply proper mathematical statistical techniques
/ F. Interpret the results
1. Consider all the observed data
2. Confine conclusions to strict deductions from the evidence at hand
3. Test questions suggested by the data by independent experiments
4. Arrive at conclusions as to the technical meaning of results as well
as their statistical significance
5. Point out implications of the findings for application ami for further
work
6. Account for any limitations imposed by the methods used
7. State results in terms of verifiable probabilities
* G. Prepare the report
1. Describe work clearly giving background, pertinence of the problems
and meaning of results
2. Use tabular and graphic methods of presenting data in good form for
future use
3. Supply sufficient information to permit reader to verify results and
draw his own conclusions
4. Limit conclusions to objective summary of evidence so that the work
recommends itself for prompt consideration and decisive action.'
The reader should realize, of course, that the two lists (of steps in
designing experiments) presented in this section are only guides. Very
seldom will the various steps be tackled and settled in the particular
order given. The statistician does not operate in such a mechanical and
routine fashion. Questions will be asked and answers received which will
trigger new lines of thought, and thus the planning conference will find
itself jumping from one step to another in a seemingly haphazard man-
ner. Furthermore, it is not surprising to find, as the conference pro-
gresses and new information is brought forth, the same step being con-
sidered several times. Regardless of the repetition inherent in such a
procedure, it is a good procedure.
In summary, then, the designing of an experiment can be a time-
consuming and, occasionally, a painful process. Thus, the use of check
lists such as those presented earlier can be most helpful (as a supple-
ment to common sense) in making relatively certain that nothing has
been overlooked.
10.17 ILLUSTRATIONS OF THE STATISTICIAN'S AP-
PROACH TO DESIGN PROBLEMS
To illustrate the manner in which a statistician approaches a design
problem, a series of examples will be considered. The first of these will
demonstrate the application of Kempthorne's nine steps, while the
* Charles A. Sicking, "Some uses of statistics in the planning of experiment?,''
Industrial Quality Control, Vol. 10, No. 4, Jan., 1954, p. 23.
remainder will illu.<
through 10.1C.
Example 10.12
Suppose a ma«
random series of
random binary el
cent of the time ;
experiment be de-
ness of the maehi
The preceding
statement of the
for the probabilit
The devising of ai
simple. In this c;
times, say n, recoi
enough agreemen
we accept //; if t!
alternative hypotl
care of is the deti
that are required '
desired that the p:
should be no grea-
shown. Note caret
value of n would ;-.
ing a false hypot
example. Step 5 c
will be analyzed i
make certain that
statistically indepc
some. When discus
to, and all that.ren
that we should p
probability of prod
is not sufficient. AY
sions only hold for
randomly selected
Had other devices
our experiment she
tion from the alliei
The reader will j
tration to Example
is "dress up" the i
the design of an ex
Example 10.13
Consider the pro
effects of eight trea
life of a particular
homogeneous batte
mucli information,
assign the batterie
-------
QA LAB AUDIT PROCEDURES FOR A PCB DISPOSAL DEMONSTRATION
--Engineers in CRB screen applications for PCB disposal permits
to weed out those processes that are illegal or that simply won't work
--A QAPP must be submitted as part of the permit application for
any lab associated with a disposal process and the lab must undergo
a QA performance audit
--In most on-site labs, analyses of PCBs are carried out on portable
gas chromatographs equipped with electron capture detectors
\
--A demonstration for successful applicants is scheduled in which
the PCB disposal process must be repeated successfully three times to
prove the technology is sound and any associated lab must be operated
under the pressure of field conditions in the presence of the auditor
Laboratory analytical equipment is most commonly used:
to quantitate the concentration of initial feed material to
be destroyed during the process;
to declare the treated material and all wastes "clean" or to
contain less than 2 ppm PCBs; and
to monitor some processes during treatment
--Before the demonstration, company representatives are told what
to expect during the QA audit of the lab, and at the start of the
demonstration, they are given a letter outlining procedures in detail
--The lab operator is usually given four QA samples to analyze and
quantitate on-site during the demonstration in my presence to prevent
cheating
*
--If a problem is discovered during the analysis of my QA samples
or of process samples, the analyst and I attempt to track down and
solve the problem during the demonstration """""*
--Lab practices in general are also evaluated, and an audit form
is completed and discussed with company representatives at the end
of the demonstration
--Some common problems encountered for which labs are failed:
inept GC operators;
malfunctioning GC's;
bad lab practices such as poor housekeeping leading to
contamination problems
--The majority of the labs audited are well run but, if a lab
fails, it is CRB policy that the whole PCB disposal process must be
redemonstrated before a permit can be issued
--Although PCB demonstrations are costly, take place outdoors in
all kinds of weather, and average one week of 12 hour days, they are
important because the resulting permitted processes directly help to
protect the environment from PCBs as well as any other toxic wastes
that can be destroyed using the same technology
-------
!D
(Xi
Accuracy
Code:
111.0 .
107.0 .
103.0 .
99.0 .
95.0 .
g
H 91.0 .
D
0
jj 87.0 .
K
X 83.0 .
79.0 .
75.0 .
Ul
STORE! tt: Spike Concr0 . 050080 $
en
CO
o
o
n
CO
CO
_ _ _ UML=100.si
: 98.0 g
CO
... 9^. 0 ...^if 0 ^ " 9"! 0 ftUCs94.5 w
_ _ _ w-tf LHL=88.1 ?
01/24 01/27 01/28 02/06 02/07 02/12 03/11 03/12 03/13 03/18/86
o
L'J
------- |