RESEARCH TRIANGLE INSTITUTE
IERL-RTP DATA QUALITY MANUAL
Second Edition
by
N. Sexton
F. Smith
Contract No. 68-02-3146
Task No. 22
EPA Project Officer: W. B. Kuykendal
for
Process Measurements Branch
Industrial Environmental Research Laboratory
U.S. Environmental Protection Agency
Research Triangle Park, North Carolina 27709
October 1980
RESEARCH TRIANGLE PARK, NORTH CAROLINA 27709
-------
CONTENTS
Section Page Revision Date
1 Introduction 1 0
2 Purpose and Scope of Data
Quality Manual 1 0
2.1 Purpose 1 0
2.2 Scope 1 0
3 Quality Policies and Objectives 1 0
3.1 Quality Policies 1 0
3.1.1 Coverage of the Quality
Assurance Program 1 0
3.1.2 Levels of Quality
Application 3 0
3.2 Quality Objectives 4 0
4 Organization for Data Quality 1 0
4.1 Organizational Structure 1 0
4.2 Functional Responsibilities 1 0
4.2.1 Quality Assurance Manager ... 1 0
4.2.2 Quality Assurance Officer ... 1 0
4.2.3 Process Measurements
Branch 4 0
4.2.4 Project Officer ........ 5 0
4.2.5 IERL-RTP Contractors 5 0
4.2.6 Functional Relationships .... 5 0
5 Implementation Plan and Schedule 1 0
5.1 Implementation Schedule 1 0
5.2 Estimated Cost of Implementation ... 3 0
6 Elements of a Quality Control Program ... 1 0
6.1 General Statement 1 0
6.2 Facilities and Equipment 3 0
6.3 Configuration Control 3 0
6.4 Personnel Training 3 0
6.5 Documentation Control 4 0
6.6 Control Charts 5 0
6.7 In-Process Quality Control 5 0
6.8 Procurement and Inventory
Procedures 5 0
6.9 Preventive Maintenance 7 0
6.10 Reliability 8 0
6.11 Data Processing 8 0
6.12 Feedback and Corrective Action .... 11 0
6.13 Calibration 11 0
-------
CONTENTS (continued)
Section Page Revision Date
7 Guidelines for Quality Assurance
Programs 1 0
7.1 General Statement 1 0
7.2 The Request for ProposalQuality
Control Aspects 2 0
7.3 Evaluation of Quality Control in
the Proposal 2 0
7.4 Evaluation of the QA Project Plan ... 3 0
7.5 The Onsite Qualitative Systems
Audit 5 0
7.6 The Performance Audit 5 0
7.7 Data Quality Assessment 6 0
7.8 QA Evaluation of the Final Report ... 6 0
8 References 1 0
Appendixes
A. Traceability Protocol for Estab-
lishing True Concentrations of
Gases Used for Calibrations
and Audits of Air Pollution
Analyzers (Protocol No. 2) I 0
B. Standard QA Enclosures for RFP's ... 1 0
C. Qualitative Onsite Systems Audit
Checklist 1 0
D. Standard Techniques Used in
Quantitative Performance Audits .... 1 0
E. Definitions and Statistical
Techniques Useful in Quality
Assurance Programs 1 0
F. Some Standard Ambient Air and
Source Sampling Techniques 1 0
-------
FIGURES
Section Page Revision Date
1 Quality assurance review for
extramural projects 2 0
2 IERL-RTP organization chart 2 0
3 Data quality program organization 3 0
4 Flow chart of functional relationships ... 6 0
5 QAMS implementation requirements and
schedule 2 0
6 IERL-RTP QA Program implementation
schedule 4 0
7 Overview of a data quality program 2 0
8 Standard quality control chart 6 0
9 Change request form 12 0
10 Typical calibration curves 14 0
11 Format for QA project plans 4 0
-------
Section No. 1
Revision No. 0
Date: November 3, 1980
Page 1 of 1
SECTION 1
INTRODUCTION
The Industrial Environmental Research Laboratory (IERL-RTP) has long
recognized the importance of quality assurance as an integral part of its
research and measurement activities. Heretofore, quality assurance has been
practiced on a project-by-project basis, with the preparation and implementa-
tion of a QA plan being the responsibility of the EPA Project Officer and the
Contractor, with support, if desired, from the IERL-RTP QA Officer and the
QA Contractor.
The development and implementation of a formal IERL-RTP data quality
program was initiated in December 1974 by the preparation and distribution to
senior staff members of the "Planning Document for an IERL-RTP Quality Assur-
ance Program."1 This planning document grouped all IERL-RTP projects into
five categories. A sixth category, environmental assessments, was later
added. Further progress in developing and implementing an IERL-RTP data
quality program was realized through the preparation and onsite trial imple-
mentation of QA procedures to IERL-RTP projects.2 3 4
An IERL-RTP QA Officer has been designated and given the responsibility
of establishing and maintaining a Laboratory-wide QA program that encompasses
all new EPA or EPA-funded work involving environmentally related measurements.
Environmentally related measurement activities include all field and laboratory
investigations that generate data and all data processing activities. The
QA Officer is also responsible for coordination of IERL-RTP1s QA Program with
the EPA Quality Assurance Management Staff (QAMS) to ensure that the IERL-RTP
QA Program is consistent with the Agency-wide mandatory QA Program. This
Agency-wide program was formally established through a June 14, 1979, directive
issued by EPA Administrator Costle.
-------
Section No. 2
Revision No. 0
Date: November 3, 1980
Page 1 of 1
SECTION 2
PURPOSE AND SCOPE OF DATA QUALITY MANUAL
2.1 PURPOSE
The purpose of this manual is to provide guidance for the continued
implementation and maintenance of an integrated data quality program for the
Industrial Research Laboratory, Research Triangle Park (IERL-RTP), projects.
2.2 SCOPE
This manual describes the administrative systems pertaining to the mainte-
nance and improvement of an IERL-RTP data quality program. The administrative
systems include: quality policies that provide guidance for the implementation
of a data quality program and quality objectives to guide in the designing of
quality control and quality assurance plans (Section 3); organization, includ-
ing key quality personnel and groups (Section 4); and the current schedule for
the quality assurance program (Section 5).
Guidelines for developing project-specific quality control plans and
quality assurance plans are given in Sections 6 and 7, respectively.
-------
Section No. 3
Revision No. 0
Date: November 3, 1980
Page 1 of 4
SECTION 3
QUALITY POLICIES AND OBJECTIVES
This section contains the EPA and IERL-RTP policies to be followed in the
establishment and maintenance of a data quality program and the objectives to
be realized through a well-planned and conscientiously applied data quality
program. A time schedule and details for implementation of these policies are
given in Section 5. The organizational structure for establishing and main-
taining the data quality program is given in Section 4.
3.1 QUALITY POLICIES
3,1.1 Coverage of the Quality Assurance Program
The IERL-RTP QA Program should be comprehensive, encompassing both in-house
and contract experiments, tasks, and projects that either generate or use
experimental data. It should be integrated, in that all experiments, tasks,
and projects must have a QA Project Plan delineating the practices and proce-
dures to be implemented at each level (e.g., operator, bench chemist, project
leader) and each phase of the project. An IERL-RTP QA Program Plan will be
prepared annually and evaluated and approved by the Quality Assurance Manage-
ment Staff. All experiments, tasks, and projects within IERL-RTP or funded by
IERL-RTP will also have a QA Project (or task) Plan for monitoring the effec-
tiveness of the QC program. The IERL-RTP QA Officer and the Project Officer
will approve these QA Project Plans.
A specific format for Requests for Proposals (RFP's) is provided by
Laboratory Directive 79, "Quality Assurance Requirements for Contracts, Grants,
and Interagency Agreements," issued by the IERL-RTP Laboratory Director. Each
procurement plan, as per the directive, will be accompanied by the form "Qual-
ity Assurance Review for Extramural Projects" (Figure 1). Copies of the
appropriate forms, completed by the Project Officer, will be sent to, signed
by, and retained by the IERL-RTP QA Officer.
The standard enclosure packages referred to in Figure 1 apply to the
following project requirements:
-------
Section No. 3
Revision No. 0
Date: November 3, 1980
Page 2 of 4
I. GENERAL INFORMATION
1. Descriptive Title:
2. Sponsoring Program Office: _
3. Approximate Dollar Amount:
4. Duration:
II. DESCRIPTION OF WORK
This project requires the generation of environmental measurements
No
III. QUALITY ASSURANCE (projects requiring environmental measurements)
(Complete this section only if section II response was yes.) Y**
1. Submission of a written QA Program Plan (commitment of the offerer's
management to meet the QA requirements of the scope of work) is to
be included in the contract proposal. (Use RFP Enclosure 1)
2. Submission of a written QA Project Plan is to be included in the
contract proposal. (Use RFP Enclosure 21
3. An onsite evaluation of proposer's facilities will be made prior to
award to ensure that a QA system is operational and exhibits the
capability for successful completion of this project (Use RFP
Enclosure 3)
4. Audit samples or devices are available for the parameters relevant to
this project.
5. Performance on available audit samples shall be required as part
of the evaluative criteria (Use RFP Enclosure 4)
6. A written QA Project Plan is required as part of the contract
(Use RFP Enclosure S)
7. Participation in systems audits will be required as part of the contract
(Use RFP Enclosure 6)
8. Participation in performance audits will be required as part of the
contract (Use RFP Enclosure 71
9. Periodic QA Reports will be required as part of the contract
(Use RFP Enclosure 8)
IV. DETERMINATION (Projects requiring environmental measurements)
The following will be included in the evaluation of all contract proposals for this project:
1. Written QA Program Plan (required)
2. Written QA Project Plan
3. QA Performance Audit Results
4. QA System Audit Results
5. Other QA Aspects (specify)
6. Relative Weighting of QA to Total Evaluation ( ): 100
IER/QA Officer lERL/Project Officer.
Figure 1. Quality assurance review for extramural projects.
-------
Section No. 3
Revision No. 0
Date: November 3, 1980
Page 3 of 4
Preaward
Demonstrations
of Capability
Postaward
Assessments
of Capability
1. QA Program Plan
2. QA Project Plan
3. Systems Audit
4. Performance Audit
5. QA Project Plan
6. Systems Audit
7. Performance Audit
8. QA Reports
Use of these enclosures (shown in Appendix B) will facilitate the procure-
ment process by ensuring that all bidders start from a common baseline and
ultimately will result in a more uniform application of QA procedures to all
IERL-RTP-funded programs. These enclosures will be available to the Project
Officer through the Contracts Office for inclusion in RFP's.
As indicated in the Administrator's memo, QA considerations should consti-
tute 5 to 30 percent of the technical evaluative criteria fur proposals to
RFP's involving environmentally related measurements. The actual weightings
will be determined by the Project Officer as shown in Figure 1.
3.1.2 Levels of Quality Application
Quality practices and procedures can be implemented at two levels.
1. Quality control. The design and implementation of QC practices
and procedures required to ensure that data quality is sufficient
to meet project requirements are the responsibility of the
individual or organization conducting the project. For example,
on projects conducted under contract, the QA Project Plan will
be prepared by the contractor and reviewed and approved by the
EPA Project Officer with assistance from the QA Officer. A
guidelines document is being prepared by IERL-RTP to assist in
the preparation of QA Project Plans. In-house projects will
have QA Project Plans prepared by the responsible EPA staff
member, with assistance from the QA Officer. A guidelines
document is also being prepared for these in-house projects.
2. Quality assurance. Quality assurance procedures for independent-
ly monitoring and assessing the efficiency and adequacy of
individual QC programs will be established and administered by
the IERL-RTP QA Officer. The QA procedures may be applied
throughout the duration of the project. However, at any speci-
fic time during the project life, either at the request of the
subject Project Officer or if deemed necessary by the QA Officer,
the ongoing quality control program may be assessed using
accepted QA techniques.
-------
Section No. 3
Revision No. 0
Date: November 3, 1980
Page 4 of 4
3.2 QUALITY OBJECTIVES
The primary objective of the IERL-RTP data quality program is to ensure,
assess, and document that the quality (i.e., precision, accuracy, complete-
ness, and representativeness) of measurements made by and/or experimental data
used in IERL-RTP activities and publications is commensurate with the end use
of the data. The objective of a properly structured QA program is not to
generate the most accurate or most precise data but to produce the level of
accuracy necessary to meet measurement program objectives. Management, admin-
istrative, statistical, investigative, preventive, and corrective techniques
will be employed to maximize the end effectiveness of the data.
Specific data quality objectives are:
1. To establish acceptable limits (precision, accuracy, and com-
pleteness) on data quality as a function of project objectives,
available resources, and measurement method capabilities;
2. To establish recommended standard procedures and require their
use to ensure the comparability of like data between projects;
3. To establish guidelines for the selection and use of additional
measurement methods necessary to ensure the collection of data
of acceptable quality (i.e., of acceptable precision, accuracy,
and completeness) on a project-by-project basis;
4. To develop and implement QC programs on each specific IERL-RTP
project;
5. To develop and implement the QA procedures necessary to independ-
ently monitor the efficiency of the individual project QC
programs;
6. To identify areas requiring new or improved measurement methods
to achieve the level of quality required to satisfy project
objectives.
-------
Section No. 4
Revision No. 0
Date: November 3, 1980
Page 1 of 6
SECTION 4
ORGANIZATION FOR DATA QUALITY
4.1 ORGANIZATIONAL STRUCTURE
The IERL-RTP organizational structure is shown in Figure 2. Figure 3
presents the laboratory's data quality program organization.
The chief of the Process Measurements Branch (PMB) is designated as the
QA Manager and on data quality matters reports to the IERL-RTP Director. The
OA Officer directs the activities of the QA group, which is composed of the
PMB staff members and contract support. The QA Officer is directly responsible
to the QA Manager.
The organizational hierarchy shows the quality control line moving from
the appropriate Division Director, to Branch Chief, Project Officer, then
contractor.
4.2 FUNCTIONAL RESPONSIBILITIES
The functional responsibility assignments for individuals and organiza-
tional components are given in this section.
4.2.1 Quality Assurance Manager
The chief of the Process Measurements Branch is the Quality Assurance
Manager. This person is responsible for the design, development, implemen-
tation, and maintenance of the IERL-RTP data quality program. The chief
directs the efforts of the QA Officer and thus the data quality activities of
the Process Measurements Branch.
4.2.2 Quality Assurance Officer
The Quality Assurance Officer is responsible for overseeing all IERL-RTP
data quality efforts. This person coordinates the activities of the Process
Measurements Branch in the data quality program. He/she works with the appro-
priate project officers in designing and implementing project-specific quality
control and quality assurance plans for both in-house and external projects.
-------
ADMINISTRATIVE OFFICER
JACK H. GREENE
629-2903
INDUSTRIAL ENVIRONMENTAL RESEARCH LABORATORY
RESEARCH TRIANGLE PARK. NORTH CAROLINA
DIRECTOR
MR. FRANK T. PRINCIOTTA
829-2821
DEPUTY DIRECTOR
(VACANT)
UTILITIES AND INDUSTRIAL POWER DIVISION
MR. EVERETT PLYLER
829-2915
EMISSIONS/EFFLUENT TECHNOLOGY BRANCH
MR. MICHAEL MAXWELL
628-2578
(ARTICULATE TECHNOLOGY BRANCH
MR. JAMES ABBOTT
629-2925
OFFICE OF PROGRAM OPERATIONS
DR. JOHN O. SMITH
629-29*1
SPECIAL STUDIES STAFF
DR. W. GENE TUCKER
629-2745
PLANNING. MANAGEMENT. AND
ADMINISTRATION STAFF
MR.C. T. RIPBERGER
629-2921
ENERGY ASSESSMENT AND CONTROL DIVISION
MR. ROBERT HANGEBRAUCK
629-2825
COMBUSTION RESEARCH BRANCH
DR. JOSHUA BOVYEN
629-2470
GASIFICATION/INDIRECT LIQUEFACTION
BRANCH
MR. T. KELLY JANES
629-2851
LIQUEFACTION AND PETROLEUM BRANCH
OR. DALE A. DENNY
629-2825
INDUSTRIAL PROCESSES DIVISION
MR. ALFRED B. CRAIG
629-2509
CHEMICAL PROCESSES BRANCH
MR. RICHARD STERN
629-2547
METALLURGICAL PROCESSES BRANCH
MR. NORMAN PLAKS
629-2733
-O O 30 CO
(u o> n> n>
to r»- < o
n> (D r+
to *
f\> ->. O
O 3
0> i
PROCESS MEASUREMENTS BRANCH
MR. JAMES DORSEV
629-2557
Figure 2. IERL-RTP organization chart.
(D O
-s
to
10
00
o
-------
Section No. 4
Revision No. 0
Date: November 3, 1980
Page 3 of 6
IERL-RTP
LABORATORY
DIRECTOR
QUALITY ASSURANCE FUNCTION
PROJECT MANAGEMENT FUNCTION
(INCLUDING QUALITY CONTROL)
DIVISION
DIRECTOR
CHIEF, PROCESS
MEASUREMENTS
BRANCH
BRANCH
CHIEF
QUALITY
ASSURANCE
CUM IHACTOR
QUALITY
ASSURANCE
OFFICER
PROJECT
OFFICER
QUALITY
ASSURANCE
STAFF
CONTRACTOR
Figure 3. Data quality program organization.
-------
Section No. 4
Revision No. 0
Date: November 3, 1980
Page 4 of 6
The QA-Officer also coordinates the IERL-RTP QA Program with the Agency-wide
QA Program through the QAMS.
4.2.3 Process Measurements Branch
The Process Measurements Branch is responsible for coordinating all
IERL-RTP QA activities. It initiates measures to ensure the fulfillment of
the overall quality objectives of the laboratory and to carry out the data
quality policies in the most efficient and economical manner. The data quality
responsibilities and authority of the Process Measurements Branch are:
1. To develop QC guidelines and QA programs, including statistical
procedures and techniques, that will help the laboratory meet
desired quality standards at minimum cost, and to coordinate
the implementation of such programs with the appropriate Project
Officer.
2. To review all measurement programs and ensure that appropriate
methods have been selected for data acquisitions.
3. To review QC activities of the various projects and make appro-
priate suggestions to the Project Officer regarding corrections
and improvement.
4. To seek out and evaluate new ideas and current developments in
the field of QA/QC and recommend implementation when advisable.
5. To advise project officers in preparing and/or reviewing requests
for proposals, work plans, project implementation, and reports
of work with respect to quality aspects of technology, methods,
and equipment.
6. To advise on packaging materials and procedures for sample
handling and on requirements for maintaining sample integrity.
7. To advise project officers concerning schedules for system
checks, calibrations, and other checking procedures.
8. To evaluate data quality statistically and maintain related QA
records and other pertinent information.
9. To coordinate the IERL-RTP program with the Agency's QAMS
program.
10. To prepare and issue periodic QA reports on specific projects
to the Project Officer and the IERL-RTP Director.
-------
Section No. 4
Revision No. 0
Date: November 3, 1980
Page 5 of 6
11. To prepare and issue periodic QA summaries to the IERL-RTP
Director and to the QAMS Director.
Quality assurance is under the direction of the QA Manager in the Process
Measurements Branch. PMB will have the necessary in-house and contract techni-
cal support to establish, maintain, and improve the data quality program.
4.2.4 Project Officer
The IERL-RTP Project Officer is ultimately responsible for the success or
failure of the project. This person thus has the responsibility for determin-
ing the required data quality criteria and the optimum level of quality control
for the project and for seeing that the QA program is implemented and main-
tained. The Project Officer is responsible for quality control practices,
beginning with the preparation of the RFP and extending through the final
report. Standard QA enclosure packages for RFP's, encompassing several options,
have been prepared (see Section 3.1.1).
4.2.5 IERL-RTP Contractors
For IERL-RTP projects conducted under contract, the contractor is respon-
sible for designing, developing, implementing, and maintaining a QC program to
ensure that the experimental data generated will be of suitable quality to
satisfy the project requirements, and for documenting that quality. The
quality control plan (QA Project Plan) must be approved by the IERL-RTP Project
Officer and the IERL-RTP QA Officer.
4.2.6 Functional Relationships
Once a task has been defined on a task-order contract or a contract
negotiated, the sequence of events and interrelationships among the Process
Measurements Branch, the Project Officer, and the contractor are as illustrated
in Figure 4. The first column in the figure shows the QA functions of the
Process Measurements Branch. The second column gives the functions of the
Project Officer (in-house project) or the Project Officer/contractor (contracts,
grants, and cooperative agreements). The lines across columns show the points
of interaction between different components of the QA Program.
-------
Section No. 4
Revision No. 0
Date: November 3, 1980
Page 6 of 6
QUALITY ASSURANCE GROUP
(Process Measurements Branch
Quality Assurance Officer)
QUALITY CONTROL GROUP
(Project Officer/Contractor)
Develop Standard
Measurement
Methods
Establish
Data Quality
Specifications
Review
Measurements
Problems
Define
Quality Control
Guidelines
Select
Msssuroniftnt
Methods
Develop
Quality Control
Program
Develop
Quality Assurance
Program
Implement
Quality Control
Perform
Quality Assurance
Audits
Issue
Quality Control
Reports
Issue
Quality Assurance
Repom
Figure 4. Flow chart of functional relationships.
-------
Section No. 5
Revision No. 0
Date: November 3, 1980
Page 1 of 4
SECTION 5
IMPLEMENTATION PLAN AND SCHEDULE
The comprehensive data quality program for IERL-RTP was predicated on the
philosophy that a quality program should be implemented gradually and in
phases so that the results from one phase can be used constructively in imple-
menting the next phase. Also, the differences involved in implementing data
quality programs in ongoing projects as compared to new projects have been
considered.
In accord with the recent QAMS directives, the implementation plan now
emphasizes the application of quality control and quality assurance procedures
to new projects, both in-house and contract, from the program's inception.
The Process Measurements Branch is available to assist the Project Officer
(contract projects) in specifying quality control requirements for inclusion
in the Rr? and evaluating QA/QC aspects of proposals and work plans. For new
in-house projects, the Process Measurements Branch is available to help Project
Officers prepare QC plans for their projects.
The general plan and schedule for developing and implementing a labora-
tory-wide data quality program, starting with the preparation of a planning
document, are given below.
5.1 IMPLEMENTATION SCHEDULE
During 1980, QAMS developed a number of strategies, guidelines, and
exemplary reports to aid the Agency's .Regional Offices, Program Offices, and
Laboratories in the development and application of QA programs. Figure 5
shows the ongoing implementation schedules for QAMS-directed activities.
Within IERL-RTP, a number of Laboratory-specific guidelines have already
been prepared. As part of IERL-RTP's continuing QA Program and implementation
of the recent Agency-wide QA Program, some of these documents are being updated
and reformatted to be consistent with Agency specifications. Figure 6 gives
IERL-RTP's schedule for revising and updating its Laboratory-wide data quality
program to be in agreement with the Agency's QA Program.
-------
Section No. 5
Revision No. 0
Date: November 3, 1980
Page 2 of 4
Agency QA PolicyGeneral
Agency QA PolicyContracts & Grants
Strategy for Developing Mandatory QA Program
Guidelines and Specifications for Preparing
QA Plans
Guidelines for Implementing QA Requirements
for Grants
Guidelines for Implementing QA Requirements
for Contracts
Designation of QA Officers*
Model QA Program Plans (Ambient Air, Water,
Region, Laboratory)
Model QA Project Plans (Ambient Air/Pollutant-
Specific, Water/Study-Specific, Region/Area-
Specific, Laboratory/Study-Specific)
QA Clearinghouse
QA Program Plans Submitted
Agency QA Report
QA Programs/Plans Approved
QA Cost Study Report
QA Workshops (QA Officers, Project Officers,
Grants Administrators)
QA Project Plans
Administrator
Administrator
ORD/Work Group
OMTS/QAMS
OMTS/QAMS
OMTS/QAMS
DAA's, Labs, RA's
ORD PO's, RO's
ORD PO's, RO's
OMTS/QAMS
PO's, Labs, RO's
OMTS/QAMS
OMTS/QAMS
ORD Labs, RO's
OMTS/QAMS
Labs, PO's, RO's
May 30,1979
June 14, 1979
February 13, 1980
February 22, 1980
February 22,1980
February 22,1980
March 1, 1980
March 24,1980
April 30, 1980
May 1, 1980
June 2, 1980
June 2,1980
September 15, 1980
September 15, 1980
(To Be Planned)
October 1,1980,
and Continuing
'Regions and ORD have already designated QA Officers.
Figure 5. QAMS implementation requirements and schedule.
-------
Section No. 5
Revision No. 0
Date: November 3, 1980
Page 3 of 4
As shown in Figure 6, a QA Program Plan for IERL-RTP, stating general
policies and describing general requirements, was submitted for QAMS approval
in June 1980. Guideline documents to aid contractors in preparing the more
specific Contract QA Project Plans and Cooperative Agreement QA Project Plans
will be available in the fall of 1980. A guideline document to assist EPA
Project Officers in the preparation of in-house QA Project Plans will also be
available in the fall of 1980. Standard clauses for inclusion in RFP's,
detailing the QA requirements for the various types of RFP's, will be available
to aid IERL-RTP Project Officers in implementing effective and uniform quality
assurance for all lERL-RTP-funded projects. This Data Quality Manual will
serve as a general guide for IERL-RTP Project Officers in implementing the
Laboratory's QA Program. A workshop to distribute and explain this information
is planned for the fall of 1980. This workshop will serve as a means for
collecting information on the productivity of applying quality assurance for
ongoing projects.
5.2 ESTIMATED COST OF IMPLEMENTATION
A definitive cost estimate for implementing and maintaining a data quality
program cannot be made at this time. However, with the degree of importance
placed on data quality by IERL-RTP, a realistic estimate of average costs per
project are: (1) quality control costs in the range of 8 to 10 percent of the
total project budget, and (2) quality assurance costs between 2 and 5 percent
of the total project budget. From these estimates, then, the costs of the
IERL-RTP data quality program as described in this manual, when fully imple-
mented, should be in the range of 10 to 15 percent of the total laboratory
project budget.
The above cost estimates apply to new projects. The net additional cost
for implementing a data quality program to an ongoing project will depend upon
the project's current level of QC and QA activities.
-------
Section No. 5
Revision No. 0
Date: November 3, 1980
Page 4 of 4
Milestone
Submit QA Program Plan
Develop Guideline Document for the Preparation of
Contract QA Project Plans
Revise IERL/RTP Data Quality Manual
Develop Standard QA Clauses for Inclusion in RFP's
Develop Guideline Document for the Preparation of
In-house QA Project Plans
Develop Guideline Document for the Preparation of
Cooperative Agreement QA Project Plans
Prepare a Training Course for IERL-RTP Project
Officers on Quality Assurance Principles and
Implementation of the Laboratory QA Program
Submission of QA Project Plans
Date
June 6,1980
July 25, 1980
August 15, 1980
August 30,1980
August 30, 1980
September 15, 1980
September 30,1980
October 1, 1980, and
Continuing
Figure 6. IERL-RTP QA Program implementation schedule.
-------
Section No. 6
Revision No. 0
Date: November 3, 1980
Page 1 of 16
SECTION 6
ELEMENTS OF A QUALITY CONTROL PROGRAM
6.1 GENERAL STATEMENT
The ultimate goal of a data quality program is to ensure the quality of
the data to a predetermined satisfactory level.
For the purposes of this document, a data quality program will be divided
into two separate functions: activities internal to the project contractor
(quality control) and activities externally carried out by an independent
organization (quality assurance). Figure 7 outlines the divisions and further
subdivisions of a data quality program as discussed in the contoxc of this
document. In this section, the internal functions of a data quality program
are discussed.
The QC Program is a sequence or scheme of activities initiated within the
line organization to ensure and assess the quality of data throughout a project.
A QC program can take many forms. It can be an in-depth program equal in
stature to the project or it can be secondary in importance to the project
goals. In short, a QC program serves to:
1. Evaluate the overall adequacy of the project concerning data
quality,
2. Identify existing and potential problems in the data-producing
system, from measurement to data reduction;
3. Stimulate research into and discussion of alternative methods
for obtaining data of the required quality; and
4. Improve any low-quality data or deficiencies that might exist.
In the following subsections, each specific area of a QC program is discussed.
6.2 FACILITIES AND EQUIPMENT
A good starting point in the assessment of an ongoing project is a general
survey of the facilities, equipment, and personnel available for day-to-day
operation. Are they adequate for the job at hand? Do standards exist for the
internal evaluation of facilities, equipment, and materials?
-------
Data Quality
Program (DQP)
Quality Assurance
QA for RFP's
Evaluation of QC in proposal
Work plan review
Qualitative systems audit
Quantitative performance audit
Det2 quality assessments
Final report review
Section No. 6
Revision No. 0
Date: November 3, 1980
Page 2 of 16
Quality Control
Activities
(QC Program)
Facilities and equipment
Configuration control
Personnel training
Documentation control
Control charts
In-process quality control
Procurement and inventory procedures
Preventive maintenance
Reliability
Data processing
Feedback and corrective action
Calibration
Figure 7. Overview of a data quality program.
-------
Section No. 6
Revision No. 0
Date: November 3, 1980
Page 3 of 16
The laboratories and data processing and other operational areas should
be neat and orderly, within common-sense limits imposed by the nature of the
operation. Laboratory benches, particularly areas where critical operations
such as weighing are carried out, should be kept clear of all but necessary
tools, glassware, etc. Personal items (coats, hats, lunch boxes) should not
be left in the work area. Provision should be made for storage of these items
in personal lockers. A neat, well-organized laboratory area inspires neatness
and organization in the laboratory workers.
Good laboratory maintenance, particularly for certain types of instru-
mentation, requires complete manuals that are readily available to appropriate
personnel. Responsibility for keeping up with any necessary manuals should be
assigned, with the understanding that the person responsible will devise a
check-in/checkout system for quick location of each document.
The specific requirements for all facilities, equipment, consumables, and
services are determined by the kinds of measurements made in a particular task
or project and the specific objectives (both technical and QA) of the program.
tvaluaiion of task-specific facilities, equipment, consumables, and services
is the responsibility of the Project Officer, in cooperation with the QA Officer.
6.3 CONFIGURATION CONTROL
For IERL-RTP projects of moderate to long-term duration, design changes
in the system must be documented unfailingly. Procedures for such documenta-
tion should be in writing and be accessible to any individual responsible for
configuration control. It is all too easy, as the system is modified repeat-
edly, to allow one key person to hold great amounts of vital information
largely by memory. Much of this information would be lost if this person were
no longer available. Current engineering schematics should be maintained on
both the system and subsystem level, and all computer programs should be
listed, dated, and flow charted. Changes in computer hardware and software
must be documented, even when such changes are apparently trivial. Significant
design changes must be documented and forwarded to the EPA Project Officer.
6.4 PERSONNEL TRAINING
For long-term projects, a planned training system for new employees is
highly desirable. This system should include motivation to produce data of
-------
Section No. 6
Revision No! 0
Date: November 3, 1980
Page 4 of 16
acceptable quality. A part of the program should involve "practice work" by
the new employee. The quality of the work can be immediately verified and
discussed with the supervisor, with appropriate corrective action taken. This
system is preferred to on-the-job training, which may vary in quality.
Key personnel (laboratory supervisors, senior engineers) should be required
to document their specialized knowledge and techniques as completely as possible.
They should each have an assistant, if the program personnel situation allows,
who can take responsibility when the senior person is unavailable. A most
undesirable situation arises when replacement personnel must gain knowledge of
the program through trial and error (see Subsection 3.3). This is not an
infrequent occurrence, however, when budgeting constraints override other
priorities.
A thorough personnel training program should focus particular attention
on those people whose work directly affects data quality (calibration person-
nel, bench chemists, etc.). These people must be cognizant of the quality
standards fixed for the project and the reasons for those standards. They
must be made aware of the various means of achieving and maintaining quality
data. As these people progress to higher degrees of proficiency, their accom-
plishments should be reviewed and then documented. A motivating factor for
high performance could be direct and obvious rewards (monetary, status, or
both) offered in a manner visible to other comparable personnel.
6.5 DOCUMENTATION CONTROL
If the project generates a number of documents, procedures for making
revisions to these documents must be clearly spelled out. The revisions
themselves should be written and distributed to all affected parties, thus
ensuring that the change will be implemented and become permanent. If a
technical document change pertains to an operational activity, that change
should be analyzed for side effects. The change should not be rendered perma-
nent until any harmful side effects have been controlled.
Revisions to computer software should be written with reasons for the
changes clearly spelled out. The revisions should be distributed to all
affected parties.
-------
Section No. 6
Revision No. 0
Date: November 3, 1980
Page 5 of 16
6.6 CONTROL CHARTS
For demonstration or pilot-plant programs, or any project where data are
taken on a long-terra basis, control charts are essential as a routine check on
the consistency or "sameness" of the data precision. A control chart should
be kept for each measurement that directly affects the quality of the data.
Typically, control charts are maintained for duplicate analyses, percent
isokinetic sampling rates, calibration constants, and the like. A sample
control chart is given in Figure 8. The symbol s (sigma) represents one
standard deviation of the difference, d, in two duplicate measurements, one of
which is taken as a standard or audit value. Two s is taken as a warning
limit and 3s as a control limit. If a laboratory measurement differs from the
audit value by more than 3s, the technique is considered out of control.
Control charts are dealt with in depth in a number of standard texts on quality
control of engineering processes.5
6.7 IN-PROCESS QUALITY CONTROL
During routine operation, critical measurement methods should be checked
for conformance to standard operating conditions (flow rates, reasonableness
of data being produced, and the like). The capability of each method to
produce data within specification limits should be ascertained by means of
appropriate control charts. When a discrepancy appears in a measurement
method, it should be analyzed and corrected as soon as possible.
For all standard methods, the operating conditions must be clearly defined
in writing, with specific reference to each significant variable. Auxiliary
measuring, gaging, and analytical instruments should be maintained operative,
accurate, and precise by regular checks and calibrations against stable stan-
dards that are traceable to a primary standard furnished by the National
Bureau of Standards, if available.
6.8 PROCUREMENT AND INVENTORY PROCEDURES
Purchasing guidelines for all equipment and reagents having an effect on
data quality should be well-defined and documented. Performance specifi-
cations should be documented for all critical items of equipment. Chemical
reagents considered critical to an analytical procedure are best procured
-------
Section No. 6
Revision No. 0
Date: November 3, 1980
Page 6 of 16
2a
-2o
-3a
CHECK NO.
ACTION LIMIT
UCL
WARNING LIMIT
-CL
WARNING LIMIT
ACTION LIMIT
LCL
8
10
DATE/TIME
OPERATOR
PROBLEM AND
CORRECTIVE
ACTION
Figure 8. Standard quality control chart.
-------
Section No. 6
Revision No. 0
Date: November 3, 1980
Page 7 of 16
from suppliers who agree to submit samples for testing and approval prior to
initial shipment. In the case of incoming equipment, there should be an
established and documented inspection procedure to determine if procurements
meet the quality control and acceptance requirements. The results of this
inspection procedure should be documented.
Whenever discrepant materials are detected, the materials should be
returned or disposed of, at the discretion of the quality control officer/
manager.
Once an item has been received and accepted, it should be documented in a
receiving record log giving a description of the material, the date of the
receipt, results of the acceptance test, and the signature of the responsible
individual. It is then placed in inventory, which should be maintained on a
first-in, first-out basis. It should be identified as to type, age, and
acceptance status. In particular, reagents and chemicals that have limited
shelf life should be identified as to shelf expiration date and issued from
stock only if they are still within that date.
Likewise the reliability and quality of all services (e.g., analytical
services, balance maintenance) provided shall be assessed. Evaluation of
services is usually accomplished by appropriate systems or performance audits.
6.9 PREVENTIVE MAINTENANCE
For long-term projects, it is important that preventive maintenance
procedures be clearly defined and written for each measurement system and its
support equipment. When maintenance activity is necessary, it should be
documented on standard forms maintained in logbooks. The maintenance record
of each system should be reviewed periodically to assess the adequacy of its
maintenance schedule and parts inventory.
Preventive maintenance schedules and procedures should be present for all
measurement and sampling systems used. These procedures should be explicit
and treat such items as expendables replacement (e.g., in-line filters and
pump diaphragms for sampling trains).
For field programs, all possible preventive maintenance (PM) should be
performed prior to arrival at a test site to minimize the probability of
downtime. All maintenance activities shall be performed by suitably qualified
technical personnel using accepted,, documented procedures according to the PM
-------
Section No. 6
Revision No. 0
Date: November 3, 1980
Page 8 of 16
plan. The desirability of full- or part-time equipment operator and/or mainte-
nance support must be considered. Frequently, sophisticated instrumentation
performs poorly or not at all when many occasional users have access to it.
On the other hand, minor but frequent maintenance often keeps an instrument
operating at peak performance. In such cases, the cost of a full-time dedi-
cated operator is justified.
Provisions for handling unscheduled maintenance should also be documented
to minimize downtime and the loss of data. All maintenance, scheduled or
unscheduled, should be recorded on standard forms in a specific manner.
Personnel will then have these records as a functional history of the system
in question.
6.10 RELIABILITY
The reliability of each component of a measurement system relates directly
to the probability of obtaining valid data from that system. It follows that
procedures for reliability data collection, processing, and reporting should
be clearly defined and in written form for each system component. Reliability
data should be recorded on standard forms and kept in a logbook. If this
procedure is followed, the data can be used in revising maintenance and/or
replacement schedules.
6.11 DATA PROCESSING
Data processing encompasses all manipulations performed on raw ("as
collected") information to change its form of expression, its location, its
quantity, or its dimensionality. This includes data collection and record-
keeping, validation, storage, transfer, reduction, and analysis.
Provisions for a complete, permanent, easily accessible record of the raw
experimental data should be made prior to, during, and following com-
pletion of task experimental work. This should include a written record (in
ink, in a bound, page-numbered notebook) of equipment serial numbers, re-
agents and supplies used, as well as a record of equipment modifications and
other seemingly inconsequential information that will permit more accurate
analysis of the data at a later date. When data are logged by computers, it
is important that adequate provision be made for redundant and physically
separate long-term storage of such records.
-------
Section No. 6
Re-'ision No. 0
Date: November 3, 1980
Page 9 of 16
Recordkeeping of this type serves at least two useful functions: (1) it
makes possible reanalysis of a set of data at a future time when the model
has changed significantly, thus increasing the cost-effectiveness of the
data; and (2) it may be used in support of the experimental conclusions if
various aspects of the study are called into question. This latter point
goes to the heart of scientific research: objectively, it is often possible
to interpret data in more than one wayand the raw data should be available
for evaluation by qualified professionals; subjectively, when recordkeeping
habits are sloppy, suspicion is quickly aroused that other aspects of the
project are of similarly poor quality.
In addition to the issues discussed above, the investment in the
design of suitable data logging forms for repetitively measured parameters
will be repaid. Computerized data acquisition systems have many advantages.
They ensure data completeness, higher productivity of technical personnel, and
ease of reading the raw data. However, they must be closely monitored for
false or erroneous signals that may not be easily detectable.
The QA Project Plan should address both manually collected and computer-
ized data acquisition systems. The internal checks that must be used to
ensure suitable quality in the data collection process should be identified.
Validation of raw data should also be addressed. Data validation has
been defined as "the process whereby data are filtered and accepted or rejected
based on a set of criteria." In the processing aspects of data validation,
the QA Project Plan should clearly indicate the procedures employed to ensure
that raw data are not altered, the data reduction method used, and a clearly
defined audit trail.
The validation process may include many forms of manual or computerized
checks, but it clearly involves specified criteria. Validation criteria may
include evaluation of the data with respect to physically determined checks
(e.g., a record indicating a negative weight is not reasonable). Similarly,
as the sophistication of the model increases, relational checks may also be
used.
Data validation procedures must be defined for each project. An environ-
mental assessment and a demonstration project will have entirely different
-------
Section No. 6
Revision No. 0
Date: November 3, 1980
Page 10 of 16
procedures, since in one case the data are taken on a "one-point" basis and in
the other a great quantity of data is accumulated over a long period of time,
usually years. Whatever the nature of the project, it is important that the
criteria for data validation be documented. Whenever practical, acceptance
limits should be established, these limits being subject to modification as
the program continues. Any required data validation activities should be
recorded in standard form in a logbook. Where possible (as in most demonstra-
tion projects), validation criteria should be programmed so that routine
data-taking will include automatic flagging of invalid data.
Data storage involves keeping the data in such a way that they are not
degraded or compromised, and that any datum (value) desired may be found
(uniquely identified). At every stage of data processing at which a "perma-
nent" collection of data is stored, there should be a physically separate
copy for purposes of integrity and security.
Study data must be securely stored (archived) in a suitable manner.
Such aspects as storage media, conditions, and location; access by authorized
personnel; and retention time must be addressed in the QA Project Plan.
_/ Each QA Project Plan will describe procedures to be used to characterize
data transfer error rates and how information loss is minimized in the
transfer. An overall admissible error rate, e.g., 5 percent, should be
specified by which the required component error rates for individual transfer
steps may be evaluated.
Data reduction includes all processes that change either the values or
numbers of data items: the original data set from which it is generated
cannot be recovered from it. QA Project Plans Will address both the internal
correctness of the data reduction processes and their appropriateness as
reflected by the end uses of the reduced (altered) data.
Data analysis involves comparison of a conceptual model against suitably
manipulated (i.e., reduced) data. It frequently includes computation of
summary statistics and their standard errors, confidence intervals, tests of
hypotheses relative to the parameters, and model validation (goodness of fit
tests). QA Project Plans will address the reliability of computations
(software QA), appropriateness of the model(s) as a framework for investi-
-------
Section No. 6
Revision No. 0
Date: November 3, 1980
Page 11 of 16
gating the study questions, and robustness of statistical procedures to model
inaccuracies (methodologic QA).
The QA Project Plan should also address potential problems in the data
analysis scheme.
6.12 FEEDBACK AND CORRECTIVE ACTION
Closely tied to the detection of invalid data is the establishment of a
closed loop mechanism for problem detection, reporting, and correction. Here
it is important that the problems are reported to those personnel who can take
appropriate action. In the QA Project Plan, a feedback and corrective action
mechanism should be written out, with individuals assigned specific areas of
responsibility. Again, problems encountered and actions taken should be
documented on standard forms (such as the one shown in Figure 9), which are
kept in a logbook. If appropriate, a periodic summary report on problems and
corrective action should be prepared and distributed to the appropriate levels
of management. This report should include: a listing of major problems for
the reporting period, names of persons responsible for corrective action,
criticaiity of problems, due dates, present status, trend of quality perform-
ance (i.e., response time), and a listing of items still open from previous
reports.
6.13 CALIBRATION
Calibration is the process of establishing the relationship of a measure-
ment system output to a known stimulus. In essence, calibration is a reprodu-
cible reference point to which all sample measurements can be correlated.
This process is a key element of any scientific measurement program, since
without a valid calibration or reference system, the validity of the data from
the measurement program will be questionable.
A sound calibration program should document frequency, conditions, stan-
dards, and records reflecting the calibration history of a measurement system.
Calibration procedures should be well-documented, step-by-step pro-
cedures for performing the needed referencing of a given system to a stand-
ard^). Whether the procedure uses a specific standard (as in the calibration
of spectrophotometer) for the referencing procedure or visual analysis by
-------
Section No. 6
Revision No. 0
Date: November 3, 1980
Page 12 of 16
CHANGE REQUEST
TITLE OF CNAN6E:
REQUESTED 8V:
SUPERVISOR SIGNATURE:
DATE:
DATE:
REQUIRED BY (DATE):
CHARGE SAFETY RELATED:
O YES D NO ;
EMERGENCY:
a YES D NO
DESCRIPTION OF CHANGE (ATTACH DIAGRAMS OR ADDITIONAL PAGES AS HECESSARY):
REASON AND JUSTIFICATION FOR CHANGE OR CONSEQUENCES IF NOT MADE:
EPA PROGRAM MANAGER APPROVAL:
(CONTRACTOR) PROGRAM MANAGER APPROVAL
(CONTRACTOR) QA PROGRAM MANAGER REVIEW:
DOCUMENTS AFFECTED:
1.
2.
1
4.
DATE RECEIVED:
EVALUATED BY:
REVISION TO DATA TAPE FORMAT REQUIRED:
Q YES O NO
REMARKS
Figure 9. Change request form.
-------
Section No. 6
Revision No. 0
Date: November 3, 1980
Page 13 of 16
trained personnel (e.g., a pathologist reading a microscope slide), a clearly
written and concise procedure is needed. A procedure of this type will help
minimize the bias that could be introduced into a system via operator tech-
nique. Calibration procedures for some devices such as wet test meters,
rotameters, etc., can be obtained from sources such as ASTM. Other procedures
may have to be developed in-house and must undergo extensive evaluation to
determine, as nearly as possible, the accuracy and precision (i.e., the
replicability, repeatability, and reproducibility of the procedure). To
ensure that the same calibration or reference point is maintained for a
measurement system, it is essential that a calibration schedule be initiated,
whether it involves simple daily checks or full-scale, multipoint calibrations.
Provisions for action to be taken if an unforeseen circumstance occurs
should be specified. Adherence to an exercise of this nature can minimize
the generation of erroneous and/or indefensible data.
Environmental conditions are another type of reference point that must
be dealt with when calibrating measurement systems. If the system is sensi-
tive to environmental conditions (temperature, pressure, light, humidity,
etc.), the calibration will not be valid unless the documented conditions
are maintained as required and the samples are analyzed in a like manner.
A system calibration should include a multipoint calibration on a
specified frequency. Since all instrumental methods are not characteris-
tically the same, the frequency for performing a multipoint will not be
addressed. Some systems may require a multipoint daily and others weekly. A
multipoint can give information concerning the linearity, the precision, and
the overall performance of the instrument. By handling the calibration points
with a specific routine, the quality of the calibration curve can be judged.
Figure 10 is an example of two calibration curves constructed using a linear
least squares fit. This type of procedure ensures that the calibration line
is drawn in a reproducible manner each time a multipoint is prepared. The
curves in Figure 10 are fitted with a linear equation in the form of
y = mx + b
-------
Section No. 6
Revision No. 0
Date: November 3, 1980
Page 14 of 16
LU
CO
o
0.
V)
0.5
0.4
0.3
0.2
m « 1.1019
fa .0023
r 1.0000
S. 0.0008
0.1 0.2 0.3 0.4
CONCENTRATION (PPM)
0.5
as
0.4
0.3
D.
V)
c 0.2
0.1
m = 1.1538
b - .0094
r - .9988
S = .0104
0.1 0.2 0.3 0.4
CONCENTRATION (PPM)
0.5
Figure 10. Typical calibration curves.
-------
Section No. 6
Revision No. 0
Date: November 3, 1980
Page 15 of 16
where
y = the system response,
m = the slope,
x = the concentration of the standards,
b = the intercept.
Parameters used to evaluate the quality of the calibration include the
correlation coefficient, r, and the standard error of estimate, s . In this
case the correlation coefficient is a measure of strength of the assumed
linear relationship between y and x. A calculated value of r = 1 (or -1)
indicates that all points fall on a straight line. Calculated values less
than one indicate that either the points are randomly scattered about the
regression line or they show a nonlinear relationship.
The standard error of estimate, s , is the standard deviation of the
differences between the actual y value and the value predicted by the regres-
sion equation for the same x value. Therefore, it is a measure of precision.
The calculated standard error of estimate is frequently used as an estimate of
the replicability (a) of the measurement method. A rule of thumb is that any
point deviating from the regression line by more than ±2 s is suspect and
should be repeated, especially if the correlation coefficient is less than
0.995. In any case, a calculated s > h percent of full scale for a 5-point
calibration should be sufficient cause to inspect the calibration/ analyzer
system and procedures and repeat the calibration after any identified problems
have been corrected.
With these definitions in mind, a comparison of the two curves in Figure 10
shows that one curve is more precise than the other. The correlation coeffi-
cient is equal to 1.0000, which indicates a perfect fit out to four decimal
places. A calculated s of 0.0008 ppm means that the standard deviation of
the points from the line is 0.0008 ppm. Inspection of the second curve reveals
that the quality of this curve is not as high as that of the first, as evidenced
by a smaller correlation coefficient and a larger standard error of estimate.
A calibration program should have provisions for determining the accepta-
bility of a calibration. For example, criteria should be specified for the
required number of calibration points, the calculated correlation coefficient,
-------
Section No. 6
Revision No. 0
Date: November 3, 1980
Page 16 of 16
and the standard error of estimate. Each point that does not fall on the
regression line should be tested in some fashion (e.g., is it within ±X percent
of the line) and the intercept should be tested for significant difference
from zero. Since all measurement systems have different characteristics, a
criteria specific to each system should be formulated.
The accuracy of the calibration standards is an important point to con-
sider since all data will be in reference to the standards used. A program
for verifying the accuracy of all working standards against primary grade
standards should be initiated. Where possible, this program should include
the use of NBS standard reference materials. All working gas standards pur-
chased should not be put into service until the concentration, as specified by
the manufacturer, is verified against an acceptable primary standard. All too
often, standards degrade or are prepared under loose quality control guidelines
and are not of the accuracy desired. Appendix A, Traceability Protocol for
Establishing True Concentrations of Gases Used for Calibration and Audits of
Air Pollution Analyzers (Protocol No. 2), provided detailed criteria for
traceability to primary standards and for performing calibrations.
-------
Section No. 7
Revision No. 0
Date: November 3, 1980
Page 1 of 7
SECTION 7
GUIDELINES FOR QUALITY ASSURANCE PROGRAMS
7.1 GENERAL STATEMENT
The objective of a quality assurance program is to independently evaluate
the quality control program of a project. The QA program must be appropriate
to the work being done and to the QC program for the project data. As mentioned
earlier, each project must have provision for an adequate quality control
program. Section 7.2 gives appropriate quality control statements for the
RFP, and Sections 7.3 and 7.4 deal with the evaluation of quality control in
the proposal and the work plan. The initial stage of a QA program should be
assistance in planning and developing the project data QC program. Sections
7.5 and 7.6 discuss audits, and Sections 7.7 and 7.8 address data quality
assessments and evaluation of the final report.
Short-term experimental projects must have rigorous quality control of
the ensuing data. Quality assurance on such projects may consist of "one-shot"
audits, or there may be no formal quality assurance because of time constraints.
A minimum QA approach would involve sampling a percentage of the raw data
collected and verifying the calculations. Precision and accuracy of all
measurements should be ascertained. The techniques used and general approach
could also be reviewed for appropriateness, and an attempt could be made to
compare the work being done with that of other investigators. This can be
done offsite, if necessary, and at minimal expense.
For IERL-RTP projects of moderate to long duration, the assessment of
quality control should normally consist of a series of systems and performance
audits. The frequency of such audits obviously should be dictated by the
specific project, but a minimum of once each calendar year is recommended.
The initial systems and performance audits should take place prior to startup
or at least within the first quarter of the first project year. Subsequent
scheduling should be dependent on the requirements of management and the
apparent quality of the day-to-day data being obtained. More frequent audit-
ing may be necessary in the initial stages of the project. Judicious use
-------
Section No. 7
Revision No. 0
Date: November 3, 1980
Page 2 of 7
of mailable audit materials when available and review of QC reports may aid in
minimizing the number of onsite audits.
7.2 THE REQUEST FOR PROPOSALQUALITY CONTROL ASPECTS
An RFP is designed to state, as clearly as possible, what the objectives
of the project are; e.g., to design, construct, and maintain a given control
system, systematically examining the interaction of appropriate system parame-
ters. The quality of the data obtained from the project will depend upon many
factors, including instrumentation, personnel, sampling technique, sample
size, and statistical expertise. It is therefore critical that the RFP be as
explicit as possible in delineating what quality of data (specifically, accur-
acy and precision) is expected, and how that quality is to be insured. Gener-
ally speaking, the RFP should require that the bidding organizations address
each of the major areas of quality control discussed in Sections 6.2 through
6.13 of this manual.
Since most RFP's are limited in length, it would be inappropriate in most
cases to include more than a brief statement of quality control requirements.
Nevertheless, it is most important that the bid solicitation be as explicit as
possible concerning quality control. In those cases where an RFP is quite
lengthy, the quality control statement may be several pages long. The RFP
should specify the expected number of audits and anticipated frequency/dates
so that bidders to IERL-RTP can plan for QA costs and time requirements.
As mentioned in Section 3.1.1, standard statement/enclosures regarding QA
Programs have been prepared for several types of projects. These standard
enclosures will aid the IERL-RTP Project Officer in planning an adequate QA/QC
program. The enclosures will also aid IERL-RTP in establishing uniform QA
requirements. The enclosures will not be complete in all instances and addi-
tional information may be needed to meet special project requirements. The
standard QA enclosures are shown in Appendix B.
7.3 EVALUATION OF QUALITY CONTROL IN THE PROPOSAL
The proposal should contain a statement of the precise position the
bidder takes regarding quality control programs. This statement is hereafter
referred to as the bidder's QA Program Plan. This should include past projects
-------
Section No. 7
Revision No. 0
Date: November 3, 1980
Page 3 of 7
and the quality control program effectiveness in that project. In particular,
there should be a clear and explicit response to the quality control require-
ments stated in the RFP. This response must be compared directly, item-by-item,
with other proposals submitted against the RFP. The evaluation should result
in a determination of a "figure of merit" for the bidder's quality control
organization and the competence of the staff.
There should be provision for changes in procedures when it is evident
that data being obtained are not sufficiently accurate or appropriate for the
intent of the project as outlined by the Project Officer.
If a contractor has a good proposal but is unclear on some phases of data
quality, it would seem worthwhile to have him clarify his proposal by asking
him specific questions. If the answers to these questions are still vague, it
is a good indication that the quality for these phases of the project may be
questionable if this contractor carries out the project.
7.4 EVALUATION OF THE QA PROJECT PLAN
The work plan should be a detailed accounting of the actual steps to be
taken to complete the work delineated in the proposal and should be in direct
accord with the requirements of the RFP and other agreements with the Project
Officer. Particular attention should be paid to critical quality control
areas to realize the collection of data having acceptable precision, accuracy,
representativeness, and completeness. Figure 11 lists the required topics to
be considered. The statement concerning QA/QC in the work plan is referred to
as the bidder's QA Project Plan. A guidelines document on QA Project Plans
has been prepared to aid contractors and Project Officers in planning the
QA/QC for a project (Document No. EPA- ).
In cases where the submitted proposal has been accepted but lacks the
completeness required by the Project Officer, the problem areas should be
directly addressed in the work plan, showing the details of the work to be
done.
The work plan is generally required within 30 days after award of the
contract and must be submitted to the Project Officer before any work is begun
by the contractor. The plan can be accepted in draft form, which will allow
for minor changes prior to the final plan's acceptance and approval. The QA
-------
Section No. 7
Revision No. 0
Date: November 3, 1980
Page 4 of 7
1. Title page, with provisions for approval signature
2. Contents
3. Project description
4. Project organization and responsibilities
5. QA objectives for measurement data in terms of precision, accuracy, completeness,
representativeness, and comparability
6. Experimental design
7. Personnel qualifications
8. Facilities and equipment
9. Preventive maintenance procedures and schedules
10. Consumables and supplies
11. Recordkeeping
12. Documentation control
13. Configuration control
14. Sample collection
15. Sample custody
16. Sample analysis procedures
17. Calibration procedures and references
18. Data validation
19. Data processing and analysis
20. Internal quality control checks
21. Performance and systems audits
22. Specific procedures to routinely assess data precision, accuracy, and completeness of
specific measurement parameters involved
23. Feedback and corrective action
24. Quality assurance reports to management
25. Research report design
Figure 11. Format for QA project plans.
-------
Section No. 7
Revision No. 0
Date: November 3, 1980
Page 5 of 7
Project Plan must be submitted (through the Project Officer) for approval of
the IERL-RTP QA officer.
The QA Officer within IERL-RTP will review all QA Project Plans, provide
input, recommend change, and approve final plans. The QA Officer will maintain
a current file of all approved QA Project Plans and Standard Operating Proce-
dures for all measurement activity programs under the auspices of IERL-RTP.
7.5 THE ONSITE QUALITATIVE SYSTEMS AUDIT
The objective of the onsite qualitative systems audit is to assess and
document: facilities; equipment; systems; recordkeeping; data validation;
operation, maintenance, and calibration procedures; and reporting aspects of
the total QC program for a project. The review should:
1. Identify existing system documentation; i.e., maintenance
manuals, organizational structure, operating procedures, etc;
2. Evaluate the adequacy of the procedures as documented; and
3. Evaluate the degree of use of and adherence to the documented
procedures in day-to-day operations, based on observed condi-
tions and a review of applicable records on file.
To aid the auditor in performing the review, a checklist is included as
Appendix C. This checklist should be modified, as appropriate, for various
projects.
7.6 THE PERFORMANCE AUDIT
In addition to a thorough onsite systems review, quantitative performance
audits should be periodically undertaken. The objective of these audits is to
evaluate the validity of project data by independent measurement techniques.
It is convenient to classify the major measurement methods into five areas:
source gas and particulates, ambient air, water, solids, and process parameters
(physical measurements). Appendix D lists in table form a number of standard
techniques for auditing in these three areas. The specific techniques vary
widely from project to project, but for the analytical phase the audit tech-
nique generally involves use of reference samples of known composition and/or
splitting a sample among several laboratories for independent analyses. It is
desirable to perform calibration checks on individual system components,
-------
Section No. 7
Revision No. 0
Date: November 3, 1980
Page 6 of 7
and/or do side-by-side sampling runs to compare both sampling and analysis
technique precision and accuracy.
7.7 DATA QUALITY ASSESSMENT
Standard methods exist for estimation of the precision and accuracy of
measurement data. Efficient use of the audit data requires that a rationale
be followed that gives the best possible estimates of precision and accuracy
within the limits imposed by timing, number of samples taken, and the general
situation at the project site.
Appendix E lists statistical definitions and techniques often used in QA
work. Other statistical techniques exist that may apply to specific projects
(or to highly specialized areas of a given project). For projects of suffi-
cient duration, it is usually worthwhile to acquire the services of a statis-
tical consultant to most effectively treat the available data.
As a general guide to expected data quality for a number of reference
methods, Appendix F lists in table form both ambient air and source sampling
methods. An estimate of the method bias and precision with comments on major
error sources is given, and the appropriate EPA quality assurance guideline
document for each method is referenced.
7.8 QA EVALUATION OF THE FINAL REPORT
The final report summarizes the raw data, data quality, the precision and
accuracy of measurement methods, the methods of engineering and statistical
analysis, and quality control chart information. All raw data should be
included in the final report, if possible. If it is excessive, then one
computer printout might be filed with IERL-RTP to back up the laboratory
notebook maintained by the contractor should any problems arise in interpreta-
tion and analysis at a later date.
Methods of analysis of data quality should be provided unless they are
well-documented in the literature, in which case they may be referenced.
A statement of limitation, if any, and applicability of the results
should be given in the report. This may be part of an executive summary or a
separate section.
The results of the project as reported in the final report are assessed
by the QA officer and PMB against the data quality objectives set forth in the
-------
Section No. 7
Revision No. 0
Date: November 3, 1980
Page 7 of 7
RFP and the details provided in the work plan. The contractor could be rated
according to a checklist of items to be used in similar studies. A copy of
the evaluation is given to the director of IERL-RTP, to the project officer,
and to the QA officer. This file of project evaluation closes the loop and
provides useful information for future projects, proposal evaluations, RFP
statements, and alternate improvements of data quality in all IERL-RTP projects
in the area of activity.
-------
Section No. 8
Revision No. 0
Date: November 3, 1980
Page 1 of 1
SECTION 8
REFERENCES
1. Glossary and Tables for Statistical Quality Control, the American
Society for Quality Control, Milwaukee, Wisconsin, 1973.
2. Quality Assurance Handbook for Air Pollution Measurement Systems,
Volume 1, Principles. EPA-600/9-76-005.
3. "Planning Document for a Control Systems Laboratory Quality Assurance
Program," Final Report for EPA Contract No. 68-02-1398, Task 8, December
1974.
4. "Guidelines for Demonstration Project Quality Assurance Programs,"
Final Report for EPA Contract No. 68-02-1398, Task 20, January 1976.
5. "A Quality Assurance Program for the EPA Wet Limestone Scrubber Demon-
stration Project, Shawnee Steam-Electric Plant, Paducah, Kentucky,"
Final Report for EPA Contract No. 68-02-1398, Task 20, January 1976.
6. "Development and Trial Field Application of a Quality Assurance Program
for IERL Projects," Final Report for EPA Contract No. 68-02-1398,
Task 20, January 1976.
7. EPA-600/1-79-013.
8. Quality Assurance Handbook for Air Pollution Measurement Systems,
Vol. 1. Principles. EPA-600/9-76-005.
-------
APPENDIX A
TRACEABILITY PROTOCOL FOR ESTABLISHING TRUE CONCENTRATIONS OF
GASES USED FOR CALIBRATIONS AND AUDITS OF AIR POLLUTION
ANALYZERS (PROTOCOL NO.2)
SOURCE: Quality Assurance Handbook for Air Pollution Measurement
Systems. Vol. II Ambient Air Specific Methods, EPA-600/4-
77-027a, May 1977 (Revised June 15, 1978).
-------
Section No. A
Revision No. 0
Date: October 20, 1980
Page 1 of 12
7.0 TRACEABILITY PROTOCOL FOR ESTABLISHING TRUE CONCENTRATIONS
OF GASES USED FOR CALIBRATIONS AND AUDITS OF AIR POLLUTION
ANALYZERS (PROTOCOL NO. 2)
The traceability procedure described here is intended to
minimize systematic and random errors during the analysis of
.1
calibration and audit gas standards and to establish the true
concentrations by means of National Bureau of Standards, Standard
Reference Materials (NBS-SRM's)". The procedure provides- for a
«
direct comparison between the calibration and audit gas standards
and an NBS-SRM or a gas manufacturer's primary standard (GMPS)
which is referenced to an NBS-SRM; all comparisons are made using
instruments calibrated with applicable NBS-SRM's. Traceability
may be performed by the gas standard manufacturer at the time of
purchase or by the user after purchase.
This procedure is applicable to any continuous, semicon-
tinuous, or periodic analysis instrument which meets the per-
Iwiuiaiicc requirements in the following sections.
7.1 Establishing Traceability of Commercial Cylinder Gases
to NBS-SRM Cylinder Gases
7.1.1 Procedure for Instrument Calibration - The following
procedures for periodic multipoint calibrations and daily instru-
ment span checks are prescribed to minimize systematic errors.
Separate procedures for instrument span checks are described for
linear and nonlinear instruments. For this purpose, a linear
instrument is defined as one which yields a calibration curve
which deviates by £2% of full scale from a straight line drawn
from the point determined by the zero gas to the highest cali-
bration point. To be considered linear, the difference between
the concentrations indicated by the calibration curve and the
straight line must not exceed 2% of full scale at any point on
the curve.
7.1.1.1 Instrument Multipoint Calibration - A multipoint cali-
bration curve is prepared monthly using two SRM cylinder gases
-------
Section No. A
Revision No. 0
Date: October 20, 1980
Page 2 of 12
and the zero gas. The zero gas must not contain more than 0.2%
of the full-scale concentration of the component being analyzed.
In addition, the zero gas must be free of any impurity that will
give a response on the analytical instrument. The SRM cylinder
gases recommended for traceability of commercial cylinder gases
are listed in Table 7.1.
The multipoint calibration is accomplished by diluting the
highest concentration SRM with zero gas using a calibration flow
system. Obtain the instrument response for six points represent-
ing 0%, 10%, 30%, 50%, 75%, and 100% of the instrument's full
scale. Plot the data and construct the calibration curve.
Obtain the instrument response for the other SRM of lower con-
centration without dilution. Compare the apparent concentration
from the calibration curve to the true concentration of the lower
SRM. -If the difference between the apparent concentration and
the true concentration of the lower SRM exceeds 3% of the true
concentration, repeat the multipoint calibration procedure. Test
the calibration curve for linearity as defined above and proceed
to either 7.1.1.2 or 7.1.1.3.
7.1.1.2 Instrument Span Check for Linear-Response - At the start
of each day that cylinder gases are to be analyzed, check the
instrument's response to the highest SRM (or GMPS) in the range
to be used and check its response to zero gas. Adjust the re-
sponse to the value obtained in the most recent multipoint
calibration, and proceed to Subsection 7.1.2. Cylinder gases
analyzed with a linear instrument must not have a concentration
greater than 15% above the highest available SRM concentration.
7.1.1.3 Instrument Span Check for Nonlinear-Response - At the
start of each day that cylinder gases are to be analyzed, check
the instrument's responses to two SRM's (or GMPS's) in the range
of calibration gases to be analyzed and check responses to zero
gas 'as follows. First, set the instrument zero with zero gas and
then adjust the instrument response to the highest SRM (or GMPS)
to the value obtained in the most recent multipoint calibration.
-------
Section No. A
Revision No. 0
Date: October 20, 1980
Page 3 of 12
Table 7.1 NBS-SRM AVAILABLE FOR TRACEABILITY OF
CALIBRATION AND AUDIT GAS STANDARDS
Cylinder Gases
SRM
1683
1684
1685
2613
2614
1679*
Type
Nitric oxide in N2
Nitric oxide in N2
Nitric oxide in N2
Carbon monoxide in air
Carbon monoxide in air
Carbon monoxide in N2
Size,
liters at STP
870
870
870
870
870
870
Nominal cone,
ppm
50
100
250
18
42
95
*This SRM should not be used with Flame lonization Detector
(FID) analytical instruments.
Permeation Tubes
SRM
1625
1626
1627
1629
Type
Sulfur dioxide
Sulfur dioxide
Sulfur dioxide
Nitrogen dioxide
Tube
length,
cm
10
5
2
Permeation
rate,
ug/min
at 25°C
2.8
1.4
0.56
1.0
Concentration, ppm,
at flow rates of
1 £/min
1.07
0.535
0.214
0.5
5 £/min
0.214
0.107
0.0428
0.1
-------
Section No. A
Revision No. 0
Date: October 20, 1980
Page 4 of 12
Next, obtain the response to the lower SRM (or GMPS). If the re-
sponse to the lower SRM (or GMPS) varies by greater than 3% from
the response obtained in the most recent multipoint calibration,
a full multipoint calibration must be performed as in 7.1.1.1,
otherwise, proceed to 7.1.2. Calibration gases analyzed with a
nonlinear instrument must not have a concentration greater than
the highest available SRM concentration.
7.1.2 Procedure for Analysis of Cylinder Gases for True Concen-
tration - The following procedure is designed to establish the
*
true concentration of a cylinder gas. The analyses involve the
direct comparison of the cylinder gas to the SRM (or GMPS) in
order to compensate for variations in instrument response between
the time of daily span check and the time of analysis; signifi-
cant variations in response often result from changes in room
temperature, line voltage, and so forth. Analyses are performed
in triplicate to expose erroneous data points and excessive
random variations in the instruments response. After the gas
cylinder has been filled, a minimum of 4 days holding time must
be observed before the following protocol analyses are initiated.
1. Analyze each cylinder gas directly against the nearest
SRM (or GMPS) by alternate analyses of the SRM and calibration
gas in triplicate (3 pairs). Adjust the instrument span if
necessary prior to the analyses, but do not adjust the instrument
during the triplicate analyses. The response to zero gas must be
obtained with sufficient frequency that the change in successive
zero responses does not exceed 1% of full scale.
2. For each of the six analyses, determine the apparent
concentration of the SRM (or GMPS) or cylinder gas from the
calibration curve.
3. For each pair of analyses, one SRM (or GMPS) and one
cylinder gas, calculate the true concentration of the cylinder
gas by:
-------
Section No. A
Revision No. 0
Date: October 20, 1980
Page 5 of 12
True cone, of cyl. gas = apparent cone, of cyl. gas
v true cone of SRM (or GMPS) «vma-r-i«rt ; »
apparent cone, of SRM (or GMPS)* *gura.°n /-i
4. Determine the mean of the three values for the trv.e
concentration of the cylinder gas.
5. If any one value differs from the mean by greater than
1.5%, discard the data, reset the instrument span, if necessary,
and repeat steps 1 to 4.
7.1.3 Use of Gas Manufacturer's Primary Standards - The GMPS's
are gas mixtures prepared in pressurized containers and analyzed
against SRM cylinder gases. Their purpose is to conserve SRM's
where large quantities of gas cylinders are analyzed. A GMPS may
be substituted for an SRM for instrument span checks in Subsec-
tions 7.1.1.2 and 7.1.1.3 and for the cylinder gas analysis in
Section 7.1.2 if the following conditions are met. In no case
may a GMPS be substituted for an SRM for the required instrument
multipoint calibrations in Subsection 7.1.1.1.
1. A GMPS must have been analyzed against SRM cylinder
gases as described in Subsections 7.1.1 and 7.1.2 within 30 days
of their use for cylinder gas analysis. It is preferred that the
GMPS be analyzed on the days that instrument multipoint calibra-
tions are performed.
2. A GMPS must not have changed in concentration by more
than 1%/mo (average) for the 3-mo period prior to their use for
cylinder gas analysis.
7.1.4 Verification of Cylinder Gas Stability - The stability of
reactive gases (including the cylinder gases of nitric oxide and
carbon monoxide) must be verified before use. The stability of
each cylinder gas is verified by performing a second set of trip-
licate analyses (using the procedure in Subsection 7.1.2) a
minimum of 7 days after the first set of triplicate analyses.
The mean of the second triplicate analyses must not differ from
the mean of the first triplicate analysis by more than 1.5%.
-------
Section No. A
Revision No. 0
Date: October 20, 1980
Page 6 of 12
7.1.5 Reanalysis of Cylinder Gases - Reanalysis by the gas
manufacturer or user must be performed every 6 mo from the last
analysis date by the procedure in Subsection 7.1.2. Cylinder
gases used for audits may need to be analyzed more frequently
than every 6 mo.
7.1.6 Minimum Cylinder Pressure - No cylinder gas should be used
below a cylinder pressure of 200 psi, as shown by the cylinder
gas regulator.
7.1.7 Cylinder Label and Analysis Report - Each gas cylinder
should contain the following minimum traceability information on
a gummed label affixed to the cylinder wall and/or a tag attached
to the cylinder valve:
1. Cylinder number.
2. Mean concentration of cylinder gas, ppm or mol%.
3. Balance gas used.
4. Last analysis date.
5. Expiration date (6 mo after last analysis date).
In addition, a written analysis report should be prepared
which certifies that the cylinder gas has been analyzed according
to this protocol. The analysis report should contain the follow-
ing information:
1. Cylinder number.
2. Mean concentration of cylinder gas, ppm or mol%.
3. Replicate analysis data.
4. Balance gas used.
5. NBS-SRM numbers used as primary standards.
6. Analytical principle used.
7. Last analysis date.
The user should maintain a file of all analysis reports for 3 yr.
-------
Section No. A
Revision No. 0
Date: October 20, 1980
Page 7 of 12
7.2 Establishing Traceability of Commercial Permeation Devices
To NBS-SRM Permeation Tubes
7.2.1 General - The use of permeation devices to generate cali-
bration gas mixtures is widely used, especially with reactive
gases for which generation of low (sub-ppm) concentrations is not
practical using other techniques. If the permeation device is
held at a constant temperature with a constant flow of dry air, a
constant concentration of the permeating gas will result in the
gas stream.
Permeation devices of different'designs are available. The
standard permeation tube consists of a Teflon tube filled with
the liquified gas sealed at each end with Teflon plugs. Permea-
tion occurs along the length of the tube. A modification of this
is the permeation device consisting of a metal or glass tube
filled with the liquified gas with a seal of Teflon at one end
and the gas permeating through the end of the device. These
devices may be used to generate a lower permeation rate than
standard tubes. They also have an extended shelf life and some
may be operated at a higher temperature than standard permeation
tubes. The procedures for determining traceability are the same
for permeation tubes or devices although precautions for use may
vary.
Two gases of primary importance to air pollution analysis
are SO_ and N02. The NBS has developed and made available a per-
meation tube for SO2 and a permeation device for NO2- These
SRM's listed in Table 7.1 are considered primary standards.
Permeation tubes and devices are also available commercially
or can be made by the users themselves. When these tubes and
devices are compared to NBS-SRMfs they may be treated as second-
ary or working standards. The traceability or comparison of com-
mercial permeation devices to SRM's follows.
7.2.2 Procedure for Instrument Calibration - For direct compari-
sons of NBS-SRM's with commercial permeation tubes and devices, a
calibrated analysis instrument having a measurement range of
-------
Section No. A
Revision No. 0
Date: October 20, 1980
Page 8 of 12
0.50 ppm is needed. A multipoint calibration of the analysis
instrument in the specified measurement range (0 to 0.50 ppm for
S02, 0 to 0.50 ppm for N02) at 0%, 15%, 30%, 50%, and 85% of
the instrument full scale is necessary.
A multipoint calibration is accomplished by passing a flow
of 50 to 500 cm /min of dry air or nitrogen over the SRM and
mixing this flow with dilution flows of 1 to 10 £/min. (Precau-
tions for the use of permeation devices in Subsection 7.2.5 must
be followed. ) The concentration of gas in ppm by volume can be
calculated using the following formula:
Cppm = -FT-L Equation 7-2
where
C = V/V of pormeand transferred to a gas flowing over
the SRM, ppm;
P = known weight loss of SRM at calibrated temperature
from NBS-SRM value, pg/min;
G = molar volume = 24.45 £ at 25°"C and 760 mm Hg;
M = molecular weight of gas in SRM, g/mole; and
L = flow rate in £/min corrected to STP (25°C, 760 mm
Hg).
A plot of the SRM-generated concentration versus the instru-
ment response should give a linear relationship over the measure-
ment range of the instrument. The calibration curve should be
constructed using least squares regression computation.
7.2.3 Procedure for Analysis of Commercial Permeation Tubes and
Devices for True Permeation Rate - Once the multipoint calibra-
tion of the instrument is completed using the SRM, the permeation
rate of the commercial permeation tube or device can be deter-
mined using the analysis instrument and the calibration curve.
Prior to analysis of the commercial permeation tube or device, it
must be equilibrated at the temperature at which it will be used.
-------
Section No. A
Revision No. 0
Date: October 20, 1980
Page 9 of 12
The commercial tube or device must be equilibrated at a specified
temperature with a fixed flow of dry air or nitrogen passing over
it for at least 36 h before analysis (see Subsection 7.2.5).
The commercial permeation tube or device may have a higher
or lower permeation rate than the SRM used to calibrate the
analyzer. However, by adjusting the dilution flow, the concen-
tration generated can be adjusted to read at some value approxi-
mately 50% of full scale on the calibrated instrument range. The
procedure for analyzing the tube or device should be as follows:
1. The commercial permeation tube or device should equili-
brate when held at a constant temperature, with dry air or nitro-
3
gen at 50 to 500 cm /min passing over it for 36 h prior to
analysis.
2. On the day of analysis, the zero of the instrument
should be set using zero air before span points are generated.
3. Two span points (at 50% and 85% of full scale) should
be generated using the SRM to check that the previously generated
multipoint calibration of the instrument is still valid. If
either of the two span points deviate from the calibration curve
more than 2%, a new multipoint calibration must be run before
analysis.
4. Once the calibration of the analyzer is checked, a
concentration, C,, should be generated using the commercial tube
or device which should be 60% to 85% of the full-scale response
of the instrument calibration. The instrument response for the
concentration should be recorded, and the concentration related
to the SRM from the calibration curve.
A second concentration, C2, should be generated using the
commercial tube or device at a different flow rate, which is be-
tween 20% to 50% of the full-scale response of the instrument.
The instrument response for this concentration C, should be re-
corded, and the value for the concentration obtained from the
calibration curve.
-------
Section No. A
Revision No. 0
Date: October 20, 1980
Page 10 of 12
The permeation rate for the commercial permeation tube or
device can be determined relative to the SRM with the following
information.
1. Flow rate (F,) over the commercial tube or device for
given response Cppm.
2. The response C of the commercial tube or device in ppm
at flow F, based on the multipoint calibration curve using the
SRM.
3. The formula to be used to calculate the permeation rate
(Pr) is:
p - CPPm M L
rr ~ G
where
P = permeation rate,
C = response C for tube or device at F^ from
calibration curve, ppm;
M = molecular weight of gas in commercial
permeation tube or device, g/mole;
L = flow rate F, over commercial tube or device
to generate response C _, £/min; and
G = molar volume = 24.45 £ at 25°C and 760 mm Hg.
Once the permeation rate of the commercial tube or device
is calculated using C-,, the permeation rate can be calculated
using C2 in ppm and using F~ in £/min.
The values for the permeation rate based on two generated
concentrations using two flow rates should agree within 4%. If
the two values do not agree within 4%, the linearity of the
analysis instrument and the measured flow rates over the com-
mercial tube or,device should be checked.
7.2.4 Use of Commercial Permeation Tube or Device as Standards -
The procedure described above is a method of relating the permea-
tion rate of a commercial tube or device with the permeation rate
of an NBS-SRM.
-------
Section No. A
Revision No. 0
Date: October 20, 1980
Page 11 of 12
The commercial -permeation tube or device once calibrated may
be used as a secondary or working standard if the requirements in
Section 7.2.5 and 7.2.6 are observed.
The accuracy of the concentration produced is dependent not
only on the accuracy of calibration of the permeation tube or
device, but also on the accuracy at which the rate of flow over
the tube or device and the temperature are known.
7.2.5 Precautions for Use and Storage of Permeation Tubes and
Devices -
1. When a tube is transferred from the storage state to
the use state, the commercial thin-walled permeation tube must be
equilibrated at a fixed temperature and flow rate for at least
24 hours before using. Thick-walled permeation tubes or devices
must be equilibrated for at least 36 hours before using. For
commercial tubes or devices consult manufacturer's specifica-
tions.
2. The rate of permeation for standard permeation tubes
rr.ay be adversely affected by exposure to temperature greater than
35°C and by exposure to moisture. Permeation devices may be
operated at temperatures as high as 50°C. Check manufacturer's
specifications.
3. The permeation tube or device must be stored under dry
conditions preferably between 20° to 25°C. Low temperature
storage of permeation tubes or devices is not recommended.
4. Tubes and devices should be shipped in a closed con-
tainer with a desiccant.
7.2.6 Reanalysis of Permeation Tubes and Devices - NBS-SRM per-
meation tubes and devices which are protected against excessive
heat and moisture will perform satisfactorily for the life of the
tube without recalibration. The commercial permeation tube or
device must be reanalyzed 6 mo into its life even if it is kept
in a stable, dry environment. It also must be reanalyzed after
every excursion into the field.
-------
Section No. A
Revision No. 0
Date: October 20, 1980
Page 12 of 12
Reanalysis should consist of comparison with an NBS-SRM as
described in Subsections 7.2.2 and 7.2.3.
7.2.7 Record of Use and Analysis Report for Permeation Tubes and
Devices - A record of NBS-SRM comparisons should be kept for each
permeation tube as well as a history of trips into the field and
other excursions into uncontrolled environments. In additions,
the following information should be kept:
1. Permeation tube or device number.
2. Permeation tube or device concentration pg/min at
*
specified °C.
3. Calibration data.
4. SRM numbers used in calibration.
5. Analytical measurement technique used in comparisons.
6. Comparison dates.
The user should keep a record of the NBS-SRM comparisons for
three years.
-------
APPENDIX B
STANDARD QA ENCLOSURES FOR RFP's
(to be supplied later, after EPA approval)
-------
APPENDIX C
QUALITATIVE ONSITE SYSTEMS AUDIT CHECKLIST
This checklist gives three descriptions to each facet of a typical
quality control system. In all cases, the "5" choice indicates the most
desirable and effective mode of operation; "3" is marginal and tolerable;
"1" is definitely unacceptable and ineffective as a mode of operation.
It is not always possible to describe accurately all options with only
three choices. Therefore, a "2" or "4" rating may be selected if the evalu-
ator feels that an in-between score is more descriptive of the actual situ-
ation.
After all the applicable questions are answered, an average is computed
to give an overall indication of the quality control system effectiveness.
Generally, a rating of 3.8 or better is considered acceptable.
A rating between 2.5 and 3.8 indicates a need for improvement but no
imminent threat to current project performance.
-------
Section No. C
Revision No. 0
Date: October 20, 1980
Page 1 of 21
C.I QUALITY ORGANIZATION
SCORE
(1.1) Overall responsibility for quality assurance (or
quality control) for the organization is:
(a) Assigned to one individual by title (e.g.,
Quality Control Coordinator). 5
(b) Assigned to a specific group within the
organization. 3
(c) Not specifically assigned but left to the
discretion of the various operational, ana-
lytical, inspection, and testing personnel. 1
(1.2) The Quality Control Coordinator is located in the
organization such that:
(a) He has direct access to the top management
level for the total operation, independent
of others involved in operational activities. 5
(b) He performs as a peer with others involved in
operational activities, with access to top
management through the normal chain of command. 3
(c) His primary responsibility is in operational
activities, with quality assurance as an extra
or part-time effort. 1
(1.3) Data reports on quality are distributed by the Qual-
ity Control Coordinator to:
(a) All levels of management.* 5
(b) One level of management only. 3
(c) The quality control group only. 1
(1.4) Data Quality Reports contain:
(a) Information on operational trends, required
actions, and danger spots. 5
(b) Information on suspected data/analyses and
their causes. * 3
^Management at appropriate levels in all applicable organizations such
as subcontractors, prime contractor, EPA.
97
-------
Section No. C
Revision No. 0
Date: October 20, 1980
Page 2 of 21
C.2 THE QUALITY SYSTEM
SCORE
(c) Percent of valid data per month. 1
(2.1) The quality control system is:
(a) Formalized and documented by a set of proce-
dures which clearly describe the activities
necessary and sufficient to achieve desired
quality objectives, from procurement through
to reporting data to the EPA/RTP. 5
(b) Contained in methods procedures or is implicit
in those procedures. Experience with the mate-
rials, product, and equipment is needed for con-
tinuity of control. 3
(c) Undefined in any procedures and is left to the
current managers or supervisors to determine
as the situation dictates. 1
(2.2) Support for quality goals and results is indicated
by:
(a) A clear statement of quality objectives by the
top executive, with continuing visible evidence
of its sincerity, to all levels of the organiza-
tion. 5
(b) Periodic meetings among operations personnel
and the individual(s) responsible for quality
assurance, on quality objectives and progress
toward their achievement. 3
(c) A "one-shot" statement of the desire for prod-
uct quality by the top executive, after which
the quality assurance staff is on its own. 1
(2.3) Accountability for quality is:
(a) Clearly defined for all sections and operators/
analysts where their actions have an impact on
quality. 5
(b) Vested with the Quality Control Coordinator who
must use whatever means possible to achieve
quality goals. 3
(c) Not defined. 1
98
-------
Section No. C
Revision No. 0
Date: October 20, 1980
Page 3 of 21
C.2 THE QUALITY SYSTEM (continued)
SCORE
(2.4) The acceptance criteria for the level of quality
of the demonstration projects routine performance
are:
(a) Clearly defined in writing for all character-
istics 5
(b) Defined in writing for some characteristics
and some are dependent on experience, memory
and/or verbal communication. 3
(c) Only defined by experience and verbal com-
munication. 1
(2.5) Acceptance criteria for the level of quality of the
project's routine performance are determined by:
(a) Monitoring the performance in a structured pro-
gram of inter- and intralaboratory evaluations. 5
(b) Scientific determination of what is technically
feasible. 3
(c) Laboratory determination of what can be done
using currently available equipment, tech-
niques, and manpower. 1
(2.6) Decisions on acceptability of questionable results
are made by:
(a) A review group consisting of the chief chemist
or engineer, quality control, and others who
can render expert judgment. 5
(b) An informal assessment by quality control. 3
(c) The operator/chemist. 1
(2.7) The quality control coordinator has the authority
to:
(a) Affect the quality of analytical results by
inserting controls to assure that the methods
meet the requirements for precision, accuracy,
sensitivity, and specificity. 5
(b) Reject suspected results and stop any method
that projects high levels of discrepancies. 3
99
-------
Section No. C
Revision No. 0
Date: October 20, 1980
Page 4 of 21
C.2 THE QUALITY SYSTEM (continued)
SCORE
(c) Submit suspected results to management for a
decision on disposition. 1
C.3 IN-PROCESS QUALITY ASSURANCE
(3.1) Measurement methods are checked:
(a) During operation for conformance to operating
conditions and to specification, e.g., flow
rates, reasonableness of data, etc. 5
(b) During calibration to determine acceptability
of the results. 3
(c) Only when malfunctions are reported. 1
(3.2) The capability of the method to produce within
specification limit is:
(a) Known through method capability analysis (X-R
Charts) to be able to produce consistently
acceptable results. 5
(b) Assumed to be able to produce a reasonably
acceptable result. 3
(c) Unknown. 1
(3.3) Method determination discrepancies are:
(a) Analyzed immediately to seek out the causes
and apply corrective action. 5
(b) Checked out when time permits. 3
(c) Not detectable with present controls and pro-
cedures . 1
(3.4) The operating conditions (e.g., flow rate, range,
temperature, etc.) of the methods are:
(a) Clearly defined in writing in the method for
each significant variable. 5
(b) Controlled by supervision based on general
guidelines. 3
(c) Left up to the operator/analyst. 1
100
-------
Section No. C
Revision No. 0
Date: October 20, 1980
Page 5 of 21
C.3 IN-PROCESS QUALITY ASSURANCE (continued)
SCORE
(3.5) Auxiliary measuring, gaging, and analytical
instruments are:
(a) Maintained operative, accurate, and precise
by regular checks and calibrations against
stable standards which are traceable to the
U.S. Bureau of Standards. 5
(b) Periodically checked against a zero point or
other reference and examined for evidence of
physical damage, wear or inadequate mainte-
nance . 3
(c) Checked only when they stop working or when
excessive defects are experienced which can
be traced to inadequate instrumentation. 1
C.4 CONFIGURATION CONTROL
(4.1) Procedures for documenting, for the record, and
design change in the system are:
(a) Written down and readily accessible to those
individuals responsible for configuration con-
trol. 5
(b) Written down but not in detail. 3
(c) Not documented. 1
(4.2) Engineering schematics are:
(a) Maintained current on the system and subsystem
levels. 5
(b) Maintained current on certain subsystems only. 3
(c) Not maintained current. . 1
(4.3) All computer programs are:
(a) Documented and flow charted. 5
(b) Flow charted. 3
(c) Summarized. 1
101
-------
Section No. C
Revision No. 0
Date: October 20, 1980
Page 6 of 21
C.A CONFIGURATION CONTROL (continued)
SCORE
(4.4) Procedures for transmitting significant design
changes in hardware and/or software to the EPA
project officer are:
(a) Documented in detail sufficient for imple-
mentation. 5
(b) Documented too briefly for implementation. 3
(c) Not documented. 1
C.5 DOCUMENTATION CONTROL
(5.1) Procedures for making revisions to technical
documents are:
(a) Clearly spelled out in written form with the
line of authority indicated and available to
all involved personnel. 5
(b) Recorded but not readily available to all
personnel. 3
(c) Left to the discretion of present supervisors/
managers. 1
(5.2) In revising technical documents, the revisions are:
(a) Clearly spelled out in written form and distrib-
uted to all parties affected, on a controlled
basis which assures that the change will be
implemented and permanent. 5
(b) Communicated through memoranda to key people
who are responsible for effecting the change
through whatever method they choose. 3
(c) Communicated verbally to operating personnel
who then depend on experience to maintain
continuity of the change. 1
(5.3) Changes to technical documents pertaining to
operational activities are:
(a) Analyzed to make sure that any harmful side
effects are known and controlled prior to
revision effectivity. 5
102
-------
Section No. C
Revision No. 0
Date: October 20, 1980
Page 7 of 21
C.5 DOCUMENTATION CONTROL (continued)
SCORE
(b) Installed on a trial or gradual basis, moni-
toring the product to see if the revision
has a net beneficial effect. 3
(c) Installed immediately with action for cor-
recting side effects taken if they show up
in the final results. 1
(5.4) Revisions to technical documents are:
(a) Recorded as to date, serial number, etc. when
the revision becomes effective. 5
(b) Recorded as to the date the revision was made
on written specifications. 3
(c) Not recorded with any degree of precision. 1
(5.5) Procedures for making revisions to computer soft-
ware programs are:
(a) Clearly spelled out in written form with the
line of authority indicated. 5
(b) Not recorded but changes must be approved by
the present supervisor/manager. 3
(c) Not recorded and left to the discretion of
the programmer. 1
(5.6) In revising software program documentation, the
revisions are:
(a) Clearly spelled out in written form, with
reasons for the change and the authority
for making the change distributed to all
parties affected by the change. 5
(b) Incorporated by the programmer and communi-
cated through memoranda to key people. 3
. (c) Incorporated by the programmer at his will. 1
(5.7) Change to software program documentation are:
(a) Analyzed to make sure that any harmful side
effects are known and controlled prior to
revision effectivity. 5
103
-------
Section No. C
Revision No. 0
Date: October 20, 1980
Page 8 of 21
C.5 DOCUMENTATION CONTROL (continued)
SCORE
(b) Incorporated on a trial basis, monitoring the
results to see if the revision has a net bene-
ficial effect. 3
(c) Incorporated immediately with action for de-
tecting and correcting side effects taken as
necessary. 1
(5.8) Revisions to software program documentation are:
i
(a) Recorded as to date, program name or number,
etc., when the revision becomes effective. 5
(b) Recorded as to the date the revision was made. 3
(c) Not recorded with any degree of precision. 1
C.6 PREVENTIVE MAINTENANCE
(6.1) Preventive maintenance procedures are:
(a) Clearly defined and written for all measure-
ment systems and support equipment. 5
(b) Clearly defined and written for most of the
-measurement systems and support equipment. 3
(c) Defined and written for only a small frac-
tion of the total number of systems. 1
(6.2) Preventive maintenance activities are documented:
(a) On standard forms in readily available lab
log books. 5
(b) Operator/analyst summary in log book. 3
(c) As operator/analyst notes. 1
(6.3) Preventive maintenance procedures as written appear
adequate to insure proper equipment operation for:
(a) All measurement systems and support equipment. 5
(b) Most of the measurement systems and support
. equipment. 3
(c) Less than half of the measurement systems and
support equipment. 1
-------
Section No. C
Revision No. 0
Date: October 20, 1980
Page 9 of 21
C.6 PREVENTIVE MAINTENANCE (continued)
(6.4) A review of the preventive maintenance records
indicates that:
SCORE
(a) Preventive maintenance procedures have been
carried out on schedule and completely docu-
mented. 5
(b) The procedures were carried out on schedule
but not completely documented. 3
(c) The procedures were not carried out on sched-
ule all the time and not always documented. 1
(6.5) Preventive maintenance records (histories) are:
(a) Utilized in revising maintenance schedules,
developing an optimum parts/reagents inventory
and development of scheduled replacements to
minimize wear-out failures. 5
(b) Utilized when specific questions arise and
for estimating future work loads. 3
(c) Utilized only when unusual problems occur. 1
C.7 DATA VALIDATION PROCEDURES
(7.1) Data validation procedures are:
(a) Clearly defined in writing for all measure-
ment systems. - 5
(b) Defined in writing for some measurement
systems, some dependent on experience,
memory, and/or verbal communication. 3
(c) Only defined by experience and verbal
communication. 1
(7.2) Data validation procedures are:
(a) A coordinated combination of computerized and
manual checks applied at different levels in
the measurement process. 5
(b) Applied with a degree to completeness at no more
than two levels of the measurement process. 3
105
-------
Section No. C
Revision No. 0
Date: October 20, 1980
Page 10 of 21
C.7 DATA VALIDATION PROCEDURES (continued)
SCORE
(c) Applied at only one level of the measurement
process. 1
(7.3) Data validation criteria are documented and include:
(a) Limits on: (1) operational parameters such as
flow rated; (2) calibration data, (3) special
checks unique to each measurement; e.g., suc-
cessive values/averages; (4) statistical tests;
e.g., outliers; (5) manual checks such as hand
calculations. 5
(b) Limits on the above type checks for most of
the measurement systems. 3
(c) Limits on some of the above type checks for
only the high-priority measurements. 1
(7.4) Acceptable limits as set are reasonable and ade-
quate to insure the detection of invalid data with
a high probability for:
(a) All measurement systems. 5
(b) At least 3/4 of the measurement systems. 3
(c) No more than 1/2 of the measurement systems. 1
(7.5) Data validation activities are:
(a) Recorded on standard forms at all levels of
the measurement process. 5
(b) Recorded in the operator's/anayst's log book. 3
(c) Not recorded in any prescribed manner. 1
(7.6) Examination of data validation records indicates
that:
(a) Data validation activities have been carried
out as specified and completely documented. 5
(b) Data validation activities appear to have been
performed but not completely documented. 3
(c) Data validation activities, if performed, are
not formally documented. 1
106
-------
Section No. C
Revision No. 0
Date: October 20, 1980
C.7 DATA VALIDATION PROCEDURES (continued) Page U °f 21
SCORE
(7.7) Data validation summaries are:
(a) Prepared at each level or critical point in
the measurement process and forwarded to the
next level with applicable block of data. 5
(b) Prepared by and retained at each level. 3
(c) Not prepared at each level nor communicated
between levels. 1
(7.8) Procedures for deleting invalidated data are:
(a) Clearly defined in writing for all levels of
the measurement process, and invalid data are
automatically deleted when one of the comput-
erized validation criteria is exceeded. 5
(b) Programmed for automatic deletion when com-
puterized validation criteria are exceeded
but procedures are defined when manual checks
detect invalid data. 3
(c) Not defined for all levels of the measurement
process. 1
(7.9) Quality audits (i.e., both on-site system reviews
and/or quantitative performance audits) independ-
ent of the normal operations are:
(a) Performed on a random but regular basis to
ensure and quantify data quality. 5
(b) Performed whenever a suspicion arises that
there are areas of ineffective performance. 3
(c) Never performed. 1
C.8 PROCUREMENT AND INVENTORY PROCEDURES
(8.1) Purchasing guidelines are established and docu-
mented for:
(a) All equipment and reagents having an effect
on data quality, 5
(b) Major items of equipment and critical re-
agents . 3
107
-------
Section No. C
Revision No. 0
Date: October 20, 1980
Page 12 of 21
C.8 PROCUREMENT AND INVENTORY PROCEDURES (continued)
SCORE
(c) A very few items of equipment and reagents. 1
(8.2) Performance specifications are:
(a) Documented for all items of equipment which
have an effect on data quality. 5
(b) Documented for the most critical items only. 3
(c) Taken from the presently used items of equip-
ment. 1
(8.3) Reagents and chemicals (critical items) are:
(a) Procured from suppliers who must submit sam-
ples for test and approval prior to initial
shipment. 5
(b) Procured from suppliers who certify they can
meet all applicable specifications. 3
(c) Procured from suppliers on the basis of price
and delivery only. 1
(8.4) Acceptance testing for incoming equipment is:
(a) An .established and documented inspection pro-
cedure to determine if procurements meet the
quality assurance and acceptance requirements.
Results are documented. 5
(b) A series of undocumented performance tests
performed by the operator before using the
equipment. 3
(c) The receiving document is signed by the re-
sponsible individual indicating either accept-
ance or rejection. 1
(8.5) Reagents and chemicals are:
(a) Checked 100 percent against specification,
quantity, and for certification where re-
quired and accepted only if they conform to
all specifications. 5
(b) Spot-checked for proper quantity and for
shipping damage. 3
108
-------
Section No. C
Revision No. 0
Date: October 20, 1980
Page 13 of 21
C.8 PROCUREMENT AND INVENTORY PROCEDURES (continued)
SCORE
(c) Released to analyst by the receiving clerk
without being checked as above. 1
(8.6) Information on discrepant purchased materials is:
(a) Transmitted to the supplier with a request
for corrective action. 5
(b) Filed for future use. 3
(c) Not maintained. 1
(8.7) Discrepant purchased materials are:
(a) Submitted to a review by Quality Control and
Chief Chemist for disposition. 5
(b) Submitted to Service Section for determina-
tion on acceptability. 3
(c) Used because of scheduling requirements. 1
(3.8) Inventories are maintained on:
(a) First-in, first-out basis. 5
(b) Random selection in stockroom. 3
(c) Last-in, first-out basis. 1
(8.9) Receiving of materials is:
(a) Documented in a receiving record log, giving
a description of the material, the date of
receipt, results of acceptance test, and the
signature of the responsible individual. 5
(b) Documented in a receiving record log with
material title, receipt date, and initials
of the individual logging the material in. 3
(c) Documented by filing a signed copy of the
requisition. 1
(8.10) Inventories are:
(a) Identified as to type, age, and acceptance
status. 5
109
-------
Section No. C
Revision No. 0
Date: October 20, 1980
Page 14 of 21
C.8 PROCUREMENT AND INVENTORY PROCEDURES (continued)
SCORE
(b) Identified as to material only. 3
(c) Not identified in writing. 1
(8.11) Reagents and chemicals with limited shelf life are:
(a) Identified as to shelf life expriation data
and systematically issued from stock only if
they are still within that date. 5
(b) Issued on a first-in, first-out basis, expect-
ing that there is enough safety factor so that
the expiration data is rarely exceeded. 3
(c) Issued at random from stock. 1
C.9 PERSONNEL TRAINING PROCEDURES
(9.1) Training of new employees is accomplished by:
(a) .A programmed system of training where elements
of training, including quality standards, are
included in a training checklist. The employee's
work is immediately rechecked by supervisors for
errors or defects and the information is fed back
instantaneously for corrective action. 5
(b) On-the-job training by the supervisor who gives
an overview of quality standards. Details of
quality standards are learned as normal results
are fed back to the chemist. 3
(c) On-the-job learning with training on the rudi-
ments of the job by senior coworkers. 1
(9.2) When key personnel changes occur:
(a) Specialized knowledge and skills are retained
in the form of documented methods and descrip-
tions . 5
(b) Replacement people can acquire the knowl-
edge of their predecessors from coworkers,
supervisors, and detailed study of the
specifications and memoranda. 3
(c) Knowledge is lost and must be regained through
long experience or trial-and-error. 1
110
-------
Section No. C
Revision No. 0 ,
Date: October 20, 1980
C.9 PERSONNEL TRAINING PROCEDURES (continued) Pa96 15 of 21
SCORE
(9.3) The people who have an impact on quality, e.g.,
calibration personnel, maintenance personnel,
bench chemists, supervisors, etc., are:
(a) Trained in the reasons for and the benefits
of standards of quality and the methods by
which high quality can be achieved. 5
(b) Told about quality only when their work falls
below acceptable levels. 3
(c) Are reprimanded when quality deficiencies are
directly traceable to their work. 1
(9.4) The employee's history of training accomplishments
is maintained through:
(a) A written record maintained and periodically
reviewed by the supervisor. 5
(b) A written record maintained by the employee. 3
(c) The memory of the supervisor/employee. 1
(9.5) Employee proficiency is evaluated on a continuing
basis by:
(a) Periodic testing in some planned manner with
the results of such tests recorded. 5
(b) Testing when felt necessary by the supervisor. 3
(c) Observation of performance by the supervisor. 1
(9.6) Results of employee proficiency tests are:
(a) Used by management to establish the need for
and type of special training. 5
(b) Used by the employee for self-evaluation of
needs. 3
(c) Used mostly during salary reviews. 1
C.10 FEEDBACK AND CORRECTIVE ACTION
(10.1) A feedback and corrective action mechanism to
assure that problems are reported to those who
can correct them and that a closed loop mech-
ill
-------
Section No. C
Revision No. 0
Date: October 20, 1980
C.10 FEEDBACK AND CORRECTIVE ACTION (continued) Page 16 °f 21
anism is established to assure that appropriate
corrective actions have been taken is:
SCORE
(a) Clearly defined in writing with individuals
assigned specific areas of responsibility. 5
(b) Written in general terms with no assign-
ment of responsibilities. 3
(c) Not formalized but left to the present
supervisors/managers. 1
(10.2) Feedback and corrective action activities are:
(a) Documented on standard forms. 5
(b) Documented in the station log book. 3
(c) Documented in the operator's/analyst's
notebook. 1
(10.3) A review of corrective action records indi-
cates that:
(a) Corrective actions were systematic, timely,
and fully documented. 5
(b) Corrective actions were not always system-
atic, timely, or fully documented. 3
(c) A closed loop mechanism did not exist. . 1
(10.4) Periodic summary reports on the status of cor-
rective action are distributed by the respon-
sible individual to:
(a) All levels of management 5
(b) One level of management only. 3
(c) The group generating the report only. 1
(10.5) The reports include:
(a) A listing of major problems for the reporting
period; names of persons responsible for cor-
rective actions; criticality of problems; due
112
-------
Section No. C
Revision No. 0
Date: October 20, 1980
Page 17 of 21
C.10 FEEDBACK AND CORRECTIVE ACTION (continued)
SCORE
dates; present status, trend of quality per-
formance (i.e., response time, etc.); listing
of items still open from previous reports. 5
(b) Most of the above items. 3
(c) Present status of problems and correc-
tive actions. 1
C.ll CALIBRATION PROCEDURES
(11.1) Calibration procedures are:
(a) Clearly defined and written out in step-by-
step fashion for each measurement system
and support device. 5
(b) Defined and summarized for each system and
device. 3
(c) Defined but operational procedures devel-
oped by the individual. 1
(11.2) Calibration procedures as written are:
(a) Judged to be technically sound and consist-
ent with data quality requirements. 5
(b) Technically sound but lacking in detail. 3
(c) Technically questionable and lacking in
detail. 1
(11.3) Calibration standards are:
(a) Specified for all systems and measurement
devices with written procedures for assur-
ing, on a continuing basis, traceability
to primary standards. 5
(b) Specified for all major systems with written
procedures for assuring traceability to pri-
mary standards. 3
(c) Specified for all major systems but no pro-
cedures for assuring traceability to primary
standards. 1
113
-------
Section No. C
Revision No. 0
Date: October 20, 1980
Page 18 of 21
C.ll CALIBRATION PROCEDURES (continued)
SCORE
(11.4) Calibration standards and traceability procedures
as specified and written are:
(a) Judged to be technically sound and consistent
with data quality requirements. 5
(b) Standards are satisfactory but traceability
is not verified frequently enough. 3
(c) Standards are questionable. 1 '
(11.5) Frequency of calibration is:
(a) Established and documented for each measure-
ment system and support measurement device. 5
(b) Established and documented for each major
measurement system. 3
(c) Established and documented for only certain
measurement systems. 1
(11. 6) A review of calibration data indicates that the
frequency of calibration as implemented:
(a) Is adequate and consistent with data quality
requirements. 5
(b) Results in limits being exceeded a small frac-
tion of the time. 3
(c) Results in limits being exceeded frequently. 1
(11.7) A review of calibration history indicates that:
(a) Calibration schedules are adhered to and
results fully documented. 5
(b) Schedules are adhered to most of the time. 3
(c) Schedules are frequently not adhered to. 1
(11.8) A review of calibration history and data valida-
tion records indicates that:
(a) Data are always invalidated and deleted when
calibration criteris are exceeded. 5
114
-------
C.I3 RELIABILITY (continued)
*
(c) Not defined.
(13.2) Reliability data are:
(a) Recorded on standard forms.
(b) Recorded as operator/analyst notes.
(c) Not recorded.
i
i
(13.3) Reliability data are:
Section No. C
Revision No. 0
Date: October 20, 1980
Page 21 of 21
SCORE
1
5
3
1
(a) Utilized in revising maintenance and/or re-
placement schedules.
(b) Utilized to determine optimum parts inven-
tory.
(c) Not utilized in any organized fashion.
3
1
117
-------
APPENDIX D
STANDARD TECHNIQUES USED IN QUANTITATIVE
PERFORMANCE AUDITS
-------
TABLE D-1. SOURCE GAS + PARTICULATE
Relerenca
PaMatMl materials
S02 NBS-certifiad
S02/N2
cylindtn
S02/H2SO« Spiromtttr
t»S02)
NO NBS-certifiad
cyllnden(NO/N2)
CO NBS-tertiHed
tylinden ICO/N2)
Particulates Spiramtter
Visible Smoke generator
emissions with transmiuometer
Be
MethoMi)
EPA No. 6
In situ
moniton
Entr active
moniton
EPA No. B
EPA No. 7
In situ
moniton
Extractive
moniton
EPA No. 10
In situ
moniton
(dispersive IR)
GC
EPA No. S
EPA No. 9
EPA No. 104
Cancntratim
range Bits
3 -10.000 PDm 0
Variable 0
Variable 0
0.05 mg/m3 (1-3%)
(lower limit)
2-400 mg/m3 0
(as N02I
Variable 0
Variable 0
20 - 1,000 ppm +7 ppm
Variable
0-6%
0-100% 41.4%
opacity
-20%^ tutrwgs
QA
PrtcWoa teaman! Ariit Tach«h)n
3.9 (CV) EPA-650/4-74-005-a Analysis: STO H2S04 hi 3% H202
Total procedure: SOj/N2 tylinden
10-15ICV) EPA-650/4-74-005-0 S02/N2 cylindan
10-15 (CV) EPA-650/4-74-OOS-0 S02/N2 cylinden
60 (CV) EPA-650/4-74-005-1 Calibration checks ol supportive
equipment - dry gas mater, oiiltee
meter, leek check.
7-BICV) EPA-650-/4-74-005-I Analysis: STO solution! of N§N02 in
absorbing solution
10-15 (CV) STO cylinden of NO/N2 and N02Mr
10-15 (CV) STO cytlnden of NO/N2 and N03/air
13ppm(CV) EPA-6SO/14-74-005-h STO cylinders of CO/N2 or CO/air
10-30ICVI EPA-650/14-74-005-d Calibration checks of supportrva
equipment - dry gas meter, otllice
meter, leak check.
2% opacity EPA-6SO/14-74-005-I Sida-by-slde determinatiom -D ,_, -^ cr>
Ol Q) fD (l>
44 (CV) EPA-650/14-74-005-k Synthetic solution! '(D ST ^- c«-
' V> '
l-« - 0
O "3
(Conlinood) O O 3
f* Z 0
r-J O O
ro cr
ID ^3
-J 0
r\J
CD
10
oo
o
-------
TABLE D-1 (continued)
FaRitam
C| Cy n~
el hams
(Lenll)
Inorganic
sulfur spachn
(Level 1)
CS2. COS.
Hj^.ete.
Total
hydrocarbons
SASS train
SASS train
organic!
Particle
morphology
RafareMa
Materials
NBS-SRM for CH3
NBS-SRM for propane
Bag STD's prepared
from pure component
Bag STO't praparad
from Injection
of pure component
NBS-SRM CHj/N2
NBS-SRM C3Hg/N2
NBS-Fly-Ash
NBS-Coal
--
Round -robin
analysis by
various
laboratories
CoMratratlon
Methadls) range Bias rncMra
GC-FID 0- SO ppm
GC - FPO 0-25 ppm
GC-FIO D- 21.000 ppm
SSMS
AAS
Wet methods
GC-FID
IR
1C
Grav
LRMS
GC-MS
Microscopy
polarizing
light
Decwimrt Aadlt Tecki^w
Cylindan containing C | - C4
components at various
concentrations
Cylindan containing sulfur
species - compatibility of
various components in same cylinder
must be verified.
Cylinders containing CH3 -
C3HB/air or N2.
Synthetic standard containing
inorganics of interest and/or
split samples.
Loaded XAD-2 resins with desired
organic* or split samples.
Samples containing asbestos.
fly ash. etc.
-V O 30 «/>
0) tu rt> (V
(Q c* < O
re o> <-»
..(/)_!.
»v> - o
o 3
O O 3
-h n &.
r* Z O
t_i o O
i\> cr
(D O
-1 O
00
-------
TABU; 0-2. AMBIENT AIR
PeftitMit
SO}
NO,
NO,N02.NO,
Photochemical
oxidanu -
ozone (03)
CO
Total Suspend-
ed Pertkulates
"2s
CH3SH (methyl
mercantan)
C2H3CH (ethyl
mercaptan)
Ralerence
matarieh
NBS permeation
tuba lor dynamic
calibration, or a chemi-
cal primary standard
(italic)
NBS permeation
tube or sodium
nitrite (primary
standard)
NBS cylinder (NO)
NBS permeation
tuba
UV photometry
NBS reference tank
(CO/N2)
Roots meter
Permeation tube
Permeation tubes
Mathadls)
West-Geeke
Arsenite
bubbler
Oione chemi-
luminescence
Ethylene cheml-
luminescence
NDID
Hijn volume
Tentative
method
Tentative
method (total
mercaptans)
Centra trttlen
range Bias'
2Sto1.050jjg/m3 +3.8%
(0.01 to 0.4 ppm)
20to7SOpg/m3 -3%
(24 h)
(0.01 to 0.4 ppm)
0 to 0.6 ppm
0 to 0.5 ppm -3510-15%
(0.05 to
0.50 ppm)
OtoSBmg/m3 42.5%
(0 to 50.0 ppm)
1.0fie/m3et1.7
m3/min/24 h
(minimal detectable)
1.1 to 100/io/m3 -20%
3.9pg/m3to200
liglm3 (CHjSH)
(2.0ppbto200ppb)
OA decamsnt or
0.037 to 0.013 EPA-R4.73-026d
/jg/mL
(2.26 to 0.42
/ig/mL)
8(jg/m3 EPA-650/4-74-048
(50to300fig/m3)
EPA-600/4-75-003
0.0033 + 0.0255 EPA-R4-73-028C
10,1
(0-0.5 ppm)
0.6fig/m3 EPA-R4-73-028a
3 (CV) EPA-R4-73-02Bb
+i.S 2
CV's Irom methyl 2
to henyl mtreap-
tan range Irom
1 .0 to 2.6
See notes at end of table.
AudHtediintM
EMSL Freen-dried sullite TCM
samples (analytical audit onry)
EMSL N02~ samptei
(analytical only)
No cylindar with dilution lor NO
Gnphate titration ol NO and 0,
waits ts provide dynamic
concentrations ol NO, end NO,
UV 03 generator with output
monitored with a UV photometer
Separate cylinders ol various
concentrations or dilution ol
stock cylinder ol approximately
50 ppm
Reference Flow Device
(EMSL)
Type S weights
Cylinders ol H2S in nitrogen
at various concentrations
Separate cylinders ol the
mercaptan ol interest
-
(Continued)
~o o yo O> (D O>
(Q c* < O
(D (B -* c*
CO ->. O
0 =>
0 O 3
-h 0 Z
c* Z O
|_i o O
l\> O"
o> o
-------
TAB -E D-2 (continued)
PafleJIailt
S04
(pirtiniliti)
NH3
Referent*
materials
Synthetic standard
band on direct
weighing of Na2S04
Onviimtriunv
Cilibratgd
Pfnneition tube
Matkodli)
High volume -
methyl thymol
blu» Uutomatnll
Indophenol
method
Conctntratira
range Bin*
0.3 to 45 Ml
S04/m3
20 to 700 pg/m3
(0.02510 1.0 ppm)
OAlKMOTtar
Precision* Reference mm**
0.7% 3
(0 to 10 ppm)
30 to 5%
(0.7 - 700
uo/m3|
AatitttriraNM
SOj -impregnated frlter
ttripi (EMSU
NHj -impregnated inter
strips (EMSL)
HCI
(hvdrocNwk)
Fluorides
Hydrocarbons
Mttili: B.Bt.
C«. Cd, Co,
Cr. Cu, Ft.
Mn, Hg. Pb,
Ni. and Zn
Synthttic standard
band on primary
ttandard sodium
fluoride (NeF)
Propant in air
(NBStank),
Damnation tubai
(NBS will product
mathant SRM'i
It-10 ppm) ki
tha mar futun)
NBS ralannca maler-
riali lor haavy rnatali,
NBS fly ash and syn-
thetic standards basad
on primary standards
Marcuric Ihio-
cyanata method
IASTM)
Smilautomatad
method
(I2202-02-68T)
Reference method
lor determination
ol hydrocarbons
corrected lor
methane
Gn chromatog-
raphy with llama
ioniration detec-
tion and concen-
tration step
Atomic
absorption
0.1to4.0figF/mL
Oto13.1pg/m3
(0 to 20 ppm)
carbon (as 0(4).
For methane.
0 to 6.S5 fig/m3
(0 to 10 ppm)
1to10ppb(V/V)
10 to tOOppb(V/V)
10 (CV)
120 to 100
PpmF)
30 (CV)
±5X (range) S(CV)
±5* (range) 3ICV)
10 to +20% 10 (CV)
Cylinder dilution ol stock
HCI/nitrogen
Synthetic standard ol NaF
In water
Separate cylinders ol CH4 - 03X4
at various concentrations
Synthetic standards contamlnj
tha alemenli ol inleiesl in
designated matrix
When a range ol bits or precision is given, tha comtponding concentration range ii ghnm Immediately below the precision values.
1. Fedtnl Ptgitur. Vol. 36. No. 84. April 30.1971. p. 8168.
2. Method of Air Stmpling tnd Aiufyat, Intersociety Committee, American Public Health Association, 1972.
3. Harvey P. Stem. Minunmmt ofSullitt, NTIS No. LBL-2162.
4. E.F. Peduto et al.. Guidtliim for fmironmenlil Asaament Dan Quality Prognms. Rettarch Triangle InstitU'.j, EPA Contract
No. 68-02-2156. April 1978.
-o o ?o «y»
Of Ol ID (D
CO c* < O
m n> -> r+
O O 3
-«i O -SC.
ri- Z O
|_i o O
ro cr
rt> O
-t o
ro
o
oo
o
-------
TABU D-3. WATER1
Peremeter er
pofttiUBt
Acidity
Alkalinity
Biochemical
o«yg>n demand
iBOOi
Calcium
Chemical
oxygen demand
(COD)
Chloride
Raftranca
materials
Standard
salullonli)
Standard
lolution(i)
Standard
solutionh)
~
Standard
olutiondl
Mattodh)
pH miter,
alectromatrlc
tilration
pH mater.
eleclrom etiie
titration
Automated
Sday.20°C
Titrimetrlc
Normal
Low level
Hiyh level
(lor saline waters)
Titrimetrlc
Automated
CetttUNtl UOfV
rente Bun'
10- 1.000 mg/L.
ifCiC03
-8mg
CiC03/l
(average)
10-IOOmg/l <-!%
as CaC03
O.Smg/L. +1.9%
aiCaC03
(lower limit)
15- 2.000 mj/L -4.7%
. 5 -SO mg/L +0.3%
>2SOmg/L
«0.5 to - 5 mg/t
(15 -400 mg/L)
1-250 mg/L
PetciSMat" A it mfltttftM
10 mg, CaC03/L Standard
solntiom
3.2mg,CeC03/L EMSL Mineral Stendard
0.5 (CV)
35 - IS (CV) EMSL Demand Standard
(2-mmg/lBOO)
9.2 (CV) EMSL Mineral Standard
17.8 mg/L COD EMSL Demand Standard
4.2 mg/L COO
S.4 mg/L EMSL Mineral Standard
(average)
(IS-400mg/LI
0.3 (CV)
Brial detcripliom ol rath lifted mithod ara giwn hi Mtthodi lor Chimial Aiufysii of Wtut imt Watts, EPA-625/6-74-003. Primary relarenca art also cilad. (Continued)
'Whan a range ol bin or precision b liven, tht cormponding concantration range U given immediately below the precision vahies.
-0 0 30 t/>
oi tu to at
io c+ < n
(D (D ->(-*
(/>_!.
CJ1 - O
O 3
0 O 3
-*> 0 Z
<-» z o
M O O
is» cr
(D 0
-I 0
ISS
o
»-
vo
00
o
-------
TABLE d-3 (continued)
Parameter ir
eriNUM
Nitrogen (con.)
N (trail
Nitrite
(Hand
greate
Organic
carbon
pH
PfwnoifCS
Photphorut
Residue,
totil
filterable
RvlvfVMV
meteriab
Standard
tolution(i)
Standard
solutlonh)
~
Standard
tolution(t)
Slindard
solution It)
Stmdird
tolutlond)
"
Metk«d(t)
Automated
phenate
Bmtint
Colorimetrk
So.hlet
extraction
Infrared
Oxidithn
Elntromftrk
H-AAP
with
dittitlitlon
AutomiNd
Slngls
reigmt
Automated
eolotimetric
Grmimetrit
Cmcmtrttloii
rmff BlnT
0.05- 2.0 mf/L -25%
0.1 - 2 mg/L.
uN
0.01 -1.0 mg/L.
II N
5- 1.000 mg/l -12% (tingle
e>tract*blei determination)
0.2- 1,000 mf/L 0 (tingle
entraciabiei OTtorminiilon!
>lmg/l *15-1%
IS -110 mg/L)
<1%
5- 1.000 mg/L
2-SOO|ig/l -(2-31%
0.01 - 0.5 mg/L
0.001 -1.0 mg/L -12%
10 -20.000 mg/L
Pnchlra'
29%
60-20%
as N
10.2- 1.2 mg/1)
8% (tingle
umple)
10% (tingle
uniplet
SO- 7%
(S-110mf/l)
0.1- 0.2 pH
unlit
3-4 mg/L.
extraction,
(0.5 -1.5 mf/L.
dlrvct pholomctrlci
13-1%
(4-450»if/l)
B%
48-22%
(0.04- 0.30 mg/D
"
AiffliMteiM
EMSL Nutfienu Sltnderd
EMSlNuttwntiSttndtrd
Aqutoin tohitiora ol NeNOj
No.2futloaendWe>ionoil
hiweter
EMSL Demand Standard
EMSL Mineuli Standard
Phtnol ttandanh in water
prctarve at pH <4 with
H3P04 and add 1 g/L
copper tullate
EMSL Nutrienti Standard
EMSLMinenli Standard
(Continued)
O O 70 tS>
01 o> m n
to <-» < n
o» -" o
o ^
003
-h O Z
«- Z O
»_l o O
-J O
ro
o
UD
oo
o
-------
TABIED-3 (continued)
Parameter at
poHitnt
Color
Conductivity
Dissolved
oxygen
Herdneo,
total
Metals*
Nitrogen
Reference
mrtortah
Standard
solutionh)
Standard
sduiion(s)
Standard
solutionls)
Standard
solutionk)
Standard
solution)!)
Methcdh) range
Platinum -cobalt
Spectrophoto-
metrk
Wheatstona
bridge or
Modified
WinMer
Electrode 0-20 mg/L
Titrlmetric
Automated 10 - 400 mg/L.
Atomic
ebsorption
speclroscopy
Distillation 0.05 - 1.400 mg/L
procedure
Selective Ion 0.03- 1.400 mg/L
electrode
Manual 0.05 - 1.400 mg/L
Kjeldahl
Bias* Preehtan*
__
-11-51% 6-8%
(100- 1.700 fimho/cm)
_
1%
-(1-31% 3 -10 mg/L
at CaC03 at CaCOj
(30 -450 mg/L) (30 -450 mg/l)
-9% 1-5%
__
-0.02 mg/L 60-15%
(0.2 -2 mg/L)
-7% 2-3(CVI
(0.1-0.2mg/LI
«16-(-2)% 100-25%
(0.2 -4.6 mg/L) (0.2 - 4.6 mg/l)
Audit material
Synthetic standard
EMSL Mineral Standard
Ontite WinMer of split
sample
EMSL Mineral Standard
EMSL Trace Metah
Standard
(AL, AS. Be. Cd. CO, Cr,
Cu. Fe. HH.MN, Ni.Pb.
Se.V.Zn)
EMSL Mineral Standard
(Ca. Mg, Na, K>
EMSL Nutrients Standard
' Twenty-til metals an cataloged, with optimum concentration ranges, tensitivitv, detection limit, and precision and accuracy data.
(Continued!
-O O 3D CO
o> n» n> n>
«Q c»- < n
n> (D -><-
«. (n w.
->i - o
O 3
O O 3
-*, o -z.
<+-z.o
MOO-
-J O
l\>
CD
00
o
-------
TABLE D-3 (continued)
Perameter w
prflltOTt
Rnidut, lotil
nontilttreble
Sullilt
Soiihfc
Sulfili
Threshold
odor
Turbidity
Rettrtmt
nitirWi
Sttnderd
tolutionlt)
Slesisi
tohitionh)
Stmdtrd
lolutiond)
Odor-tree
mm
Stindvd
reference
suspension
Mitkcdd)
Gravinratric
Torbidinwtrlc
Automited
chlonnftitl
TiSrimstrte
iodine
Tlltinulrte
Consilient
mm
Nephrfometrk
ConcMtriti»
nnft Ste1 f redtlMt AidH nuttiM
10 - 20,000 mg/L Split ample
-(0-3)% 6K EMSL Minenli Slmdtrd
iO-400mf/l 0 1%
> 1 nu/l Sodhim thlnuHitt ampht
> 3 m|/L Sodium thlotulleti ampin
__ . . __ __ __ __ __
O-IOnephelo- 2-3% Formiiin Turbidity Stimtanl
metric turbidity
unit
-v CD -yo tst
o» Q* n> n>
(O C* < O
ID (T> -><-*
(/) _J.
oo o
O 3
00=]
-h O Z
r* :z o
|_i o O
rv> cr
CJ
-s o
ro
o
oo
o
-------
TABLE D-4. SOLIDS
Parameter
Antitnony
Arsenic
Cadmium
Chi ui ilium
Lead
Total mercury
Organomeicury
compounds
Nickel
Reference
materials
Synthetic standard based
on potassium tertrate
K(SbOlC4H404
Synthetic standard based
on arsenic trionide
Synthetic standard based
on stock cadmium solution
Synthetic standard based
on stock chromium solution
Synthetic standard based
on Pb(N03)2
Synthetic standard
based on HgCLj
Synthetic standard
based on selected
anthyl and phenyl
mercuric chlorides
Synthetic standard
based on (NH4I2
S04 - NiS04 - CH20
Metkodh)
1) RhodamineB
method
2) Atomic
absorption
1) Ag-DEOTC
2) Atomic
absorption
Atomic
absorption
Atomic
absorption
1) Dilhizom
method
2) Atomic
absorption
Flameleu
(cold vapor)
atomic
absorption
Ges
chromatography
1) Dimethyl-
gtyoxime
method
2) Atomic
abotption
SeflcewentratW OA
matkod range Bias* PnctshM* retemnat
n>
«Q <+ < n
(D fl> -^ c*1
vo -» o
O 3
o o 3
-h 0 Z
c+ Z 0
(_i O O
. fO O"
n> o
-I O
ro
0
vo
00
o
-------
TAB1 E D-4 (continued!
Parameter
Selenium
Cyanide
Fluorids
Phenols
Sullida
SulletB
Sullite
Retinae*
materials
Synthetic itindird
brad on Ni2Se04
10H;0
Syntlwtlt standard
based on KCN
Synthetic Mmdird
bated on NiF
Synthetit standard
based on phenol
Synthetic itmdird
bind on sodium
sullate tiihydriu
Stindird based on
lulluric Kid
Synthetic standard
. bind on sodium nil lie
Method(s)
1) Diemino-
benitdine
2) Gaseous
hydiide
(atomic
absorption)
. Pyridine
pyra»lin>
method
Distillation
* letectin
Ion electrode
1) ADA
dimethyl-
iniline
2) Gnchromi-
togiiphy
Titrimetrit
(iodine method)
1) Gravimetric
(BaCL2
2H2Oppt)
21 Ion exchange
chromitography
1) Titration
2) lonenchange
chromstog-
raphy
Sri cent innate*/ OA
method rmae Bin* Pr»dil»»* lehinKtt
O-l-SjiJT1/ 1» «5% 112KCVI 1,3
2) 0 2) 10 (CV)
1) 0 to 20 jig
2)2-20ppb
Method range -IS* 17 (CV) 1.2.3
1-SfigCN
10-250/Jg|-t/ «4.9H 3(GV) 2.3
0 to 10 ppm F
Method tanje 2.3
1) 0 to 500 /jg phenol
2) > 1 ppm phenol
Method ran§e 1.2,3
>1ppm
Method range 1) 1.9% 11 S(CV) 1) 1.2.3
11 >10ppm 2) 2! 2) 4
2) > 1 ppm
Method range 1) 1,2.3
1) lo 2 ppm
2) to 1 ppm
*«! material
Partkulata or coat ample
spiked with trace metals
ol concern
Synthetic standard ol KCN
In matrix ol interest
Synthetic standard ol NaF
m n-.itil; of interest
Synthetic phenol standard
in matrh of Interest
Synthetic standard ol
sodium thiosullan in
matrix ol interest
Synthetic standard ol
standard sulluric acid
hi matrix ol interest
Synthetic standard based
on sodium Ihiosullata
(ContiiHMd)
-O O 30 W»
0) Qi n> n>
*
r* ^i O
O O 3
03
on z
-h c* "Z. O
Of\
u
t-i cr .
ro o> o
-J 0
ro
CD
i-«
U)
00
CD
-------
TACLEO-4 (continued)
Parameter
Oetergentt
(eniwiic
turleclMti)
Reference
meteileh
loiter ilkylete
tulfonate (LAS)
tveiliMe from EPA.
Cincinnati. Ohio
Method),)
Colorimetrlc
procedure bend
on initial comphn
of phenanthraline.
copper, and LAS
SeilccncMtretien/ QA
mt'hOfl fMlft DIM rVTCniM fVlfllMn
Method range 23
1) 0 to SO Ml LAS
A«dH materiel
EMSL surfactant ttendard
from EPA (Cincinnati)
REFERENCES
t. Method} for ClumiatAntlvm of Witir mil Wuut, EPA-625/6- 74-003.
t SundvdHithoiltolWmrtndWtttniater, 13th «d., APHA. AWWA, WRCF. 1971.
3. S. E. Altai. H. W. Gilmihtw, J. Pirktnson, ind C. Ouirmby, ChimM Ant/ytitof ecological Mtterialt. John Wiley md Sons. 1974.
4. H. SnuH, T. Stmnt. wd W. C. Dinmer. And. Ctiem.. Vol 47, No. 11 (1975). p. 1801.
-O O 30 t/>
O) D> O> »
n> -><-*
I-J O 3
o =j
on z
-h r* Z O
o o
i_> o-
rv> CD o
ro
o
(JO
CD
O
-------
TABLED-'... PROCESS PARAMETERS
Pwwiwtef
Flow (gas)
Flow (liquid)
Pressure
Temperature
Operating
principle
Displacement
Pilot tube
Otilin
Rottmeter
Vtnturi tub*
Displacement
Pilot tube
Orilia
Rotimiter
Vtnturi tubt
Diaphragm
Bellow
Bourdon
Mtnomtttr
Strain gaga
Electronic
Bimetallic
Liquid in |!HI
Filled system
Oplitil
Radiation
Resistance
Thirmistor
Thermocouple
Rengaef
ptrsthm
(sclml
0-t.5x1054
100 - > 10s #
O.OOS->10SI|
0- l.3« 103
1 - > 105 #
(gpm)
0.0t - 2 x 104
50 - > 10s #
0.005 -> I05I1
0.01-4. 103
1 - > 10* #
(prig)
0.01 - 10*
0.01 - 2 » 10*
IS - 15 > 10s
0.01-60
0.01 - 2 > 10s
0.01-1.2x10*
ro
-130-430
-165-620
-240-540
700 - 3.000
315-4700
-270 - 650
-7S-260
-165 - 2.7.00
Accuracy
iX foil scale*
1'
5
2
10
3
2
5
2
10
3
1
1
S
1
1
3
2
(0.01- 28° C)
2
2
1
0.1
0.1
1
RtpeitabHItv
i% fill scan*
0.05
0.05
0.25
0.059
0.05
0.2S
o.t
0.2
0.5
0.1
0.5
O.S
0.005
0.01
0.1
AetfttechatlOT
Calibration chick lor ifl with similar
proem measuring dtvicti
Calibration check lor ell with similar
proems measuring devicei
MibrstlsB check ler sfl with H-r-Ms
proems niusufin| omcvi
Calibration check lor aO with similar
process measuring devices
Accuracy means the limits within which tha stated value ol a process property might vary relative to its true vriue. Expressed in percent hill stele. Accuracy
includes errors due to the combined conformity, hysteresis, and repeatability.
* Repeatability is the ability of a transducer to reproduce output readings when the same measured value h applied to it consecutively, under identical condi-
tions, and in the same direction. Expressed as being wilhin a certain percent of full-scale output.
'All values for range of operation and accuracy are drawn from ISA Trmsduar Compendium. 2nd edition. Instrument Society of America, Pittsburgh;
parti. 1969. and part 3,1972.
9 All values for repeatability are drawn from Instrument fngineen Handbook, volume 1, Process Heaturtrmntt, 1st edition. Belt C. liptek, ChBton Book
Company, Philadelphia. 1969.
"Maximum flow limited by pipe size only.
"Limited by readout device only.
TJ O 3O «/»
Ol Ol (O (D
(Q <« < O
rT> (D -'(-*
in '
1-1 -J. o
r>o o 3
O 3
on -x.
-» c* z o
o o
i_j o-
rv> rt> o
-j o
rs>
o
u>
oo
o
-------
APPENDIX E
DEFINITIONS AND STATISTICAL TECHNIQUES USEFUL
IN QUALITY ASSURANCE.PROGRAMS
-------
Section No. E
Revision No. 0 .
Date: October 20, 1980
Page 1 of 7
I. CENTRAL TENDENCY AND DISPERSION
A. The Arithmetic Mean.
The sum of all values in a measurement set, divided by the number of
values summed. Commonly called the "average." Often denoted symbolically
by a bar over the variable symbol, as "X". '
_ n
X = Z X.
B. Range.
The difference between the maximum and minimum values of a set of
values.
R = X - X .
max mm
A rough indication of variability, particularly when the set of values is
small (<10).
C. Standard Deviation.
An indication of the dispersion of a set of numbers about the mean
value. Normal (and other) distributions are expressed as a function of
the standard deviation.
For a given set of values, the defining equation is:
1/2
s -
n
Z (X, - X)2/(n-l)
Li=l ^
For computational purposes, it is convenient to use:
1/2
or
s =
2-2
Z X/ - X *
n-1
s =
Z X.2 - ( Z X./n
i=l 1 V1=1 1'
n-1
1/2
-------
Section No. E
Revision No. 0 ,
Date: October 20, 1980
Page 2 of 7
D. Relative Standard Deviation, or Coefficient of Variation*
The dispersion of a set of values, expressed as a percentage of
the mean.
CV - (s/JC) x 100
II. MEASURES OF VARIABILITY
A. Accuracy.
The difference (either on an absolute or percentage basis) between
a measured value and an assumed "true" value. The larger the difference,
the lower the accuracy.
7B
B « X - T, or
(X-T) x 100
(see "Bias")
B. Bias.
A nonrandom* measurement error; a consistent difference either
between sets of results or between a measured value and a "true"
value. If the latter, the bias or percent bias is measured by the
relationships in A above. (See III. SIGNIFICANCE TESTS, A. t-test)
C. Precision.
A measure of agreement among individual measurements of a vari-
able, under identical or similar conditions. Precision may be ex-
pressed in several ways, and care must be exercised in the defini-
tion and use of precision measures.
One set of such measures follows:
I. Within-laboratory: The within-laboratory standard deviation
(often referred to as repeatability), s, measures the disper-
sion in replicate single determinations made by one laboratory
"'"These definitions are taken from EPA collaborative test result publica-
tions, and are applied to the various federal reference sampling and
analysis techniques. Since these techniques are frequently used by
IERL-RTP in evaluating emissions, they are particularly appropriate for
this guidelines document.
-------
Section No. E
Revision No. 0
Date: October 20, 1980
Page 3 of 7
team (same field operators, laboratory analyst, and equipment)
sampling the same true concentration
2. Between-laboratory: The between-laboratory standard devia-
tion, s, , measures the total variability in a concentration
determination due to determinations by different labora-
tories sampling the same true stack concentration. The
between laboratory variance (often referred to as repro-
2
ducibility), s, , may be expressed as
2 2^2
8, - 8- + S
0 L
and consists of a within-laboratory variance plus a labora-
2
tory bias variance, s .
L
3. Laboratory bias; The laboratory bias standard deviation,
s_ = /sfj - s^ , is that portion of the total variability
that can be ascribed to differences in the field operators,
analysts and instrumentation, and due to different manners
of performance of procedural details left unspecified in
a technique. This term measures that part of the total
variability in a determination which results from the use
of a technique by different laboratories, as well as from
modifications in usage by a single laboratory over a period
of time. The laboratory bias standard deviation is esti-
mated from the within-and between-laboratory estimates pre-
viously obtained.
A corresponding set of coefficients of variation would be CV,
CV , and CV . These are convenient to use if the precision is pro-
portional to the mean value of the variable.
-------
Section No. E
Revision No. 0
Date: October 20, 1980
Page 4 of 7
III. SIGNIFICANCE TESTS
A. t-test.
If one has an assumed "true" value, however obtained, the
existence of a significant bias in other measurements of this value
can be defined by a t-test:
d - 0
sd//n~~
where t » a parameter, the magnitude of which is referenced to
tabulated values. A t-value which exceeds the tabulated
value for given specifications of probability and number
of degrees of freedom indicates the existence (within the
definition of probability specified) of a significant
bias. The more stringent the probability requirement;
i.e., the smaller the probability chosen, the larger the
tabulated t-value.
d - the average difference between the true value and the
measured values; the average bias.
s. - the standard deviation of the differences, d .
n = the number of differences used to calculate d.
B. Chi-square test.
If one has a reasonable estimate of the expected standard devia-
tion of a set of measurements, the existence of a defined "excess
variability" can be tested as follows:
2 2
sd
f O2{x>
-------
Section No. E
Revision No. 0
Date: October 20, 1980
Page 5 of 7
2
where x /f " a random variable with tabulated values ( f - n - 1 =
number of degrees of freedom).
2
a {x} - the expected variance of the measurements of x.
o
If x /f is larger than the chosen tabulated value (with specified
probability), it is concluded that the measurements are exhibiting
excess variability. The chi-square test is a measure of the validity
of a series of measurements based on an "expected" variability. The
test is worthwhile only whenever a measurement technique has been
tested thoroughly, so that a realistic expectation can be estimated.
IV. CONFIDENCE LIMITS
Confidence limits take two forms. One form defines a numerical range
within which one has a (arbitrarily chosen) probability of finding the true
mean value of the measured variable. If the measurement variability is ex-
pressed as a standard deviation, the confidence limits as defined above can
be calculated as follows:
CL = X + ts/n
where all symbols have been previously defined. Note that as the number of
measurements, n, increases, the magnitude of CL decreases. Also, for higher
probabilities of containing the true mean within CL, the larger the value of
t and therefore the larger the size of CL.
The second form of confidence limit defines an interval within which the
next individual measurement can be expected to fall with a given probability.
The calculation of this limit, sometimes'called a tolerance limit, is by
the following relationship:
TL = X + ts
-------
Section No. E
Revision No. 0
Date: October 20, 1980
Page 6 of 7
While n, the number of faeasurements, does not explicitly appear in the
equation for TL, it does determine (along with the selected probability)
the value of t; i.e., as n increases, t decreases.
V. TESTING FOR OUTLIERS
An outlier is an extreme value, either high or low, which has question-
able validity as a member of the measurement set with which it is associ-
ated.
Detection of outliers may be on one of the following basis:
a) A known experimental aberration, such as an instrument failure or
a technique inconsistency.
b) A statistical test for significance, such as the Dixon ratio test.
This test is described below.
The Dixon criteria is based entirely on ratios of differences be-
tween observations where it is desirable to avoid calculation of s or
where quick judgment is called for. For the Dixon test, the sample cri-
terion or statistic changes with sample size. Critical values of the
statistic for various levels of significance are tabulated.
*
Table 1 below, presents selected significance (probability) levels
for criteria over the n range 3 to 25. Note that the measurement values
are fii
value.
are first, arranged in order of ascending magnitude; i.e., x is the largest
Taken from "Processing Data for Outliers," by W. J. Dixon, Biometrics,
Vol. 9, No. 1, 1953.
-------
Section No. E
Revision No. 0
Date: October 20, 1980
Page 7 of 7
Table 1. DIXON CRITERIA FOR TESTING OF EXTREME OBSERVATION
(SINGLE SAMPLE)3
n
3
4
5
6
7
8
j.
10
n
12
13
14
15
16
17
18
19
20-
21
22
23
24
25
Criterion
HO = (x2 - x-|)/(xn - XT) if smallest
value is suspected;
= (Y - Y }/(* - x ^ if
un xn-lj/un xlj 1T
largest value is suspected.
r = (ti - Y W^Y - Y \ if
11 V^O 1 ' ' » n 1 1'
smallest value is suspected;
= (xn * xn-l)/(xn - X2) 1f
largest value is suspected.
r21 = (x3 - x])/(xn_1 - X]) it-
smallest value is suspected;
= (xn- xn.2)/(xn - x2) if
largest value is suspected.
r^o ~" \ Xo Xi / / \ *»__ o ^ X^y IT
C.C. i \ l\-C \
smallest value is suspected;
- (xn - xn.2)/(xn - x3) if
largest value is suspected.
Significance Level
10%
0.886
0.679
0.557
0.482
0.434
0.479
0.441
0.409
0.517
0.490
0.467
0.492
0.472
0.454
0.438
0.424
0.412
0.401
0.391
0.382
0.374
0.367
0.360
5%
0.941
0.765
0.642
0.560
0.5Q7
0.554
0.512
0.477
0.576
0.546 .
0.521
0.546
0.525
0.507
0.490
0.475
0.462
0.450
0.440
0.430
0.421
0.413
0.406
1%
0.988
0.889
0.780
0.698
0.637
0.683
0.635
0.597
0.679
0.642
0.615
0.641
0.616
0.595
0.577
0.561
0.547
0.535
0.524
0.514
0.505
0.497
0.489
-------
APPENDIX F
SOME STANDARD AMBIENT AIR AND SOURCE SAMPLING TECHNIQUES
-------
SOURCE SAMPLING TECHNIQUES
Pollutant
so2
so2
and
S03/H2S04
N0x
CO
Partlculates
Visible
emissions
Be
EPA
Method
or
Number
6
8
7
10
5
9
104
Bias (absolute,
or percent of
mean concentra-
tion)
0
-2% (analysis
only)
-2% (analysis
only)
0
+7 ppm
No Information
+1.435 opacity
-20%, average
Precision (standard deviation,
or coefficient of variation)
within
. laboratory
3.9 (CV)
0.1 g/m3
60%
7%
13 ppm
10-30%
255 opacity
44%
between
laboratory
5.5 (CV)
0.11 g/m3
65%
10%
25 ppm
20-40%
2.5%
58%
Comments
Major error source Is dif-
ficulty of obtaining repro-
ducible tltratlon end-
points. Minimum detect-
able limit 1s 3 ppm.
Same analysis technique
as Method 6 above.
Grab sample; largest error
source Is failure to re-
calibrate spectrophoto-
meter.
Analyzer drift and CO.
Interference are largest
problems. Minimum detect-
able limit is 20 ppm.
Numerous small error
sources associated with
stack sampling.
Good results depend to a
great extent on the effec-
tive training of observers.
Reference
. EPA-650/
14-74-
005-e
EPA-650/
14-74-
005-g
EPA-650/
14-74-
005- f
EPA-650/
14-74-
005-h
EPA-650/
14-74-
005-d
EPA-650/
14-74-
005-1
EPA-650/
14-74-
005-k
a This table is a summary of information contained in the cited references, all of which are Quality Assurance Guide-
line Manuals.
-o o yo v*
o> 01 n n
«Q c+ < n
n> n> - c+
in ->.
M - o
o 3
o o 3
-h o -x.
r+ z o
co o o
IT
m -n
-j o
ro
o
«£>
CO
O
-------
AMBIENT AIR TECHNIQUES3
Pollutant
EPA Bias (absolute, Precision (standard deviation,
Method or percent
of or coefficient of variation)
or mean concentra- t.n.k4«
"«*«p ««"> laCalry
SO.
Z
NO, NO^
NO
HUx
Photochem-
ical ox1-
dants
CO
a This table
0
Chemi luml- 0
nescent
Chemlluml- -35 to -15%
5-13 wg/m3,
from
x = 0-1000
pg/m3
7-8% at ,
100 vg/m
(0.05 ppm)
c c
0.0033 +
nescent from 0.05 to 0.0255 x
0.50 ppm
NOIR +2.5
(0-0.5 ppm)
0.6 mg/nr
is a summary of information contained In the cited
Guideline Manuals published by EPA.
between
laboratory
10-25 vg/m
from x a ,
0-1000 vg/m
0.0008 +
0.0355 x
(0-0.15 ppm)
-0.0051 +
0.0690 x
(0.15-0.5 ppm)
0.3 - 1.6
mg/m3 (non-
linear varia-
tion) over
0-60 mg/m3
references, all
Collaborative test results are cited, If
Comments
Lower limit of detec-
tion Is 25 vg/m . Flow
rate changes, sampling
train leakage are prim-
ary error sources.
Lower limit of detec-
tion 1s 10 wg/nvi (0.005
ppm). Errors are asso-
ciated with calibration
and instrument drift
(from zero and span
settings).
Lower detection limit Is
0.0065 ppm
Lower detection limit is
0.3 mg/m3. Interference
of water vapor is signifi-
cant.
Reference
EPA-R4-
73-028d
d
EPA-R4-
73-028c
EPA-R4-
73-028a
of which are Quality Assurance
available, in the manuals.
b x = pollutant concentration.
. c EPA-650/4-75/016.
d Guidelines
the Ambient
for Development of a Qual
Air (Chemi luminescent),
ity Assurance Program for
Smith 4 Nelson, Research
The Continuous Measurement of Nitrogen Dioxide in
Triangle Institute, Research Triangle Park, N
.C. 27709
O O 3D
tu to rt> fl>
03 r* < O
a> n> -<<-*
-.
INS ->' O
o 3
o o 3
-ti O Z
<-» z o
u> o o
-s o
ro
o
oo
o
-------
AMBIENT AIR TECHNIQUES (cont.)
Pollutant
Partlculates
NO,
EPA
Method
or
Number
High
Volume
Arsenlte
Bias (absolute,
or percent of
mean concentra-
tion)
No Information
-3% (50-300
pg/m-3)
Precision (standard deviation,
or coefficient
within
laboratory
3X
8 pg/m3
(50-30a pg
m3)
of variation)
between
laboratory
3.7J
11 pg/m3
(50-300 pg/
m3)
Comments
Minimum detectable limit
Is 3 mg. Shorter samp-
ling periods give less
precise results, biased
high.
A tentative method.
Lower detectable limit
Is 9 pg/m3.
Reference
EPA-R4-
73-028b
EPA-R4-
73-2800
-O O TO to n
CQ <- < o
n n -> rt-
" (/i i.
U> ' - O
O 3
O O 3
-h n- z
c+ Z O
to o o
cr
m ~n
-j .o
ro
o
vo
CO
o
------- |