Verification Testing
of
Air Pollution Control Technology
Quality Management Plan
Revision 2.3
EPA Cooperative Agreement CR-83416901-0
RTI Project 0212320
Prepared by:
RTI International
Engineering and Technology Unit
Research Triangle Park, NC 27709
Prepared for:
U.S. Environmental Protection Agency
Air Pollution Prevention and Control Division
Research Triangle Park, NC 27711
March 2, 2010
Signed by Jason Hill 3/2/2010 Signed by Michael Kosusko 3/17/2010
J. Hill, APCT Center Director Date M. Kosusko, EPA Project Officer Date
Signed by James B.Flanagan 3/2/2010 Signed by Robert S. Wright 3/17/2010
J. Flanagan, APCT Center Quality Manager Date R. Wright, EPA Quality Manager Date
APCT_QMP_v2.3+v100317
3/22/2010
-------
[This page intentionally left blank.]
APCT_QMP_v2.3+v100317
3/22/2010
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
Table of Contents
Section Page
List of Acronyms v
1.0 Introduction 1
2.0 Part A: APCT Center Management Systems 2
2.1 Management and Organization 2
2.1.1 RTI International 2
2.1.2 Testing Organizations 4
2.1.3 Responsibilities of APCT Center Team Members 5
2.1.4 Quality Management 9
2.2 Quality System Description 11
2.3 Personnel Training and Qualifications 12
2.4 Procurement of Items and Services 13
2.5 Records 14
2.5.1 Procedures 14
2.6 Computer Software and Hardware 17
2.7 Planning 18
2.8 Implementation of Work Processes 18
2.9 Assessment and Response 18
2.10 Quality Improvement 20
3.0 Part B: Collection and Evaluation of Environmental Data 20
3.1 Planning and Scoping 20
3.1.1 Generic Verification Protocols 21
3.1.2 Test/Quality Assurance Plans 23
3.1.3 Standard Operating Procedures 27
3.2 Design of Verification Tests 29
3.2.1 Intended Use of Data 29
3.2.2 Project Requirements for Data Quality Objectives 30
3.2.3 Performance Characteristics for Measurement Methods 30
3.2.4 Use of Accepted Analytical Procedures 30
3.2.5 Instrument Calibration 31
3.2.6 Sample Collection, Selection, Preparation, and Site Selection 31
3.2.7 Sample Handling, Tracking, Custody, Transportation, and Storage 32
3.3 Implementation of Planned Operations 33
3.4 Assessment and Response 34
3.4.1 Frequency, Types, and Reporting of Assessments 34
3.4.2 Assessments of Quality Systems 36
3.4.3 Technical Assessments 37
3.4.4 Audits of Data Quality 39
3.4.5 Corrective Action 40
3.5 Assessment and Verification of Data Usability 41
3.6 Record Keeping and Data Management 42
4.0 References 44
in
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
List of Figures
Figure Page
Figure 1. APCT Center Management Structure 1
Figure 2. RTI Quality Policy 2
Figure 3. RTI Management Organizational Structure for the APCT Center 3
List of Tables
Table Page
Table 1. APCT Center Test-Level Responsibilities 6
Table 2. APCT Center Program-Level Responsibilities 7
Table 3. Schedule of Assessments 35
IV
-------
APCT CENTER Quality Management Plan
Revision 2.3, Effective: March 2, 2010
List of Acronyms
ADQ audit of data quality
ANSI American National Standards Institute
APCT air pollution control technology
APPCD Air Pollution Prevention and Control Division
ASQ American Society for Quality (formerly ASQC - American Society for Quality
Control)
DAS Discovery and Analytical Sciences
DQI data quality indicator
DQO data quality objective
EISD Environmental and Industrial Sciences Division
EPA U.S. Environmental Protection Agency
ETS ETS International, Inc.
ETV environmental technology verification
GVP generic verification protocol
ISO International Organization for Standardization
NIST National Institute of Standards and Technology
NRMRL National Risk Management Research Laboratory
OAQPS Office of Air Quality Planning and Standards
ORD Office of Research and Development
OTAQ Office of Transportation and Air Quality
PE performance evaluation
PO project officer
QA quality assurance
QC quality control
QM quality manager
QMP quality management plan
QSA quality system assessment
RTI RTI International
SAC Stakeholders Advisory Committee
SOP standard operating procedure
SRM Standard Reference Material
SwRI Southwest Research Institute
TO testing organization
TO-QM testing organization quality manager
T/QAP Test/quality assurance plan
TSA technical systems assessment
VR verification report
VS verification statement
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
[This page intentionally left blank.]
VI
-------
APCT CENTER Quality Management Plan
Revision 2.3, Effective: March 2, 2010
1.0 Introduction
This quality management plan (QMP) applies to RTI International's (RTFs) Air Pollution
Control Technology Verification Center (APCT Center) operated under U.S. Environmental
Protection Agency (EPA) Cooperative Agreement CR-83416901-0. The APCT Center was
established in 1995 as part of the EPA's Environmental Technology Verification (ETV) Program
to accelerate the development and commercialization of improved environmental technologies
through third-party verification testing and reporting on the tested technologies' performance.
The APCT Center verifies the environmental performance of commercial-ready technologies in
collaboration with qualified testing organizations (TOs). Figure 1 shows the management
structure governing this collaboration. Verification provides potential purchasers and permitters
with an independent and credible assessment of what they are buying and permitting.
Verification testing is performed according to approved protocols. A technology's verified
performance is reported in verification statements signed by EPA.
Center Director
Jason Hill
Stakeholder Advisory
Committee
Technical Panel
Senior Technical Staff
James T. Hanley
Kathleen M. Owen
Jeffrey W. Portzer
Jonathan W. Thornburg
Quality Manager
James Flanagan
Business Advisor
Lynn Soby
Subcontractors
AirTech Mike Hartman
ETS John Mycock
SwRI Kent Spreen
RTI Support Staff
Document Control
Reporting
Communications
Web Site
Contracts
Figure 1. APCT Center Management Structure
The primary objective of the APCT Center is to create a highly credible verification testing
program for air pollution control technologies. The quality assurance (QA) activities of this
1 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
program are designed to ensure that the results of the verification tests are credible and
technically defensible.
This document is the basis for QA for the APCT Center. It describes the policies, organizational
structure, responsibilities, procedures, and quality systems that will be followed under this
cooperative agreement to meet the requirements of American National Standards
Institute/American Society for Quality Standard E4-1994, Specifications and Guide lines for
Quality Systems for Environmental Data Collection and Environmental Technology Programs
(ANSI/ASQC, 1994). This document complies with the EPA document, Environmental
Technology Verification Program Quality Management Plan (EPA, 2008).
2.0 Part A: APCT Center Management Systems
2.1 Management and Organization
2.1.1 RTI International
RTI is an independent organization dedicated to conducting innovative, multidisciplinary
research that improves the human condition. With a worldwide staff of more than 2,600 people,
RTI is active in health and pharmaceuticals, advanced technology, survey and statistics,
education and training, economic and social development, and environmental protection.
Universities in North Carolina founded RTI in 1958 as the first scientific organization in, and the
centerpiece of, the Research Triangle Park. Today, RTI serves clients in government, industry,
academia, and public service throughout the United States and abroad.
As shown in Figure 2, RTI management has established the following quality policy.
It is the policy of RTI International to provide our clients superior-quality research,
development, and technical services that meet the highest standards of professional
performance, satisfy client requirements, and deliver exceptional value. We achieve this
through teamwork and by striving to continuously improve our products and services.
Furthermore, we work with our clients to define requirements and clarify expectations,
including cost and time constraints. We ensure that our products and services comply with
requirements and meet or exceed client expectations. Finally, we recruit, develop, and retain
highly qualified and motivated staff.
Figure 2. RTI Quality Policy
Figure 3 depicts the RTI management organizational structure relative to the APCT Center.
Reporting to the president of RTI, the Discovery & Analytical Sciences Group (DAS) is the RTI
technical research group within which the APCT Center is housed. DAS conducts research in
several disciplines, including drug discovery, microbiology, analytical chemistry, aerosol
filtration, and exposure analysis. DAS employs more than 250 natural and social scientists,
engineers, and administrative support personnel. Jennifer Hunter-Cevera, Executive Vice
President for DAS, is responsible for all aspects of DAS's financial and technical performance.
2 of 45
-------
APCT CENTER Quality Management Plan
Revision 2.3, Effective: March 2, 2010
RTI International
V. Haynes, President
Discovery and Analytical
Sciences Unit
J. Hunter-Cevera, Exec.
Vice President
Technical
Responsibility
Quality
Responsibility
Analytical Sciences
E. Hill, VP
Environmental & Industrial
Sciences Division
Bruce Harvey, Sr. Director
Quality Systems Program
C. Salmons, Mgr
Aerosol Filtration and
Protection
J. Hanley, Mgr
APCT Center
J. Flanagan
Quality Manager
J. Hill
Director
Figure 3. RTI Management Organizational Structure for the APCT Center
Within DAS is the Environmental & Industrial Sciences Division (EISD), directed by Bruce
Harvey. Providing the management staff for the APCT Center, the Aerosol Filtration and
Protection group, managed by James Hanley, conducts research and technology development
related to aerosol generation and control related to environmental, building, and personal
protection. The APCT Center's technical staff members are drawn from other RTI centers based
on their expertise.
Providing QA support for this effort are the staff of the Quality Systems Program, housed within
the Analytical Sciences division of EISD, and managed by Cynthia Salmons.
3 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
2.1.2 Testing Organizations
A TO may be part of RTI, of the subcontractors named below, or of external TOs not affiliated
with the APCT Center. TOs that are independent (i.e., not a technology manufacturer's in-house
laboratory) may conduct verification tests provided that they work under a generic verification
protocol (GVP) that accepts the participation of external TOs. They must also meet EPA quality
requirements and all performance requirements in the GVP. TOs that are not independent may
conduct verification tests provided that they work under a GVP that accepts the participation of
non-independent TOs, and they must also meet EPA quality requirements and the requirements
of the GVP.
In addition to RTI personnel, the APCT Center currently consists of the following three
subcontractors, who participate as TOs: Air Tech Environmental, ETS International, Inc., and
Southwest Research Institute (SwRI). These subcontractors participate as follows:
• Air Tech Environmental provides point source field testing support.
• ETS conducts laboratory verification tests of baghouse filtration products. Its services
also include the following project management and technical support: (1) participation in
stakeholder and technical panel meetings; (2) preparation of GVPs; (3) marketing; and
(4) business planning
• SwRI conducts laboratory verification tests of mobile source emissions control
technologies. It provides technical support to RTI as appropriate with its expertise in the
area of engine emissions testing and its familiarity with various emissions control
technologies.
Other subcontractors may be added, following qualification by the APCT Center, to expand the
capability of verifying air pollution control technologies. Participation of these subcontractors
depends on the identification of work for which they are suited. Work by subcontractors is
implemented by specific scopes of work. The APCT Center director manages all subcontractors.
All APCT Center subcontractors are required to have a quality system that meets one of the two
following criteria:
• Either the system must be documented to conform to the requirements of ANSI/ASQC
Standard E4-1994, Specifications and Guidelines for Quality Systems for Environmental
Data Collection and Environmental Technology Programs (ANSI/ASQC, 1994), and
must include a QMP in accordance with the EPA Environmental Technology Verification
Quality Management Plan (EPA, 2008) and in accordance with EPA Requirements for
Quality Management Plans, EPA QA/R-2 (EPA, 200la);
• Or the system must be registered under International Organization for Standardization
(ISO) Standard 9001:2000, Quality Management Systems - Requirements (ISO, 2000)
and must possess the Level 1 quality manual, the Level 2 procedural documents (e.g.,
departmental policy and procedure manuals), and the Level 3 quality documents (e.g.,
standard operating procedures [SOPs]) that are required for such a registration. It must
4 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
also perform any quality activities (e.g., audits of data quality [ADQs]) that are required
by EPA, but are not part of their ISO registration.
These quality system requirements and EPA's quality requirements for ETV verification tests are
passed on to each TO via this document and via RTFs contractual agreements with the TO.
The APCT Center director and the Center quality manager (QM) ensure that a TO meets EPA's
quality requirements and the GVP's performance requirements and that it has a compliant quality
system, as determined by their review and approval of the TO's test/QA plan (T/QAP) and by
the independent assessments that are conducted of the TO's quality and technical systems staff.
Documentation defining the type and extent of oversight by the director over the TO's technical
work products is established by the director as part of RTFs contractual agreements with the TO.
Each TO must also prepare a T/QAP in accordance with the requirements for an EPA QA project
plan, given in the EPA QMP for the ETV Program (EPA, 2008) and in the guidance document,
EPA Requirements for Quality Assurance Project Plans, EPA QA/R-5 (EPA, 200 Ib). The test
leader and QM of the TO prepare the plan. APCT Center staff may assist the TO in preparing
the plan, but the TO retains the final responsibility for the systematic planning for and the
subsequent implementation of verification tests that meet EPA quality requirements. The plan is
reviewed and approved by APCT Center and EPA. This document is described in greater detail
in Section 3.1.2.
The TO prepares any SOPs that are needed for the specific verification test. Although these
SOPs may contain proprietary information, they are made available for review by APCT Center
staff and by independent assessors, both of which treat any designated proprietary information as
confidential. These documents are described in greater detail in Section 3.1.3.
The TO conducts self-assessments of its quality and technical systems, and it allows independent
assessments of the systems by the APCT Center QM and the EPA QM. The TO is responsible
for developing and implementing corrective actions in response to assessment findings.
Assessments are described in greater detail in Sections 3.4 and 3.5. For a summary of test-level
responsibilities, see Table 1.
2.1.3 Responsibilities of APCT Center Team Members
The management structure for the APCT Center is presented in Figure 1. The APCT Center
director (see Figure 3) manages program activities and coordinates them with the Stakeholders
Advisory Committee (SAC). Reporting to the director are an organizationally independent
APCT Center QM (see Figure 3), test leaders, and TO quality managers (TO-QMs). Specific
responsibilities within the APCT Center are summarized in Table 2, APCT Center Program-
Level Responsibilities.
5 of 45
-------
APCT CENTER Quality Management Plan
Revision 2.3, Effective: March 2, 2010
Table 1. APCT Center Test-Level Responsibilities
Responsibility
Maintain internal communications
Oversee verification test activities
Develop and manage verification test budget
Report progress and costs to APCT Center
director
Select specific technologies for testing
Prepare T/QAP
Assist in development of T/QAP
Review and approve test-level quality
documents
Submit test-level quality documents to EPA
Oversee TO-QM activities
Oversee QA aspects of verification tests
Select and manage test staff
Conduct verification test
Provide test-level quality training
Implement test-level quality training
Assess TO's quality system
Conduct technical assessments of tests
Conduct ADQ of test data
Assess test and ADQ results
Develop and implement corrective actions
Determine effectiveness of corrective actions
Prepare test documentation
Prepare verification report and statement
(VR.VS)
Prepare quality aspects for VR and VS
Review and approve test results, VS, and VR
Submit VS and VR to EPA
Store test-level quality documents and test data
APCT Center
Director*
X
X
X
X
X
X
X
X
X
X
Quality
Manager
X
X
X
X
X
X
X
X
X
Test Leader
X
X
X
X
X
X
X
X
X
X
Testing QM
X
X
X
X
X
X
X
X
X
X
Note: ADQ=audit of data quality; APCT=air pollution control technology; QA=quality assurance; TO=testing
organization; TO-QM=testing organization quality manager; T/QAP=test/quality assurance plan; VR=verification
report; VS=verification statement.
* The director may delegate test-level technical activities in a specific technology area to RTI personnel, but he/she
retains responsibility for the quality of any delegated technical activities.
6 of 45
-------
APCT CENTER Quality Management Plan
Revision 2.3, Effective: March 2, 2010
Table 2. APCT Center Program-Level Responsibilities
APCT Center Program-Level Responsibilities
Overall APCT Center program and quality
responsibility
Coordinate APCT Center activities with EPA
Develop APCT Center implementation plan
Communicate APCT Center activities to SAC and
constituents
Assist in communicating APCT Center activities to
constituents
Prepare APCT Center QMP; revise annually as
needed
Review and approve APCT Center program-level
documents
Implement quality system (as per QMP) in APCT
Center
Communicate with EPA QM and TO-QMs
Oversee program-level quality activities
Assess APCT Center quality system on an annual
basis
Submit APCT Center program-level quality documents
to EPA
Chair and coordinate SAC meetings
Review APCT Center activities and advise director
Select and prioritize technologies, including develop
outcome projections.
Provide input for selecting and prioritizing technologies
Provide advice about specific technologies
Select technical panel
Chair and coordinate technical panel meetings
Prepare GVPs
Provide input for development of GVP
Assist development of quality aspects of GVP
Approve external TOs
Manage subcontractors
Review and approve APCT Center test-level quality
documents
Oversee verification tests by TOs
Review and approve VRs and VSs
Develop impact analyses for verified technologies
Submit test-level quality documents, VR, and VS to
EPA
Store APCT Center program-level documents and
records
APCT
Center
Director*
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
Quality
Manager
X
X
X
X
X
X
X
SAC
X
X
X
Technical
Panel
X
X
Note: APCT=air pollution control technology; DQO=data quality objective; GVP=generic verification protocol;
QMP=quality management plan; QM=quality manager; SAC=Stakeholders Advisory Committee; TO=testing
organization; TO-QM=testing organization quality manager; VR=verification report; VS=verification statement.
* The director may delegate program-level technical activities for specific technology areas to RTI personnel, but he/she
retains the responsibility for the quality of any delegated technical activities. An RTI staff member is designated to
substitute for the APCT Center director when the director is absent.
7 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
The APCT Center Director has overall responsibility for APCT Center activities, which
include oversight for all verification testing and reporting, negotiations with technology
developers and vendors, selection of specific technologies for verification testing, management
of RTI personnel, direction of subcontractor efforts, approval of participation by external TOs,
and coordination with the SAC and the EPA project officer (PO). The director serves as the chair
of the SAC. The director has overall responsibility for quality at the program level and in
verification tests. The director may delegate specific technical activities to RTI personnel, but
he/she retains responsibility for the quality of any delegated technical activities. The director
prepares GVPs and reviews and approves T/QAPs, verification reports, and verification
statements. The director submits these documents to the EPA PO for review and approval.
Deborah Franke, a senior member of the APCT Center technical staff, reports to the director and
substitutes for the director if that individual is absent. She will be responsible for any functions
delegated by the director.
The APCT Center QM is responsible for preparing and revising this QMP. The QM assists the
director in the preparation of GVPs. The QM communicates directly with the EPA QM and with
TO-QMs on quality-related issues. The QM reviews and approves T/QAPs, verification reports,
verification statements, verification test results, and associated quality records. The QM is free
from personal and external barriers to independence, is organizationally independent from data
collection activities, and is able to maintain an independent attitude and appearance. The QM
provides technical assistance to TOs regarding EPA quality requirements. The QM
communicates directly with TO-QMs on quality-related issues. The QM is also responsible for
conducting independent assessments of the quality systems of TOs and independent technical
assessments (i.e., technical systems audits [TSAs] and performance evaluations [PEs]) of
verification tests in cooperation with the EPA QM. The QM is responsible for determining the
effectiveness of corrective actions implemented in response to independent assessment findings.
The QM will review this QMP on an annual basis and will revise it as necessary to reflect any
changes that have occurred in the organization and the policy of the APCT Center or in EPA
quality requirements since the last revision. These responsibilities are described in greater detail
in Section 2.1.4.
The SAC is responsible for advising the director with respect to potentially useful technologies
and other program-related issues. With the advice and concurrence of the EPA PO, the director
chairs and coordinates the SAC to guide and direct APCT Center activities, to assist in selecting
and prioritizing technologies for verification testing, and to serve as a communications link with
their constituents. The SAC is made up of 20 to 25 leaders from various segments of the air
pollution control community, including regulators/permit writers, technology
developers/vendors, buyers/users, environmental associations, and EPA representatives,
including the PO. The EPA PO is consulted concerning the director's selection of SAC members.
EPA's Office of Research and Development (ORD), its Office of Air Quality Planning and
Standards (OAQPS), and its Office of Transportation and Air Quality (OTAQ) are expected to
contribute SAC members. The SAC members have broad experience in the air pollution control
field. They provide the verification center with a diversity of opinions and viewpoints concerning
air pollution control technologies and their application.
For each TO (see Section 2.1.2), the test leader manages the verification test for a specific
technology. The test leader may be an employee of RTI, a subcontractor, or an external TO.
8 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
The test leader develops personnel requirements for verification tests, communicates with and
coordinates testing staff, identifies candidates for the technical panel, and proposes a budget for
the test program from the start of the process through preparation of the verification report. The
test leader is responsible for providing personnel to support the technical panel. The APCT VC,
in cooperation with the technical panel and the TO-QM, prepares a draft GVP; the TO test leader
prepares a test-specific T/QAP, and any needed test-specific SOPs. The test leader is responsible
for developing and implementing corrective actions that are taken in response to assessment
findings. The TO-QM is free from personal and external barriers to independence, is
organizationally independent from data collection activities, and is able to maintain an
independent attitude and appearance. The TO-QM:
• Provides support and oversight for the QA activities associated with specific verification
tests;
• Communicates directly with the QM on quality-related issues;
• Assists test leaders in the preparation of test-specific T/QAPs and any needed test-
specific SOPs;
• Conducts self-assessments of the TO's quality and technical systems and determines
whether the tests were implemented in accordance with the T/QAP and EPA quality
requirements;
• Conducts ADQs of the verification test results;
• Reviews and approves test results, verification reports, and verification statements; and
• Determines whether the test results attain data quality objectives (DQOs) and whether
they support the verification reports and verification statements.
The responsibilities of the TO-QM are described in greater detail in Section 2.1.4.
Technical panels are convened to provide input to the director in the development of GVPs for a
specific technology and to ensure that the interests of their constituencies are met. They provide
the director with advice about specific technologies to be selected for testing. Technical panel
members include technology developers, technology users, and the regulatory community. The
members of each technical panel are technical experts with specialized knowledge and fields of
interest that are related to the specific technology being verified and to the associated test
methods.
2.1.4 Quality Management
Overall responsibility for quality at the APCT Center program level and in all verification tests
remains with the director. The director has resources available to ensure conformance with EPA
quality requirements. The director has the authority to issue a stop work order in the event that
unsafe work or work of inadequate quality is identified. The director reviews and approves
GVPs, T/QAPs, verification reports, and verification statements.
9 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
The independence and objectivity of the APCT Center quality system is bolstered by the APCT
Center QM being located in a different RTI administrative unit (i.e., the Discovery and
Analytical Sciences group, or DAS) from RTFs APCT Center testing personnel (i.e., Center for
Aerosol Technology). Likewise, TO-QMs work with test leaders, but are not part of the
administrative units generating the verification test data. Reports of quality document reviews,
reviews of verification test results, and assessment findings go directly from the QM to the
director and from the TO-QMs to the test leaders. These individuals are free of any real or
perceived conflicts of interest as might occur from too close of an administrative association with
the data collection activities. They have no stake in the outcome of the verification tests other
than that the environmental data be collected objectively and in accordance with the quality
documents and EPA quality requirements.
The APCT Center QM is responsible for reporting to the APCT Center director whether
verification tests are performed in compliance with EPA quality requirements and with the
quality requirements in this document, in GVPs, and in test-specific T/QAPs, and whether test
results demonstrate that test data attain DQOs. The following are the specific responsibilities of
the APCT Center QM:
• Prepares this document, reviews it on an annual basis, and revises it as needed;
• Assists test leaders and TO-QMs regarding quality issues relating to specific verification
tests;
• Reviews and approves the GVPs, T/QAPs, and any needed SOPs that are developed by
test leaders, TO-QMs, and TOs;
• Communicates with the director, test leaders, TO-QMs, and the EPA QM regarding
quality-related issues;
• Conducts self-assessments of the APCT Center quality system, independent assessments
of TOs' quality systems, and test-specific technical assessments of verification tests in
cooperation with the EPA QM;
• Determines the effectiveness of corrective actions implemented in response to
independent assessment findings;
• Reviews and approves the test results and the QA and quality control (QC) data from
verification tests to determine whether test data attain DQOs; and
• Reviews and approves verification test reports and verification statements.
TO-QMs handle those QA activities directly associated with specific verification tests. They
may be employees of RTI, a subcontractor, or an external TO, but they must be independent of
the administrative unit performing the verification tests and must be free of any real or perceived
conflicts of interest. Following are the specific responsibilities of the TO-QM:
10 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
• Assists test leaders in the preparation of T/QAPs, including data quality indicator (DQI)
acceptance criteria that must be attained so that test data attain the DQOs specified in the
approved GVP;
• Assists TOs, in cooperation with test leaders, in the preparation of any needed SOPs to be
attached to the T/QAPs, if another level of technical detail is needed to describe test
activities;
• Communicates with test leaders, TO staff, and the QM regarding quality-related issues;
• Ensures, in cooperation with test leaders, that TOs conduct verification tests in
conformance with EPA quality requirements, the quality requirements in this document,
GVPs, T/QAPs, and any needed SOPs;
• Oversees test-specific quality training and ensures that all TO personnel understand the
QA requirements;
• Conducts self-assessments of the TOs' quality and technical systems and determines the
effectiveness of corrective actions implemented in response to self-assessment findings;
• Performs an ADQ for at least 10 percent of verification test data;
• Reviews and approves the test results, and the QA and QC data from verification tests, to
determine whether test data attain DQOs; and
• Assists the test leader in the preparation of the quality-related content of each verification
report and verification statement.
2.2 Quality System Description
The quality system for the APCT Center has five levels.
The foundation level consists of the Environmental Technology Verification Program Quality
Management Plan (EPA, 2008), which addresses all aspects of EPA's ETV program, and
conforms to the ANSI/ASQC E4-1994 standard (ANSI/ASQC, 1994).
The second level consists of this QMP, which describes APCT Center's quality system. It is a
program-specific application of the EPA ETV QMP.
The third level consists of technology-specific GVPs, which apply the principles of the ETV
and APCT Center QMPs to specific technology testing scenarios. They are prepared by a Center
technical staff person, with the assistance of technical panels and TOs for verification tests of
specific technologies. These GVPs are developed with assistance from stakeholder technical
panels and are submitted to the EPA PO and EPA QM for approval. These documents define the
types and characteristics of data that must be present in verification statements and reports in
order for technology-specific verification tests to be accepted as credible by EPA and other
stakeholders. Technical panels provide for the verification tests to ensure that the interests of
their constituencies are met. The GVPs are applicable to any TO that is approved to perform
11 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
verification tests for the APCT Center. They are not specific to a particular TO. The GVPs are
developed to assure there is no bias towards or against a specific technology or a specific TO
during verification tests. They are based upon detailed examination by all stakeholders of the
issues associated with the acceptance of the technologies. They specify strong EPA oversight
and QA activities to assure fairness and consistency in the conduct of the verification tests and
credibility in the verification test results.
The fourth level consists of the test-specific T/QAP, which is prepared jointly by an APCT
Center staff person and the TO. It is reviewed by the APCT Center director and QM and is
submitted to the EPA PO AND EPA QM for approval. The T/QAP describes how the TO
produces the verification test data as specified in the approved GVP. All data must be generated
in accordance with the approved T/QAP.
The T/QAP must meet all the requirements for QA project plans, which are specified in EPA
Requirements for Quality Assurance Project Plans, EPA QA/R-5 (EPA, 200 Ib) and contain all
the required elements of a QA project plan. Additional information regarding QA project plans
can be found in Guidance for Quality Assurance Project Plans, EPA QA/G-5 (EPA, 2002a).
The T/QAP must contain a detailed description of the planned verification test, including the
TO's organizational structure, its management and personnel, the test schedule, test
documentation, sampling and analytical methods, and other operational procedures. It also must
specify the QA and QC procedures, calibration traceability, and DQIs for obtaining verification
data of sufficient quantity and quality to satisfy the DQOs specified in the GVPs. If GVPs
specify the needs of verification data users, then T/QAPs specify what the verification data
generators will do to meet these needs. The T/QAP must describe how verification test data will
be reconciled with the DQOs. T/QAPs are described in more detail in Section 3.1.2.
The final level consists of the assessments of the TO's quality system and technical
assessments (e.g., TSAs, PEs, and ADQs) that will be conducted during the verification tests by
the QM in cooperation with the EPA QM. The findings of the quality system and technical
assessments will be reported by the QM to the APCT Center director. Assessments of the
quality system and technical assessments are described in greater detail in Sections 2.9 and 3.4.
2.3 Personnel Training and Qualifications
The director is responsible for assessing the needs, providing the resources, and monitoring the
progress of professional development and general training for RTI-based APCT Center
personnel. The director identifies general training needs by evaluating personnel qualifications,
experience, and performance based on each position's job description and documented
performance expectations, as applicable. General training needs are assessed and documented as
part of the evaluation of each employee's performance in the APCT Center.
The QM and TO-QMs are selected based on the following qualifications:
• Educational background or a degree relevant to the application of QA principles to
technology demonstration projects and programs;
• Work experience specific to QA of technology demonstration projects and programs;
12 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
• Familiarity with the ETV quality management system and quality requirements, as
demonstrated by work experience with the ETV program or on the job training on the
ETV program, ETV QMP and APCT QMP; and
• Work experience in quality management and QA.
Assessors have a minimum of 4 years' full-time appropriate practical workplace experience (not
including training), at least 2 years of which have been in QA activities. They have undergone
training to the extent necessary to ensure their competence in the skills required for carrying out
assessments and for managing assessments. They are free from personal and external barriers to
independence, are organizationally independent from data collection activities, and are able to
maintain an independent attitude and appearance.
Test leaders are selected based on the following qualifications:
• Educational background or a degree directly relevant to the technology;
• Work experience specific to the technology; and
• Work experience in project management.
Test leaders are responsible for assessing TO personnel qualifications to perform verification
tests and for identifying test-specific training needs. They identify test-specific training needs by
evaluating personnel qualifications, experience, and performance based on the requirements in
the project's scope of work. Particular areas that are evaluated include health and safety training
and procedures for handling confidential information, as applicable. Training reassessment is
conducted whenever an individual's job function changes, such as reassignment to a new work
group, job redesign, reorganization, or promotion. Supplemental training is provided when
deficiencies in performance are observed.
TO personnel are selected by the test leader based on the following qualifications:
• Educational background and/or a degree directly relevant to the technology; and
• Work experience specific to the technology.
TO personnel assigned to a verification test are given only those tasks and responsibilities that
are commensurate with their training, education, and work experience. For example, a laboratory
analyst is taught unfamiliar sampling or measurement procedures through externally provided
training courses or through in-house training. Personnel performing data verification or
validation have the appropriate background in science or engineering and appropriate
background in the measurements being verified or validated. They also must demonstrate
competence in the measurement procedure to an experienced supervisor.
2.4 Procurement of Items and Services
RTI Policy No. 4001, Institute Procurement, and RTFs Office of Purchasing Standard
Operating Procedures Manual should be consulted for a general description of quality systems
13 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
for procurement of items and services. The operations of RTFs Office of Purchasing are
included under the certification of one of RTFs separately registered ISO 9001:2000 technical
units. Those certified Purchasing procedures apply to all technical units of RTF The most
frequent service procured under this cooperative agreement is verification testing performed by a
subcontractor acting as a TO. RTI and its subcontractors have contractual agreements regarding
such verification tests, and no new procurement documents are needed for these tests. In
preparation for a specific verification test, the director and the subcontractor's management
discuss the technical requirements for the test that have been established by the technical panel,
and they reach an agreement on the technical activities that the subcontractor will perform to
satisfy these technical requirements. The subcontractor also prepares a T/QAP that satisfies
EPA's quality requirements and the GVP's quality requirements, including the DQOs for the
test. EPA's quality requirements for the ETV Program together with RTFs additions flow
through to the subcontractors via the contractual agreements.
During verification tests, the subcontractor management regularly reports its technical progress
to the director. The QM assesses the subcontractor's quality system, conducts independent
technical assessments during verification tests, and reviews and approves the written reports of
ADQs that are conducted by TO-QMs after verification tests.
After the verification test data have been verified by the TO staff, the TO-QM conducts an ADQ,
reconciles verification test results with the DQOs, and prepares a written assessment report for
review by the APCT Center QM. The test leader, with the cooperation of the TO-QM, prepares
the verification report and verification statement, which are reviewed and approved by the
director, the QM, and the EPA PO AND EPA QM.
2.5 Records
APCT Center quality records are the following documents:
• APCT Center QMP (this document);
• Minutes of stakeholder meetings and technical panel meetings;
• GVPs;
• T/QAPs and SOPs;
• Raw data (all written and electronic data generated when a verification test is conducted);
• Verification reports and verification statements; and
• Project reviews and assessment reports.
2.5.1 Procedures
• All reports, internal documents and records will be clearly identified by title, date, author
or responsible person, and report, version or document number. Document numbers
(e.g., SOPs) will follow an organizationally defined numbering system.
14 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
The number of documents being generated for the project is not great enough to warrant a
document indexing system. A master list of all documents and records for the project is
maintained in the project computer directory.
Control of quality records, including procedures for identifying, collecting, indexing,
accessing, filing, storing, maintaining, and disposing of quality records will follow RTI
policies. APCT SOP 100, Document Control, provides detailed procedures for this
project.
Standard RTI procedures for maintaining, establishing, and implementing an appropriate
chain of custody and confidentiality for evidentiary records will be followed.
Obsolete records and documents will be identified as such. They will be maintained in
clearly marked separate computer directories or in separate hardcopy files. As
appropriate, they will be removed to archival storage once the project is completed.
APCT SOP 100, Document Control, provides detailed procedures for this project.
Record retention times will be based on contractual and statutory requirements or will
follow RTI practices.
This QMP is prepared by the QM, and is reviewed and approved by the director and the EPA PO
and EPA QM. It will be reviewed by the QM on an annual basis and will be updated as
necessary. The updated document will be reviewed and approved by the director and the EPA
PO AND EPA QM. It will be retained by the APCT Center for a period of not less than 7 years
after final payment of the assistance agreement in accordance with the requirements of Part A,
Section 5.3 of Environmental Technology Verification Program Quality Management Plan
(EPA, 2008). The QM is responsible for establishing procedures to securely store this document.
Minutes from SAC meetings and technical panel meetings are prepared by APCT Center staff
and are reviewed by meeting attendees and the director. They are posted on APCT Center web
pages (see http://etv.rti.org/apct/advisory/index.html and http://etv.rti.org/apct/tech/index.cfm,
respectively). They will be retained by the APCT Center for a period of not less than 7 years
after final payment of the assistance agreement. The director is responsible for establishing
procedures to securely store these documents.
GVPs are prepared by the APCT director, with the assistance of the APCT Center staff, and are
reviewed by technical panels. They will be reviewed by the director and approved by the EPA
PO and EPA QM before their use. They will be retained by the APCT Center office for a period
of not less than 7 years after final payment of the assistance agreement. The director is
responsible for establishing procedures to securely store these documents.
T/QAPs and any technical SOPs that are needed for verification tests will be prepared by the test
leaders with the assistance of technology-specific QMs. They will be reviewed and approved by
the director, the QM, and the EPA PO and EPA QM before their use. The T/QAPs will be
retained by the APCT Center and by the TOs for a period of not less than 7 years after final
payment of the assistance agreement. The QM and the test leaders are responsible for
establishing procedures to securely store T/QAPs in their respective organizations. The test
leader is responsible for establishing procedures to securely store any technical SOPs.
Administrative SOPs are numbered consecutively, are stored online in the APCT folder, and are
15 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
maintained with other APCT quality documents. The director will request his staff to develop
administrative SOPs as necessary, however, he/she maintains responsibility for them.
Raw data (electronic and printed) collected during verification tests, and any calculations or
documents derived from such data will be retained by the TO or by the APCT Center for a period
of not less than 7 years after the final payment of the assistance agreement. These data,
calculations, and documents will be clearly identified by verification test, date, observer/author,
and originating TO. The test leader is responsible for establishing procedures to securely store
these data, calculations, and documents.
Verification reports, including 3- to 5-page verification statements that thoroughly document the
verification test results, will be prepared by the APCT director with input from the APCT QM,
TO test leader and TO-QM. The format for the reports will include:
• Thorough description of the technology that was tested;
• Test methods used and a justification for their selection;
• Organizations conducting the test and providing QA oversight;
• Operating parameters and conditions under which the testing was performed;
• Statistical analysis of the test results;
• Reconciliation of the test results with the DQOs;
• Statement regarding independent and self-assessment findings; and
• Any limitations on use of the test data.
The TO will explain and document any necessary deviations from the GVPs or T/QAPs, will
document raw data, and will present QA results.
These reports will be reviewed and approved by the director and the QM and submitted to the
EPA PO and EPA QM for approval. They will be retained by the APCT Center for a period of
not less than 7 years after final payment of the assistance agreement. The director is responsible
for establishing procedures to securely store these documents.
Reports of self-assessment findings of TO quality and technical systems will be prepared by the
TO-QM and will be sent to the test leader. Reports of ADQs of verification test results will be
prepared by the TO-QM and will be sent to the test leader and the QM. Reports of independent
assessment findings of TO quality and technical systems will be prepared by the QM and will be
sent to the TO-QM and the director. Reports of self-assessments of the APCT Center quality
system will be prepared by the QM and will be sent to the director. These reports will be
available for review during independent assessments. These reports will be retained by TOs
and/or by the APCT Center for a period of not less than 7 years after the final payment of the
assistance agreement. The test leader and the director are responsible for establishing procedures
to securely store these project reviews and assessment reports in their respective organizations.
16 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
2.6 Computer Software and Hardware
Any computer hardware/software configurations developed to support APCT Center verification
tests will be tested prior to use. The results of the testing will be documented. The
configurations will be properly maintained and documented subsequent to testing. All
applications and configurations will be tested using a test data set or by running a shakedown test
of the system to ensure that they are operating according to specifications.
The APCT Center or the TOs will maintain, control, and document such configurations,
including:
• Retaining computer support personnel to correct any hardware or software failure with
minimal downtime to the program;
• Tracking upgrades to hardware and revisions to software developed by the APCT Center
or the TOs; and
• Documenting software names, versions, and copyright dates.
The APCT Center expects to use only standard commercial software for office operations (e.g.,
word processing software, spreadsheet software) and does not expect to establish acceptance
criteria for such software. This is consistent with Part A, Section 6.1 of Environmental
Technology Verification Program Quality Management Plan (EPA, 2008).
The APCT Center expects to use standard commercial software for data management operations,
including data reduction (e.g., spreadsheets). Such calculations will be carefully checked using
alternative methods (e.g., calculator) and results will be verified by use of standard input (i.e.,
"calibration") data. Additionally, they will be spot-checked by the QM (APCT Center and/or
TO) as part of the normal ADQ process.
The APCT Center may use computer hardware/software configurations that are integral to
measurement and testing equipment that are calibrated for a specific purpose (e.g., engine
testing). All computer hardware or software that are anticipated to be used in the APCT Center
will only be able to alter the response of the equipment due to alteration of constants (e.g.,
calibration data). All such operations will be documented in applicable SOPs by the TO.
The APCT Center has no current plans to change any computer hardware/software
configurations developed to support the APCT Center or verification tests. Any changes that
might occur are unlikely to have an effect on the technical and quality objectives of verification
tests because these objectives are established in the GVPs and are independent of any
configurations developed to implement the GVPs.
Computer databases containing test data will be password-protected. Electronic data transfers
will be protected by encryption or other means to prevent tampering and ensure confidentiality.
17 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
2.7 Planning
The director developed an implementation plan for the APCT Center under the previous
cooperative agreement. This plan describes the technical approach that will be followed to
implement the APCT Center verifications. Elements of the technical approach include the
following:
• Organizational Phase/Management - This element discusses management functions,
outreach, the SAC, the QMP, GVPs, and technology identification and prioritization.
• Technology Verification - This element addresses vendor solicitation, GVP and T/QAP
development, testing and evaluation, QA, verification reports, and verification
statements.
• Schedule - This element presents the anticipated project schedule.
• Success Measures - This element describes measures that are planned to ensure the
success of the APCT Center.
• Reports - This element describes the reports that the director will submit to the EPA PO.
• Verification Statements - This element describes the verification statements that will
summarize the verification test results.
The implementation plan also includes sections on APCT Center management, SAC
management, technology verification management, subcontractors, quality management, the
APCT Center QMP, and verification QA/QC.
2.8 Implementation of Work Processes
Verification testing will proceed after a GVP and the T/QAP for a specific technology category
have been approved by the EPA PO and EPA QM. Depending on the technology type, several
T/QAPs can sometimes fall under one GVP. Verification tests will be conducted at locations
appropriate for the technology being tested, as established by the approved GVP. The
verification test will then be carried out according to the approved T/QAP. The data from the test
will be compiled, validated, and reported in a form consistent with the objectives of the test.
After the test has been completed and the data have been validated by the TO's staff, the TO-QM
will perform an ADQ on the data. See Section 3.3 for more information on the implementation of
work processes during verification tests.
2.9 Assessment and Response
The goal of an assessment is accomplished by examining the processes used by an organization
to plan, implement, and assess the effectiveness of the QA activities that are described in its
QMP and that are applied to programs that collect or use environmental data. The purpose of the
assessment is to provide valid feedback to management on the adequacy, implementation, and
effectiveness of the quality system. The assessment may also examine human resource issues,
such as whether personnel have adequate QA training.
18 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
The EPA QM, the QM, and/or the TO-QMs will perform self-assessments and independent
assessments of the APCT Center quality system and TOs' quality systems. They will also
perform self-assessments and independent assessments (i.e., TSAs and/or PEs) of measurement
systems during verification tests. The TO-QM will conduct an ADQ for all critical
measurements at the end of each verification test. See Section 3.4 for additional details.
Assessors will have a minimum of 4 years' full-time appropriate practical workplace experience
(not including training), at least 2 years of which should have been in QA activities. They will
have undergone training to the extent necessary to ensure their competence in the skills required
for carrying out assessments and for managing assessments. The assessors assigned to conduct a
specific assessment will collectively possess adequate professional proficiency for the tasks
required. They will be free from personal and external barriers to independence, will be
organizationally independent, and will be able to maintain an independent attitude and
appearance. They will use due professional care in conducting the assessment and in preparing
related reports.
Assessors will have sufficient authority, organizational freedom, and access to programs,
managers, documents, and records to:
• Identify both quality problems and noteworthy practices;
• Propose recommendations for resolving quality problems; and
• Independently confirm implementation and effectiveness of corrective actions.
Reports of self-assessment findings of TO quality and technical systems will be prepared by the
TO-QM and will be sent to the test leader. Reports of ADQs of verification test results will be
prepared by the TO-QM and will be sent to the test leader and the QM. Reports of independent
assessment findings of TO quality and technical systems will be prepared by the QM and will be
sent to the TO-QM and the director. Reports of self-assessments of the APCT Center quality
system will be prepared by the QM and will be sent to the director. These reports will be
available for review during independent assessments.
The reports of assessments of quality or technical systems will contain a statement on the
effectiveness of the systems that were assessed. The reports will give all of the details of the
assessment necessary to understand the current status of the project and to estimate whether
DQOs and DQI acceptance criteria will be attained. It will include an introduction describing the
purpose and scope of the assessment, the technical basis for the assessment, an executive
summary (as needed), a detailed account of findings and further observations, a list of the
assessors, and a list of the TO managers and personnel. The report will recommend corrective
actions, if such are indicated by the findings.
Reports of the ADQs will describe the results of custody tracing, a study of data transfer and
intermediate calculations, a review of QA and QC data, a study of project incidents that resulted
in lost data, and a review of study statistics. The ADQ reports end with conclusions about the
quality of the data from the project and their fitness for their intended use.
19 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
As specified in Sections A9.4 and B4.3 of Environmental Technology Verification Program
Quality Management Plan (EPA, 2008), responses by the assessees to adverse findings and
recommendations are required within 10 working days of receiving the assessment report. The
lead assessor has the responsibility to correct any errors in fact that are demonstrated by evidence
to the contrary. Any disputes encountered as a result of assessments will be presented in writing
to the director, who will resolve the disputes. The test leader will be responsible for developing
and implementing any corrective actions. The assessors will follow up with appropriate
documentation to confirm the implementation and effectiveness of the corrective actions.
If assessors identify a severe problem affecting verification quality, the QM and the TO-QMs
will notify the director to halt the verification until the problem is addressed. If assessors identify
a problem endangering the health and safety of personnel, they have the responsibility to bring
the danger to the immediate attention of the director, the test leader, and the onsite testing
personnel.
2.10 Quality Improvement
The APCT Center director is responsible for improving the quality of the APCT Center's system
and processes used to plan and produce verification test data. He/she is assisted by the APCT
Center QM and all others on the Center team.
Activities will be based on the technical expertise of APCT Center staff and stakeholders, review
of the results of verification tests, QA assessment findings, and inputs from many other sources.
GVPs will be reviewed with the technical panels and revised to ensure that they are consistent
with stakeholder objectives, DQOs, and resource constraints pertinent to actual testing. T/QAPs
and SOPs will be reviewed and revised to ensure that the testing environment meets specified
DQOs.
When problems with the APCT Center system and processes, or with technology-specific items,
are determined by the Center director to be significant, the probable root cause shall be identified
and the appropriate changes determined, implemented, and documented in a timely manner.
3.0 Part B: Collection and Evaluation of Environmental Data
This section contains the specifications and guidelines that apply to test-specific environmental
activities involving the generation, calculation, analysis, evaluation, and reporting of test data.
3.1 Planning and Scoping
After a GVP has been approved by the EPA PO and EPA QM, it will form the basis for a T/QAP
that will be prepared for each verification test or each group of very similar verification tests. If
another level of detail is needed for describing test activities, an SOP will be written and attached
to the T/QAP.
T/QAPs and any SOPs that are needed will be prepared by the test leader with the assistance of
the TO-QM. They will be reviewed and approved by the director, the QM, and the EPA PO and
EPA QM. The test leader and TO-QMs will be responsible for monitoring the implementation of
the T/QAPs and any needed SOPs by the TO. All data collected during a verification test must be
20 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
generated in accordance with the GVP, T/QAP, and SOPs prepared for that test. The T/QAP and
SOPs provide concrete steps to implement the GVP and will provide sufficient detail to
demonstrate that:
• The verification test's technical objectives are identified and agreed upon;
• The intended measurements and data acquisition methods are consistent with the
verification test objectives;
• DQI acceptance criteria specified in the T/QAP and SOPs are consistent with the DQOs
specified in the GVP;
• The assessment procedures are sufficient for determining whether measurement data of
the type and quality needed and expected are obtained and whether measurement data
attain DQOs; and
• Any potential limitations on the use of the data can be identified and documented.
3.1.1 Generic Verification Protocols
GVPs provide the necessary framework for development of the more detailed T/QAP. The
specific content and level of detail given in GVPs may vary between different air pollution
control technologies in response to the testing and quality requirements for each technology.
To start the development of a GVP, the test leader, the TO-QM, and other APCT Center
personnel will review existing test methods that might be applicable and prepare a review of
their strengths and weaknesses. They will prepare a draft GVP for discussion purposes, either by
synthesizing existing test methods or using their technical judgment. DQOs for critical
measurements will be clearly defined in the draft GVP. During the development of the draft
GVP, technical panel members will provide input into the GVP to ensure that their constituents'
interests are met. To the extent practical, the GVP will be generally accepted by the technical
panel members. The draft GVP will be reviewed by the director and the QM before it is sent to
the EPA PO and EPA QM for approval. The GVP will remain a draft document until completion
of the first verification so that appropriate corrections can be made.
Each GVP will contain elements that are common to all technologies, as well as elements that
are specific to the technology being tested. It will ensure that all important information is
obtained from the verification test. Examples of information that could be important are
emissions control efficiency, emission rates, energy usage, wastewater and solid waste effluents,
byproducts, reliability, operating limitations, operation and maintenance requirements, and scale-
up factors or procedures. The GVP must be structured so that any specific information required
for compliance with new and existing regulatory standards will be obtained during the
verification test. The forms of standards (e.g., whether requirements are based on a percent
reduction, outlet emissions limit, or production-based limit) will be considered in developing the
GVP.
21 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
In general, the GVP may address the following issues:
• General description of the APCT Center;
• Responsibilities of all involved organizations;
• Experimental design;
• Equipment capabilities and descriptions;
• Description and use of field test sites;
• Description and use of test sites;
• QA/QC requirements;
• Data collection, handling, and reporting;
• Requirements for other documents;
• Health and safety; and
• References.
The QA/QC requirements section of the GVP typically describes the activities that verify the
quality and consistency of the work. The GVP will include DQOs for critical measurements,
which are qualitative or quantitative statements that:
• Clarify the objectives of the verification test;
• Define the most appropriate type of data and amount of data to collect;
• Determine the most appropriate experimental conditions under which to collect the data;
and
• Specify tolerable limits on the uncertainty of all critical measurements that will be used
as the basis for establishing DQI acceptance criteria, such as accuracy, precision, limit of
detection, and correct chemical identification. Limits on uncertainty may also be
expressed as tolerable limits on decision errors (for testing hypotheses) or as acceptance
widths for confidence or probability intervals (for estimating parameters).
The process of developing DQOs may be viewed as a strategic planning effort, based on the
scientific method that is used to prepare for data collection. DQOs apply to all verification tests
of a technology. The process is described in Guidance for the Data Quality Objectives Process,
EPA QA/G-4 (EPA, 2006). It provides a systematic procedure for identifying performance
variables that must be assessed during the verification tests and the data quality that is needed for
a credible measurement of that variable. The DQO process may or may not have a statistical
basis as is appropriate for the specific technology to be verified.
22 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
The technical panel may suggest specific QA procedures in the GVP or may allow TO-QMs and
test leaders to develop these procedures. If these procedures vary between tests, the more
appropriate documents in which to describe them may be the T/QAPs and any SOPs that may be
needed.
The QA/QC requirements section of the GVP typically describes the activities that verify the
quality and consistency of the work and provides data quality descriptors, such as accuracy,
precision, representativeness, completeness, comparability, and detection limit, as appropriate.
Preparation and use of appropriate QA procedures (such as QC samples, blanks, split and spiked
samples, and PE samples) to verify performance of the technology being tested can be described.
Frequency of calibrations and QC checks and the rationale for them can be described. Procedures
for reporting QC data and results can be given. Who is responsible for each QA activity, and
who has the responsibility for identifying and taking corrective action can be specified.
However, if these items vary between tests within a given technology, the more appropriate
document in which to describe them may be the T/QAP.
The GVP may cite documents or procedures (e.g., documentation of related procedures, the
published literature, or methods manuals) that explain, extend, or enhance the GVP. The specific
location of any reference not readily available from a full citation in the reference section should
be given (as in a facility-specific SOP) or attached to the GVP.
3.1.2 Test/Quality Assurance Plans
T/QAPs are specific for one or more very similar verification tests that are conducted by a
particular TO under a particular GVP. These plans address all emissions and process data that
will be gathered in the verification test and include a project description, TO management
organization and responsibilities, DQI acceptance criteria, site selection and sampling and
monitoring procedures, analytical procedures and calibration, data reduction and reporting, QC
checks, technical assessments, and calculations. Data quality will be pre-eminent. All QA
requirements must be met, including the requirements for review and approval of the T/QAP by
the director, the QM, and the EPA PO AND EPA QM.
T/QAPs for the APCT Center must be developed in accordance with EPA Requirements for
Quality Assurance Project Plans, EPA QA/R-5 (EPA, 200 Ib) and EPA's Guidance for Quality
Assurance Project Plans, EPA QA/G-5 (EPA, 2002a).
The required QA category and associated QAPP requirements for each will be determined by the
EPA PO at the beginning of a project, with the concurrence of the EPA QAM. Two QA
categories are applicable to verification testing preformed by the APCT VC:
Category II - establishes QAPP requirements for important, highly visible Agency projects
involving areas such as supporting the development of environmental regulations or standards;
Category III - establishes QAPP requirements for projects involving applied research or
technology evaluations
23 of 45
-------
APCT CENTER Quality Management Plan
Revision 2.3, Effective: March 2, 2010
The graded approach applies to other aspects of the quality system. The following table
summarizes these requirements for the two different QA Categories.
QA Requirement
QAPP Preparation
Technical Systems Audits
Audits of Data Quality**
Category II
EPA R-5
Required for each project
Required (25% of the data sets)
Category III*
NRMRL QAPP Requirements
Two Category III/IV projects/year
Not Required
* Additional sets of requirements may also be required depending on the nature of the project.
**If problems are found, all data sets will be audited.
The T/QAP will include the following elements, where appropriate, for the specific verification
test and as applicable for the QA Category designation. It must note and explain those elements
that are not appropriate for the test:
• Title and approval sheet - Include title of plan, name of the TO, and the names, titles, and
signatures of appropriate approving officials, and their approval dates.
• Table of contents and distribution list - List sections, figures, tables, references, and
appendices. List all the individuals who will receive copies of the approved T/QAP.
• TO personnel and responsibilities - Identify the individuals participating in the test and
discuss their specific roles and responsibilities. Ensure that the TO-QM is independent of
the administrative unit generating the data. Provide a concise organizational chart
showing the relationships and the lines of communication among all participants,
including subcontractors.
• Schedule - The anticipated start and completion dates for the project should be given. In
addition, this discussion should include an approximate schedule of important project
milestones, such as the start of environmental measurement activities.
• Verification test description and test objectives - State the specific problem to be solved
or decision to be made and include sufficient background information to provide a
historical and scientific perspective for this particular project.
• Identification of the critical measurements, DQI acceptance criteria for critical
measurements, verification test schedule, and milestones - Describe the relationship
between DQI acceptance criteria and DQOs specified in the GVP. EPA requires the use
of a systematic planning process to define DQOs and DQI acceptance criteria.
24 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
Documentation and records management - Describe the process for ensuring that TO
personnel have the T/QAP. Itemize the information and records that must be included in
the data report package and specify the desired reporting format for hardcopy and
electronic forms, when used. Identify any other records and documents applicable to the
project, such as assessment reports and verification reports, that will be produced.
Specify or reference all applicable requirements for the final disposition of records and
documents, including location and length of retention period.
Experimental design - Describe the experimental design or data collection design for the
verification test. Classify all measurements as critical or noncritical.
Sampling procedures - Describe the procedures for collecting samples and identify the
sampling methods and equipment. Describe the process for preparing and
decontaminating sampling equipment.
Sample handling and chain-of-custody procedures - Describe the requirements and
provisions for sample handling and custody in the field, laboratory, and transport.
Include examples of sampling documentation.
Analytical procedures - Identify the analytical methods and equipment required. Where
appropriate, the methods can be identified by method number, date, and regulatory or
literature citation. List any method performance standards. For nonstandard method
applications, appropriate method performance study information must be presented or
cited. If previous performance studies are not available, they must be developed during
the verification testing and included as part of the project results.
Test-specific procedures for assessing DQI acceptance criteria - Identify required
measurement QC checks for both the field and the laboratory. State or reference the
required control limits for each QC check and corrective action required when control
limits are exceeded and how the effectiveness of the corrective action shall be determined
and documented. Describe or reference the procedures to be used to calculate each of the
QC acceptance criteria.
Instrument calibration and frequency - State the frequency of each type of QC check.
Identify the certified equipment and/or standards used for calibration. Describe or
reference how calibrations will be traceable to nationally recognized performance
standards. If no such nationally recognized standards exist, document the basis for the
calibration. Indicate how records of calibration shall be maintained and be traceable to
the instrument.
Data acquisition and data management procedures - Identify and describe all data-
handling equipment and procedures to process, compile, and analyze the data. Describe
the data management scheme, tracing the path of the data from their generation to their
final use or storage. Describe record-keeping procedures. Discuss the control mechanism
for detecting and correcting errors and for preventing loss of data.
Self-assessments of quality and technical systems and ADQs - Identify the number,
frequency, and type of assessment activities. Describe how and to whom the results of
25 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
the assessments shall be reported. Define the scope of authority of the auditor, including
stop-work orders.
• Corrective action procedures in response to technical assessment findings - Discuss how
corrective actions will be developed and implemented by the TO in response to
assessment findings. Include details on how the corrective actions will be verified and
documented.
• Test status reports and assessment reports - Identify the frequency and distribution of
reports issued to inform management of the status of the test, assessment findings, and
other significant QA problems and recommended solutions. Identify the preparer and
recipients of the reports and the actions that recipients are expected to take as a result of
the reports.
• Data reduction, data review, data validation, and data reporting - State the criteria used to
review and validate (i.e., accept, reject, or qualify) data in an objective and consistent
manner. Describe the process to be used for validating and verifying data, including the
chain of custody for data. Discuss how issues shall be resolved. Describe how the results
are conveyed to the data users.
• Reporting of DQIs for critical measurements and reconciliation of verification test data
with DQOs - Describe how the data will be reconciled with the DQOs. Outline the
proposed methods to analyze the data and determine possible anomalies or departures
from assumptions established in the planning phase of the verification test. Describe how
the reconciliation process will be documented.
• Limitations of the data - Describe how issues will be resolved and discuss how
limitations on the use of the data will be reported.
Although the approved T/QAP must be implemented as prescribed, it is not inflexible. Because
of the complex and diverse nature of ETV Program verification testing, changes to already
approved plans are often needed. When such changes occur, the QM will determine if the
change significantly impacts the technical and quality objectives of the project. When a change
is significant, the test leader, with the assistance of the TO-QM, will modify the plan to
document the change and submit the revision for approval by the EPA PO AND EPA QM. The
change can be implemented only after the revised plan has been approved.
DQIs are qualitative and quantitative measures of principle quality attributes. They are used in
T/Q APs for specifying requirements for the acceptability or utility of measurement data.
Historically, DQIs sometimes have been incorrectly equated with DQOs, which are
specifications for decision-making. In the ETV Program, DQOs are specified by a technical
panel and are documented in the GVP. DQIs and DQI acceptance criteria are developed by a TO
to allow it to determine from QC checks during a verification test whether the verification test
data will attain the DQOs at the completion of the test.
While DQOs state what the verification test data user's needs are, they do not provide sufficient
information about how these needs can be satisfied. The TO staff who will participate in
generating the data need to know the DQI acceptance criteria that must be satisfied to attain the
26 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
DQOs. One of the most important features of the T/QAP is that it links the DQOs with DQI
acceptance criteria. Although the level of rigor with which this is done and documented will
vary widely depending on the technology being tested and the measurement systems involved,
establishing this linkage in the T/QAP represents an important advancement in the
implementation of the APCT Center quality system.
The establishment of acceptance criteria for the DQIs sets quantitative goals for the quality of
data generated in the verification tests. For the quantitative measurement parameters, the T/QAP
must describe test-specific measurement and calculation procedures for assessing DQIs.
Acceptance criteria for DQIs should be consistent with the DQOs in the GVPs and, where
possible, acceptance criteria for quantitative DQIs should be derived from quantitative DQOs.
The six principal DQIs that are related to environmental measurements and sampling are
precision, accuracy, representativeness, comparability, completeness, and sensitivity. Secondary
DQIs include selectivity, recovery efficiency, memory effects, limits of quantitation,
repeatability, and reproducibility.
EPA's Guidance on Data Quality Indicators, EPA QA/G-5i (EPA, 2007a), provides more
information about how to prepare DQIs and how to establish them in the context of DQOs.
3.1.3 Standard Operating Procedures
If another level of detail beyond the T/QAP is needed for describing test activities, technical
SOPs must be written by the TO. The SOPs must be prepared and approved before the start of
the verification test. They must be available for review by the director, the QM, and EPA as part
of the review and approval process for T/QAPs. They must also be available for review during
self-assessments and independent assessments. If requested by a TO, the APCT Center and EPA
will hold confidential SOPs that are supplied by the TO for review.
A technical SOP is a set of written instructions that document a routine or repetitive activity
followed by a TO. The development and use of SOPs are integral parts of a successful quality
system because they provide testing personnel with the information needed to perform a job
properly, and they facilitate consistency in the quality and integrity of verification test data.
SOPs describe both technical and administrative operational elements of a TO that would be
managed under a T/QAP. EPA's Guidance for Preparing Standard Operating Procedures
(SOPs), EPA QA/G-6 (EPA, 2007b) provides more information about how to prepare SOPs.
The technical SOP needs to include the specific steps aimed at initiating, coordinating, and
recording or reporting the results of the activity, and it should be tailored only to that activity.
Cited published methods may not contain pertinent information for conducting the procedure in-
house. The SOP should fit within the framework presented here, but this format can be
modified, reduced, or expanded as required. The technical SOP should address three areas:
1. Procedural Area - This section can include the following items as appropriate for the
verification test:
• Scope and applicability (describing the purpose of the process or procedure and any
organizational or regulatory requirements);
27 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
• Summary of method (briefly summarizing the procedure);
• Definitions (acronyms, abbreviations, and specialized forms used in the SOP);
• Health and safety warnings (indicating activities that could result in possible personal
injury), including medical assistance and site evacuation plans;
• Cautions (indicating operations that could result in personal injury or loss of life and
explaining what will happen if the procedure is not followed or is followed incorrectly;
warnings should also be listed at the critical steps in the procedure);
• Interferences (describing any component of the process that may interfere with the
accuracy of the final product);
• Personnel qualifications (denoting the minimal experience the SOP follower should have
to complete the task satisfactorily);
• Equipment and supplies (listing and specifying, where necessary, equipment, materials,
reagents, chemical standards, and biological specimens); and
• Procedure (identifying all pertinent steps, in order, and materials needed to accomplish
the procedure). For example:
« Instrument or method calibration and standardization;
« Sample collection;
« Sample handling and preservation;
Sample preparation and analysis (such as extraction, digestion, analysis, identification
and counting procedures);
« Troubleshooting;
« Data acquisition, calculations and data reduction requirements, such as listing any
mathematical steps to be followed;
« Computer hardware and software (used to manipulate analytical results and report
data); and
« Data and records management (e.g., identifying any forms to be used, reports to be
written, and data and record storage information).
2. QC and QA Area - QC procedures are designed to allow the evaluation of the quality and
consistency of the verification test results by the TO. Examples of QC procedures
include instrument calibrations and QC materials (e.g., blanks, split and spiked samples,
and PE samples) to verify the performance of the technology being tested. Criteria for
determining success can be included in the SOP, as appropriate. The frequency of
required calibrations and QC checks and the rationale for the decision regarding the
frequency can be described. Acceptance criteria for DQIs and corrective actions required
when these criteria are exceeded can be described.
28 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
The SOP can specify and describe any QA procedures that are integral parts of the
verification test, including self-assessments and independent assessments (e.g., TSAs,
PEs, and ADQs). It can specify who or what organization is responsible for each QA
activity and where or how PE samples are to be procured and/or verified.
3. Reference Area - The SOP should fully reference related documents or procedures, such
as related SOPs, published literature, and methods manuals. However, such reference
citations cannot substitute for the thorough description of the method being followed in
the TO. All references that are noted in the SOP should be fully cited, and references that
are not readily available should be attached to the SOP.
As with the technical SOPs, administrative SOPs can be written for a wide variety of activities
(e.g., reviewing documentation such as contracts, T/QAPs and QMPs; inspecting the work of
others; determining organizational training needs; developing information on records
maintenance; validating data packages; or describing office correspondence procedures). The
administrative SOP needs to include a number of specific steps aimed at initiating the activity,
coordinating the activity, and recording or reporting the results of the activity. For example, an
assessment SOP should specify the authority for the assessment, technical criteria for
assessments, what will be done with the results, and who is responsible for developing and
implementing corrective action. Administrative SOPs for APCT would be written by RTI staff
under the direction of the APCT director.
Technical SOPs are the responsibility of test leaders, who manage the SOPs according to their
laboratory quality systems. Administrative SOPs are the responsibility of the APCT director,
they are numbered consecutively, maintained in the APCT folder on the RTI server, and handled
as other APCT records. The ESE administrative SOPs are available on the RTI intranet and will
be used as applicable for APCT.
3.2 Design of Verification Tests
Data collection operations will be performed during the verification tests in conformance with
the GVPs and test-specific T/QAPs. These operations must be designed so that one can
determine whether the DQOs for the verification tests have been attained. Specific
considerations regarding the design of data collection operations will be addressed in the T/QAP
for each verification test. EPA has established an Existing Data policy (EPA ETV QMP
Appendix C) which much be complied with for APCT.
3.2.1 Intended Use of Data
The key use of information generated by verification tests is to prepare verification statements,
which are independent and credible third-party assessments of the environmental performance
characteristics of commercially ready technologies through the evaluation of objective and
quality-assured data. These statements will include information such as pollutant emissions
removal efficiency rates and energy consumption rates as a function of operating conditions. The
data thus generated may be used for a variety of purposes, including regulatory compliance
decisions. For this reason, it is important that the data be reliable, defensible, and of known
quality. Key acceptance criteria are described in the next section.
29 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
The APCT Center will not disseminate interim results from verification tests. As such, it will not
be necessary to identify and state restrictions on the use of interim results.
3.2.2 Project Requirements for Data Quality Objectives
APCT Center personnel and TO personnel will establish quantitative DQOs in the GVPs with
input from technology-specific technical panels. These DQOs address the end use of the data
and the data quality required for stakeholder decisions that are based on those data. Thus, the
quantitative DQOs address the required quality in terms of tolerable limits for the uncertainty of
measurement data from the verification tests. Examples of DQOs include the types of
measurements to be made, the critical variables applicable to each technology, and the degree of
uncertainty permissible in the verification statement.
During the TO's development of its T/QAP, the higher level DQOs are broken down into
component parts in order to derive quantitative DQI acceptance criteria (e.g., precision,
accuracy, representativeness, completeness, and comparability) for each critical measurement.
DQIs address the requirements that are placed on the nuts-and-bolts aspects of the sampling and
measurement systems, which is why they are specified in the T/QAP. Specific procedures for
determining attainment of DQI acceptance criteria will be listed in the T/QAP. The T/QAP will
be developed by the test leader with the assistance of the TO-QM and the QM. It will be
reviewed by the director, the QM, and the EPA QM. Finally, it must be approved by the EPA
PO before the verification test begins.
3.2.3 Performance Characteristics for Measurement Methods
Each T/QAP is developed for a verification test of a particular technology by a particular TO. It
will list all critical measurements to be made during that test, the procedures to be used for these
measurements, and the performance characteristics of the measurement procedures. The APCT
Center has extensive experience in developing and verifying new and/or nonstandard
environmental measurement methods. Before any new or nonstandard measurement method is
used in a verification test, its performance characteristics will be determined over the expected
range of test conditions. If previous performance studies for this method are not available,
studies will be developed before the verification test and will be included as part of the T/QAP.
The types and frequencies of calibrations and the QC samples that are necessary to track a new
method's performance will be determined during method validation and specified in the
method's SOP.
3.2.4 Use of Accepted Analytical Procedures
Appropriate, approved procedures will be used for sampling and analysis and for method
development and evaluation, when available. EPA-approved methods of sampling and analysis
will be used whenever possible. The required quality and consistency of these methods and of
new or nonstandard methods will be specified in the T/QAP. These methods will be fully
described in the T/QAP and in any SOPs that are needed. The APCT Center will also employ
standard, recognized statistical and data assessment methods.
30 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
3.2.5 Instrument Calibration
Calibration of an analytical method establishes the quantitative relationship between the quantity
of the analyte (e.g., in concentration units of parts per million) and the method's response (e.g.,
in volts). This relationship is used to convert subsequent method responses into the
corresponding analyte quantity. Because the response of many methods has the tendency to drift
with time, the calibration must be checked periodically to maintain a high degree of accuracy.
Sampling and analytical equipment will be calibrated in accordance with the GVP, the T/QAP,
and any needed SOPs, and with the sampling or analytical method for which the equipment is
used. The frequency of calibration depends upon the type of equipment, the particular compound
or element being measured, and the concentration level(s) of the compound or element. Analysts
will evaluate instrument performance characteristics such as span drift, zero drift, noise, and
linearity.
Analyses must fall within the calibrated range of the instrument. Samples will be screened so
that calibration standards are appropriate for the anticipated measurement range of the samples.
Calibration standards will be traceable to national standards, such as National Institute of
Standards and Technology (NIST) Standard Reference Materials (SRMs) or NIST-traceable
weights, when such standards are available. The stability of calibration standards will be
monitored through the use of independently prepared samples that are analyzed with the
calibration standards.
Calibration documentation will be maintained with each method in a central location and will be
readily available for review by assessors and TO management. It will include such information
as the instrument being calibrated, raw calibration data, calibration equations, analyzer
identifications, calibration dates, certification periods, analyzer locations, calibration standards
used and their traceabilities, identification of calibration equipment used, and personnel
conducting the calibration.
3.2.6 Sample Collection, Selection, Preparation, and Site Selection
EPA sampling and site selection protocols, where available and appropriate, will be followed for
all verification tests. When testing is conducted by RTI or by a TO under RTFs supervision, any
needed sampling SOPs will be established before testing begins and will be attached to the
T/QAP. If sampling is performed by an unsupervised TO, adequate documentation of samples
and the assessment and evaluation of the sampling procedures used to generate the data will be
required.
Sample selection will be performed according to an approved experimental design that specifies
the number and types of samples to be taken to achieve the DQOs and the specified DQI
acceptance criteria described above. Where appropriate, statistical techniques will be used to
develop the experimental design. The TO will ensure that all qualitative criteria for sampling
representativeness are met so that valid, unbiased test results are obtained. Proper sample
collection and statistical sampling techniques will be used where necessary to avoid sampling
bias.
31 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
3.2.7 Sample Handling, Tracking, Custody, Transportation, and Storage
To ensure that verification test samples and data are both secure and traceable, test leaders and
TO-QMs will develop written SOPs for sample handling, sample tracking, and chain of custody.
These SOPs will be applicable to specific verification tests and will be documented in the
T/QAPs for those tests. They will provide appropriate protection against inadvertent loss of
samples or data, will ensure that verification test data generated by the TO are traceable, and will
protect participants' proprietary information. A graded approach will be used in these
procedures such that documentation and security levels are commensurate with the intended use
of the verification test data and the degree of confidence needed in the quality of these data. A
general discussion of sample handling and custody is given in EPA's Guidance for Quality
Assurance Project Plans, EPA QA/G-5 (EPA, 2002a).
The written sample custody procedures in the T/QAPs will include the following elements, as
appropriate, for the specific verification test:
• List the names and responsibilities of all sample custodians in the field and in
laboratories;
• Give a description and example of the sample numbering system;
• Define acceptable conditions (e.g., sample preservation, temperature, transit time, etc.)
and plans for maintaining sample integrity in the field prior to and during shipment to the
laboratory;
• Give examples of sample log sheets, chain-of-custody forms, and sample labels that will
be used to maintain sample custody and to document sample handling in the field and
during shipping;
• Describe the method of sealing shipping containers with chain-of-custody seals to detect
tampering;
• Describe procedures that will be used to maintain the chain of custody and to document
sample handling during transfer from the field to the laboratory, within the laboratory,
and among test laboratories;
• Provide for the archiving of all shipping documents and associated paperwork;
• Discuss procedures that will ensure sample security at all times;
• Describe procedures for within-laboratory chain of custody together with verification of
the printed name, signature, and initials of persons who are responsible for custody of
samples, extracts, or digests during analysis at the laboratory; and
• Describe procedures to document the disposal or consumption of samples.
32 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
Less rigorous documentation of chain-of-custody procedures will be used when:
• Samples are generated and immediately tested within a facility or site; and
• Samples are continuous, rather than discrete or integrated samples, and are subjected to
real-time or near-real-time analysis (e.g., continuous monitoring).
A sample custodian will be designated by the TO for any verification test involving large
numbers of samples. The sample custodian performs sample-receiving inspection, physical
acceptance of a group of samples intended for subsequent treatment or analysis, analysis
tracking, and/or sample repository operation.
Necessary preservation techniques will be used for all perishable environmental samples as
provided in the applicable methods. Perishable samples will be shipped in coolers with frozen
ice-substitute gel packs when temperature control is necessary.
3.3 Implementation of Planned Operations
The verification testing will proceed after the T/QAP has been approved by the EPA PO and
EPA QM. Verification tests will be conducted at locations appropriate for the technology being
verified, as established by the GVP. A full-scale installation at an industrial facility or a large-
scale pilot device operated on a slipstream may be required, or a pilot facility with a source
simulator may suffice. Although cost control is important, the most important requirement for a
test site is that the data obtained be convincing. The wide variety of facilities available to the
APCT Center will ensure that the most suitable and cost-efficient location will be chosen,
consistent with the GVP requirements. If no suitable facility is available to the APCT Center,
another site will be chosen in consultation with the technical panel and the EPA PO.
Verification tests will be performed in accordance with the T/QAPs. Test leaders will be
responsible for implementing the T/QAPs and any needed SOPs, and they will periodically
report on the progress of the verification tests to the director. Their progress toward attaining the
DQOs will be most directly monitored by the TO-QMs. The QM, under the direction of the
director, will be responsible for independent assessments of TO quality systems and technical
assessments of verification tests. Such technical assessments may include TSAs and PEs at the
test site during verification tests. TO-QMs will conduct self-assessments of the quality and
technical systems and ADQs after verification tests have been completed.
Verification tests will be conducted according to the following requirements:
• All critical items and services used during the verification tests will conform to
specifications given in the GVPs, T/QAPs, and any needed SOPs.
• All sampling, measurement, and analytical instrumentation and other measurement
systems used for critical measurements during the verification tests will be checked to
determine whether they comply with DQI acceptance criteria given in the T/QAPs.
33 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
• Whenever the measurement systems do not meet the DQI acceptance criteria specified in
the T/QAPs, corrective actions will be taken before any further measurements are made
to return the measurement systems' performance to acceptable levels.
• Measurement systems will be maintained and repaired in accordance with specifications
given in the GVPs, T/QAPs, and any needed SOPs.
• All tools, gauges, and any other ancillary sampling, measuring, and testing equipment
used for critical measurements during verification tests will be maintained and repaired in
accordance with specifications given in the GVPs, T/QAPs, and any needed SOPs.
Equipment found to be out of specification will not be used until it can be recalibrated or
repaired and then demonstrated to be functioning within the specifications.
• All measurements and other verification activities will be documented in laboratory
notebooks, field and laboratory data sheets, spreadsheets, computer files, and other
appropriate storage media.
• Field and laboratory samples will be collected, handled, transported, and stored in
accordance with specifications given in the GVPs, T/QAPs, and any needed SOPs. The
chain-of-custody process will be maintained in accordance with specifications given in
these documents.
• Verification test data will be transmitted, stored, validated, assessed, processed, and
retrieved in accordance with specifications given in the GVPs, T/QAPs, and any needed
SOPs.
3.4 Assessment and Response
3.4.1 Frequency, Types, and Reporting of Assessments
Sections A9.1 and B4.2 of EPA's Environmental Technology Verification Program Quality
Management Plan (EPA, 2008) specify that EPA will conduct one independent assessment of the
APCT Center's quality system. Additional assessments of APCT Center's quality system will be
at a frequency determined by the EPA QM's professional judgment, using the guidelines
described in Table 3.
The QM will perform a self-assessment of the APCT Center quality system on an annual basis in
conjunction with the annual review and revision, as needed, of this document. This assessment
frequency is specified in EPA Requirements for Quality Management Plans, EPA QA/R-2 (EPA,
200 la). A final report of the assessment findings will be prepared and will be presented to the
EPA PO, the EPA QM, and the director.
The QM will perform at least one independent assessment of each TO's quality system. These
assessments will be conducted in accordance with EPA's Guidance on Assessing Quality
Systems, EPA QA/G-3 (EPA, 2003). The frequency of additional independent assessments of the
quality system will be determined by the QM in consultation with the EPA QM, and will be
based on their professional judgments. Factors that will be considered in making this
determination include the state of the TO's quality system during previous independent
34 of 45
-------
APCT CENTER Quality Management Plan
Revision 2.3, Effective: March 2, 2010
Table 3. Schedule of Assessments
Assessment
Type
Quality
systems audit
Quality
systems audit
Self Quality
systems audit
Technical
systems audit
Performance
evaluation
audit
Audits of data
quality
Assessment
Subject
APCT Center
quality system
TO quality
system11
TO quality
system11
TO
measurement
systems11
TO
measurement
systems11
Raw test data
and summary
data
Assessment
Authority
Approved
APCT Center
QMP (this
document)
Approved TO
quality
manual*
Approved TO
quality
manual*
Approved
T/QAP
Approved
T/QAP
Approved
T/QAP
Assessor
APCT
Center QM
APCT
Center QM
TO QM
APCT
Center QM
APCT
Center QM
TOQM
Minimum
Frequency
Within one
year after
approval of
the APCT
QMP, then
as
requested
by EPA
Prior to data
collection
and as
specified in
the
approved
T/QAP
Prior to
APCT QSA
of TO
Once per
test, unless
lower
frequency is
pre-
approved by
the EPA QM
Each test, if
feasible and
as specified
in the
approved
T/QAP
Each test
Assessment Purpose
Evaluate effectiveness of APCTVC
quality system with respect to QMP
Evaluate effectiveness of TO
quality system11
Evaluate effectiveness of TO
quality system11
Evaluate compliance of TO
measurement system with
approved T/QAP
Evaluate compliance of TO
measurement system with
approved T/QAP
Evaluate compliance of achieved
measurement quality with respect
to approved T/QAP and assess
data calculations at a rate of at
least 1 0%
Report
Recipient
APCT
Center
director
APCT
Center
director
EPAPO
EPAQM
TOPM
TOQM
TO
director,
copy to
APCT
center
director
APCT
Center
director
EPAPO
EPAQM
TO PO
TOQM
APCT
Center
director
EPAQM
APCT
Center QM
EPAQM
Note: This table identifies APCT Center-initiated assessments. We plan to coordinate with EPA staff to facilitate their awareness
and participation, if desired. QM = quality manager; TO = testing organization.
* Regardless of title (e.g., quality manual, quality systems manual).
HWith respect to Environmental Technology Verification Program/APCT Center responsibilities.
assessments and self-assessments, the level of verification testing by the TO, and the quality of
the verification tests that the TO has performed. The QM and the EPA QM may also determine
that an additional assessment is needed if there has been a significant change to the quality
system. In instances in which corrective action is needed to address findings from a previous
assessment, followup assessments may be necessary. A final report of the assessment findings
will be prepared and will be presented to the EPA PO, the EPA QM, and the director.
35 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
TO-QMs will perform self-assessments of the TOs' quality system at the frequency specified in
the quality documentation of the TO. Final reports of self-assessments will be available for
review during independent assessments.
The QM will perform at least one independent technical assessment of each TO's technical
systems. These assessments will be conducted in accordance with EPA's Guidance on Technical
Audits and Related Assessments for Environmental Data Operations, EPA QA/G-7 Final (EPA,
2000). In instances in which a TO is operating under multiple GVPs or multiple T/QAPs,
multiple independent technical assessments may be necessary. The frequency of additional
independent technical assessments will be determined by the QM in consultation with the EPA
QM based on their professional judgments. Factors that will be considered in making this
determination include the state of the TO's technical systems during previous independent
assessments and self-assessments, the level of verification testing by the TO, and the quality of
the verification tests that the TO has performed. The QM and the EPA QM may also determine
that an additional assessment is needed if there has been a significant change to the technical
systems, such as one that requires the revision of a previously approved T/QAP. In instances in
which corrective action is needed to address findings from a previous assessment, followup
assessments may be necessary. A final report of the assessment findings will be prepared and
will be presented to the EPA PO, the EPA QM, and the director.
TO-QMs will perform self-assessments of the TOs' technical systems at the frequency specified
in the TO's T/QAP. Final reports of self-assessments will be available for review during
independent assessments.
TO-QMs will perform ADQs for a random selection of 10 percent of all the verification data for
every verification test in accordance with Sections A9.1 and B4.2 of Environmental Technology
Verification Program Quality Management Plan (EPA, 2008). Final reports of the ADQs will be
prepared and will be submitted with the verification report to the EPA PO, the EPA QM, the
APTC Center director, and the APTC Center QM.
3.4.2 Assessments of Quality Systems
A quality system assessment (QSA) is an evaluation of an organization's relevant quality system
practices. The focus of the assessment is on the organization's quality system rather than on its
technical systems or the quality of data and information the organization produces to support a
verification statement. Assessments are designed to evaluate and provide objective feedback on
the organization's quality system. An assessment seeks to determine if the system has been fully
implemented and is operating in the manner prescribed by the organization's approved QMP and
if it is consistent with current EPA quality requirements. An assessment determines if an
organization's QMP, quality management structure, policies, practices, and procedures are
effective as implemented in assuring that environmental data have adequate quality for their
intended purpose.
The criteria for assessments of the APCT Center's quality system will be this document. The
criteria for assessments of a TO's quality system will be its own quality system documents.
These assessments will be conducted in accordance with EPA's Guidance on Assessing Quality
Systems, EPA QA/G-3 (EPA, 2003).
36 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
The process of assessing a quality system includes the following four stages:
1. Planning the Assessment - The scope of the assessment must be determined. The scope
may be very broad because the EPA PO may be interested in assessing how well the
APCT Center's quality system is being applied in general. However, the EPA PO may
also have specific questions that need to be addressed. Next, an assessment team will be
identified. Team members should have sound interviewing skills, no conflicts of interest,
competency in the technical fields being reviewed, and a thorough knowledge of quality
management principles. During the planning stage, the team identifies the technical
criteria (e.g., its QMP) and other background information for assessing the organization's
quality system. The team will then determine what interviews and document reviews are
necessary to properly assess the quality system. After the team has compiled and
reviewed this information, it will develop a draft assessment plan, which the EPA PO will
review.
2. Conducting the Assessment - Upon arrival at the assessment site, the assessment team
first will meet with the organization's management and key personnel. Next, the team
will interview management and key personnel as noted and scheduled in the assessment
plan. The team also may review relevant files and consider case studies. The team will
present its initial impressions of the information that it gathered during an exit meeting.
3. Evaluating the Results - The assessment team will assemble and review the information
gathered during the assessment and will evaluate it based on the technical criteria
documented in the assessment plan. The team then will formulate preliminary findings
and recommendations in a written draft findings report.
4. Reporting the Findings and Recommendations - The draft findings report will be sent to
the assessed organization for review. This step is important to ensure that there are no
factual errors in the final report. The assessed organization's comments on the draft
findings report will be reconciled by the assessment team. A final report of the
assessment findings will be prepared by the lead assessor.
3.4.3 Technical Assessments
A technical assessment is a systematic and objective examination of a verification test during its
implementation phase to determine whether environmental data collection activities comply with
the T/QAP, whether they are implemented effectively, and whether they are suitable to achieve
DQOs that have been specified in the GVP and DQI acceptance criteria that have been specified
in the T/QAP.
A technical assessment is primarily a management tool and secondarily a technical tool.
Technical assessments play an important role in documenting the implementation of the T/QAP.
They provide management a tool to determine whether data collection activities are being
implemented as planned. They provide management with both an increased understanding of the
performance of critical measurements and a basis for improving such measurements. They also
provide management a tool to take action to correct any deviations that are discovered.
37 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
The criteria for a technical assessment of a TO will be the TO's T/QAP. The TO documents the
implementation of the T/QAP. Proper use of technical assessments ensures that collected
environmental data are defensible. They can uncover deficiencies in physical facilities,
equipment, planning, training, operating procedures, technical operations, custody procedures,
documentation, and QA.
Guidance for conducting technical assessments is given in EPA's Guidance on Technical Audits
and Related Assessments for Environmental Data Operations, EPA QA/G-7 Final (EPA, 2000).
Technical assessment tools include the following:
• TSAs qualitatively document the degree to which the procedures and processes specified
in the approved T/QAP are being implemented.
• PEs quantitatively document the ability of a measurement system to obtain acceptable
results that are generated for a sample that originates outside of the verification test.
• ADQs examine verification test data (hardcopy and/or electronic) after they have been
collected and verified by TO personnel.
• Surveillance assessments are used to continuously or periodically assess the
implementation of an activity or activities to determine conformance to established
procedures and protocols. Surveillance assessments include the APCT review of TO
ADQ reports and supporting documentation.
At least one TSA will be conducted for each TO. The number of PEs that will be performed
during a verification will be indicated in the T/QAP. The TO-QM will conduct an ADQ of a
random selection of 10 percent of the data for all measured parameters at the end of each
verification test in accordance with the requirements of Sections A9.1 and B4.2 of
Environmental Technology Verification Program Quality Management Plan (EPA, 2008). The
TO-QM will determine if these measurements allow attainment of the DQOs specified in the
GVP and the DQI acceptance criteria specified in the T/QAP.
Technical Systems Audits. TSAs are thorough, systematic, qualitative, onsite assessments of
the TO's measurement systems used to collect data. They also determine that TO personnel and
equipment are physically in place and functioning as stated in the planning documents. Assessors
travel to the verification site, gather objective evidence, and produce a written findings report.
Objective evidence is gathered by interviewing TO personnel, examining verification documents
and records, and observing verification activities.
A TSA is often conducted shortly after a verification test starts to allow for early corrective
action. For longer tests, TSAs are performed on a regular schedule throughout the life of the test.
TSAs may be performed in conjunction with PE assessments.
Assessment checklists are prepared based on the T/QAP and other planning documents, which
are the technical criteria for the assessment. Any undocumented or unauthorized deviation from
the T/QAP will be noted during the TSA and will be included in the written assessment report.
38 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
The draft findings report will be sent to the assessed organization for review. This step is
important to ensure that there are no factual errors in the final report. The assessed
organization's comments on the draft findings report will be reconciled by the assessment team.
A final report of the assessment findings will be prepared by the lead assessor.
Performance Evaluations. A PE is a quantitative assessment in which data are generated by a
measurement system for a sample whose composition is known to the assessor. Alternatively, an
assessor may make collocated measurements using equipment of documented high quality.
Normally PEs are performed as part of a TSA and are performed by RTI and/or EPA. Although a
PE can identify a problem quantitatively, it typically cannot determine the cause of the problem.
To the extent possible, a PE sample will not be distinguishable in any way to the measurement
system from actual samples. It will mimic actual samples in all possible aspects, except that its
composition will be unknown to the analyst and known to the assessor. It will be treated
routinely and not subjected to any special treatment. It will be used to determine if the
measurement system's results are within the DQI acceptance criteria specified in the T/QAP. PE
results will be used to estimate the degree of bias in the measurement system.
A draft report of the PE findings will be sent to the assessed organization for review. This step is
important to ensure that there are no factual errors in the final report. The assessed
organization's comments on the draft report will be reconciled by the assessment team. A final
report of the PE findings will be prepared by the lead assessor.
3.4.4 Audits of Data Quality
An ADQ is an examination of data (hardcopy and/or electronic) after they have been collected
and verified by TO personnel. It is conducted to determine how well the measurement system
performed with respect to the DQOs specified in the GVP and DQI acceptance criteria specified
in the T/QAP, and to determine whether the data were accumulated, transferred, reduced,
calculated, summarized, and reported correctly. It documents and evaluates the methods by
which decisions were made during treatment of the data. It ensures that files are being
maintained and secured, that chain-of-custody records are complete, and that raw data records
are complete and in good order. Computer security will also be reviewed.
Questions to be answered in an ADQ include:
• Is there sufficient documentation of all procedures used in the data collection effort to
allow for repetition of the effort by a person or team with technical qualifications similar
to those of the original data collector?
• Can the data be replicated by the original data collector?
• Is there sufficient documentation to verify that the data have been collected and reported
according to these procedures?
• Is enough information provided to allow a potential user to determine the quality and
limitations of the data and whether the intended use of the data is appropriate?
39 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
• Are the data of sufficient quality to attain DQOs, DQI acceptance criteria, and other
measurement performance criteria?
ADQs entail tracing data through their processing steps and duplicating intermediate
calculations. A representative set of the data is traced in detail from raw data and instrument
readouts through data transcription or transference through data manipulation (either manually or
electronically by commercial or customized software) through data reduction to summary data,
data calculations, and final reported data. The focus is on identifying a clear, logical connection
between the steps. Particular attention is paid to the use of QC data in evaluating and reporting
the data set. For a large project, a statistical approach may be necessary to determine a
representative number of data sets to be examined.
A typical ADQ will begin by reviewing available data from a verification test, by determining
needed missing data, and by devising a plan for the assessment. The plan will usually include
steps involving pursuing all available needed data, collecting it, and conducting an extensive
review of the entire collection. Verification data will be statistically analyzed to determine
whether they can be used to obtain a valid estimate of the DQIs. Statistical analyses of data will
include, at a minimum, the average value of each parameter and its estimated uncertainty. When
necessary, APCT Center or TO statisticians can provide more advanced data evaluations.
The TO-QM prepares a report that details the results of chain-of-custody tracing, a study of data
transfer, recalculations, a review of QA data, a study of project incidents that resulted in lost
data, and a review of study statistics. The report states whether these measurements allow
attainment of the DQOs specified in the GVP and the DQI acceptance criteria specified in the
T/QAP. The ADQ report ends with conclusions about the quality of the data and their fitness for
their intended use. The ADQ report is submitted with the verification report to the director, the
QM, the EPA PO, and the EPA QM for review.
3.4.5 Corrective Action
Any member of the APCT staffer TO staff may initiate a corrective action at any time. The
corrective action will describe the deficiency found and identify the root cause. Corrective
actions will be tracked by APCT staff under the supervision of the APCT director and will be
handled in a timely manner relative to the severity of the problem.
After an assessment, the TO is responsible for developing, implementing, and documenting
corrective actions. It is critical that any necessary corrective action be timely and effective. In
some situations, additional assessments may be needed to verify the effectiveness of the
corrective actions. The TO will use a corrective action form to document any deficiencies that
require action and the proposed resolution. This form will include the signatures of the individual
identifying the need for corrective action and the individual responsible for implementing the
corrective action. The problem requiring corrective action, the proposed corrective action, and
the approach for evaluating the corrective action will be described.
The proposed corrective action should be reviewed by the lead assessor. This process helps
ensure that the planned actions will be effective in resolving the problem areas and deficiencies
reported by the assessment team.
40 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
The test laboratories will be responsible for the development of effective corrective actions of the
problem areas or deficiencies discovered during the assessment. They will provide a written
response to all assessment findings. Each finding will be addressed with specific corrective
action steps and a schedule to implement them. Responses to adverse findings are required
within 10 working days of receiving the assessment findings report in accordance with the
requirements of Part B, Section 4.3 of Environmental Technology Verification Program Quality
Management Plan (EPA, 2008). The corrective action should address the following:
• Measures to correct each deficiency;
• Identification of all root causes of significant deficiencies;
• Determination of the existence of similar deficiencies;
• Corrective actions to preclude recurrence of similar deficiencies;
• Assignment of corrective action responsibility; and
• Completion dates for each corrective action.
Test laboratories will implement corrective actions and provide requested evidence of correction.
Once such evidence is received, the technical assessment will be closed unless a reassessment is
planned.
3.5 Assessment and Verification of Data Usability
Data verification will be performed by the TO in accordance with the quality requirements of the
T/QAPs and any needed SOPs. As part of preparing a T/QAP, a TO will develop methods to
reconcile measurement data with DQOs. Usually, the T/QAP will specify DQI acceptance
criteria that allow TO personnel to determine during verification tests whether the DQOs will be
attained at the end of the test.
Verification test results will be evaluated to determine the completeness, correctness, and
conformance/compliance to these requirements. The goal of the data verification is to ensure
and document that the results are what they purport to be, that is, that the reported results reflect
what was actually done. When deficiencies in the results are identified, they will be documented
for review by the director, the QM, the EPA PO, and the EPA QM. Procedures for data
verification are given in EPA's Guidance on Environmental Data Verification and Data
Validation, EPA QA/G-8 (EPA, 2002c). All validated data arising from the verification tests will
be disclosed in verification reports, even if the technology did not perform to the expectations of
the technology provider.
QA/QC summaries will accompany regular progress reports for ongoing verification tests. QA
summaries will include quantitative assessments of DQIs such as bias, precision, and
completeness. Bias will be determined using personnel, equipment, and spiking material or
reference material as independent as possible from those used in the calibration of the
measurement system. Precision will be determined from replicate measurements of the same
analyte. When possible and appropriate, the analyte will be divided in the field and will be
41 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
preserved separately to assess the variability of sample handling, preservation, and storage, along
with the variability of the analytical component of the measurement system. Completeness will
be determined as the percentage of valid data out of the number of trials necessary to meet
statistical design goals. Qualitative statements about sampling representativeness and
comparability will be provided.
For verification reports and verification statements submitted to the director from a TO, adequate
QC data must be available in the submission to assess whether the DQOs and the DQI
acceptance criteria were attained. After completion of a verification test, the TO-QM will
prepare the QA section of the verification report, which will describe both qualitatively and
quantitatively the reliability and uncertainty inherent in the results. It will present the results of
all QA activities, identify any quality problems encountered, and discuss the resolution of those
problems through corrective action procedures. The QM and the EPA QM will review
verification reports and verification statements to confirm that the verification test results are
presented correctly.
The TO-QM will conduct an ADQ of a random selection of at least 10 percent of all verification
data for each verification test in accordance with the requirements of Part B, Section 4.2 of
Environmental Technology Verification Program Quality Management Plan (EPA, 2008). The
procedures for conducting an ADQ are described in Section 3.4 and in EPA's Guidance on
Technical Audits and Related Assessments for Environmental Data Operations, EPA QA/G-7
Final (EPA, 2000). As part of the ADQ, the TO-QM determines whether QC data attain DQI
acceptance criteria and reconciles measurement data with DQOs. The ADQ report is submitted
with the verification report for the director, the QM, the EPA PO, and the EPA QM to review.
3.6 Record Keeping and Data Management
GVPs will be retained by APCT Center for a period of not less than 7 years after the final
payment of the assistance agreement in accordance with the requirements of Part A, Section 5.3
of Environmental Technology Verification Program Quality Management Plan (EPA, 2008).
T/QAPs also will be retained by APCT Center for a period of not less than 7 years after the final
payment of the assistance agreement. The director is responsible for establishing procedures to
securely store these documents.
Any SOPs, any raw data (electronic and printed) collected during verification tests, and any
calculations or documents (including verification reports and verification statements) derived
from such data will be retained by TOs and/or by APCT Center for a period of not less than 7
years after the final payment of the assistance agreement in accordance with the requirements of
Part A, Section 5.3 of Environmental Technology Verification Program Quality Management
Plan (EPA, 2008). These data, calculations, and documents will be clearly identified by
verification test, date, observer/author, and originating TO. The test leader and/or the director are
responsible for establishing procedures to securely store these data, calculations, and documents.
Any project reviews and assessment reports that are generated by the TOs or by APCT Center
will be retained by the originating organization for a period of not less than 7 years after the final
payment of the assistance agreement in accordance with the requirements of Part A, Section 5.3
of Environmental Technology Verification Program Quality Management Plan (EPA, 2008).
42 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
The test leader and/or the director are responsible for establishing procedures to securely store
these project reviews and assessment reports.
Verification reports and verification statements will be retained by APCT Center for a period of
not less than 7 years after the final payment of the assistance agreement in accordance with the
requirements of Part A, Section 5.3 of Environmental Technology Verification Program Quality
Management Plan (EPA, 2008). All validated data arising from the verification tests will be
disclosed in verification reports, even if the technology did not perform to the expectations of the
technology provider. The director is responsible for establishing procedures to securely store
these verification reports and verification statements.
43 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
4.0 References
American National Standards Institute/American Society for Quality (ANSI/ASQC), 1994.
Specifications and Guidelines for Quality Systems for Environmental Data Collection and
Environmental Technology Programs, Standard E4-1994. Milwaukee, WI: Quality Press.
International Organization for Standardization (ISO), 2000. Quality Management Systems -
Requirements, ISO Standard 9001:2000.
U.S. Environmental Protection Agency, 2000. Guidance on Technical Audits and Related
Assessments for Environmental Data Operations, EPA QA/G-7 Final. EPA Publication No.
EPA/600/R-99/080. Washington, DC: Office of Environmental Information.
U.S. Environmental Protection Agency, 200 la. EPA Requirements for Quality Management
Plans, EPA QA/R-2. EPA Publication No. EPA/240/B-01/002. Washington, DC: Office of
Environmental Information.
U.S. Environmental Protection Agency, 2001b. EPA Requirements for Quality Assurance
Project Plans, EPA QA/R-5. EPA Publication No. EPA/240/B-01/003. Washington, DC:
Office of Environmental Information.
U.S. Environmental Protection Agency, 2002a. Guidance for Quality Assurance Project Plans,
EPAQA/G-5. EPA Publication No. EPA/240/R-02/009. Washington, DC: Office of
Environmental Information.
U.S. Environmental Protection Agency, 2002b. Quality Management Plan for the National Risk
Management Research Laboratory (NRMRL). EPA Document Control Number NRMRL QA
001 rev 1.
U.S. Environmental Protection Agency, 2002c. Guidance on Environmental Data Verification
and Data Validation, EPA QA/G-8. EPA Publication No. EPA/240/R-02/004. Washington, DC:
Office of Environmental Information.
U.S. Environmental Protection Agency, 2003. Guidance on Assessing Quality Systems, EPA
QA/G-3. EPA Publication No. EPA/240/R-03/002. Washington, DC: Office of Environmental
Information.
U.S. Environmental Protection Agency, 2006. Guidance for the Data Quality Objectives
Process, EPA QA/G-4. EPA Publication No. EPA/240/B-06/001. Washington, DC: Office of
Environmental Information.
U.S. Environmental Protection Agency, 2007a. Guidance on Data Quality Indicators, EPA
QA/G-5i. Draft document.
U.S. Environmental Protection Agency, 2007b. Guidance for Preparing Standard Operating
Procedures (SOPs), EPA QA/G-6. EPA Publication No. EPA/600/B-07/001. Washington, DC:
Office of Environmental Information.
44 of 45
-------
APCT CENTER Quality Management Plan Revision 2.3, Effective: March 2, 2010
U.S. Environmental Protection Agency, 2008. Environmental Technology Verification Program
Quality Management Plan. EPA Publication No. EPA/600/R-08/009. Cincinnati, OH: Office
of Research and Development.
45 of 45
------- |