Batteiie
The Business of Innovation
Environmental Technology
Verification Program
Advanced Monitoring
Systems Center
Quality Management Plan
for the
ETV Advanced Monitoring
Systems Center
Version 8.0
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 2 of 48
QUALITY MANAGEMENT PLAN
for the
ETV ADVANCED MONITORING SYSTEMS CENTER
Version 8.0
(SIGNATURE ON FILE)
Signed by John McKernan April 6, 2011
EPA AMS CENTER PROJECT OFFICER Date
Signed by Michelle Henderson ^ '
EPA AMS CENTER QUALITY MANAGER Date
c- j u A rป- j i April 5,2011
Signed by Amy Dindal
BATTELLE AMS CENTER MANAGER Date
c- j u D D ui April 5,2011
Signed by Rosanna Buhl r
BATTELLE AMS CENTER QUALITY MANAGER Date
BATTELLE
505 King Avenue
Columbus, OH 43201
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 3 of 48
TABLE OF CONTENTS
Page
TABLE OF CONTENTS 3
LIST OF TABLES 5
LIST OF FIGURES 5
APPENDICES 5
ACRONYMS 6
1 INTRODUCTION 7
1.1 Purpose 7
1.1.1 AMS Center Purpose 7
1.1.2 Quality System Purpose 7
1.2 Scope 7
1.3 Background 7
.3.1 ETV Program 7
.3.2 Battelle Memorial Institute 8
.3.3 AMS Center 8
1.4 Definitions 8
.4.1 Verbs for Clarity 8
.4.2 Applicable EPA ETV Definitions 8
2 MANAGEMENT AND ORGANIZATION 10
2.1 Quality Policy 10
2.2 Organization and Responsibilities of the AMS Center 10
2.2.1 Battelle AMS Center Manager 10
2.2.2 Battelle AMS Center Quality Manager 10
2.2.3 Battelle AMS Center Verification Testing Leader 11
2.2.4 AMS Center Stakeholder Coordinator 12
2.2.5 Verification Organization Responsibilities 12
2.2.6 Key Staff Responsibilities 12
2.2.7 Stakeholder Responsibilities 12
2.2.8 Vendor Responsibilities 13
2.3 Responsibilities and Authority 13
2.4 Technical Activities Supported by the AMS Center Quality System 13
2.5 Management Assurance of Implementation 13
2.6 Dispute Resolution 14
3 QUALITY SYSTEM AND DESCRIPTION 15
3.1 Quality System Description 15
3.2 Quality System Implementation Tools 16
3.3 Battelle Quality System QMP 16
3.4 Roles and Responsibilities 16
4 PERSONNEL QUALIFICATIONS AND TRAINING 17
4.1 Training Policy 17
4.2 Processes for Ensuring Qualifications and Training 17
4.2.1 Staff Training 17
4.2.2 Documentation of Technical Training 18
4.2.3 Responsibilities for Training 19
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 4 of 48
5 PROCUREMENT AND ACCEPTANCE OF ITEMS AND SERVICES 20
5.1 Policy 20
5.2 Procurement Documents 20
5.3 Responses to Solicitations 20
5.4 Acceptance of Purchased Items and Services 21
5.4.1 Test Equipment 21
5.4.2 Testing Materials 21
5.4.3 Services 21
6 DOCUMENTS AND RECORDS 22
6.1 Documents 22
6.2 Controlled Documents 22
6.2.1 AMS Center QMP 23
6.2.2 Standard Operating Procedures 23
6.2.3 Quality Assurance Project Plan 24
6.3 Records 24
6.3.1 Verification Test Study Files 24
6.3.2 Verification Test Records 24
6.4 Maintaining Document Version Control 25
6.5 Compliance with Records and Documents 25
6.6 Records Management 26
6.7 Compliance 26
6.8 Sample Handling 27
6.8.1 Sample Identification Numbers 27
6.8.2 Sample Custody 27
6.9 Confidentiality 27
6.10 Document Distribution 27
7 COMPUTER HARDWARE AND SOFTWARE 30
7.1 Hardware 30
7.2 Software 30
7.2.1 Change Control 30
7.2.2 Purchased Hardware and Software 31
7.2.3 Validation Policy 31
8 PLANNING 32
8.1 AMS Center Systematic Planning 32
8.1.1 Stakeholders 32
8.1.2 Systematic Planning of Verification Tests 34
8.1.3 Data Quality Objectives 34
8.1.4 Data Quality Indicators 34
8.2 Quality Assurance Project Plans 35
8.2.1 Sample Collection Procedures 35
8.2.2 Documentation Requirements 36
8.2.3 Reference Methods 36
8.2.4 Conformance to the QAPP 36
8.3 Generic Verification Protocols 36
8.4 Existing Data 37
8.5 Roles and Responsibilities 37
9 IMPLEMENTATION OF WORK PROCESSES 38
9.1 Performance of Verification Test Activities 38
9.1.1 Kick-off Meetings 38
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 5 of 48
9.1.2 Communicating Verification Test Requirements 38
9.2 Identifying the Need for Special Procedures 38
9.3 Document Control 39
10 ASSESSMENT AND RESPONSE 40
10.1 Quality System Assessment 40
10.2 Assessment Procedures 40
10.2.1 Performance Evaluation Audit 40
10.2.2 Technical Systems Audits 40
10.2.3 Audit of Data Quality 41
10.2.4 Data Validation 42
10.2.5 Data Assessment 43
10.3 Qualifications of Assessors 43
10.4 Authority of Assessors 44
10.5 Reports to Management 44
10.6 Corrective Action 45
10.7 Dispute Resolution 45
11 QUALITY IMPROVEMENT 46
11.1 Annual QMP Review 46
11.2 Problem Identification and Resolution 46
11.3 Assessments 46
11.4 Customer Satisfaction and Feedback 47
11.5 Communication of Quality System Improvements 47
12 REFERENCES 48
LIST OF TABLES
Table 1. Records Management Responsibilities for the AMS Center 28
Table 2. The Relationship between DQOs, DQIs, and MQOs 35
Table 3. AMS Center ADQ Audit Requirements 42
LIST OF FIGURES
Figure 1. AMS Center Organization 11
Figure 2. Systematic Planning of Verification Tests 33
APPENDICES
APPENDIX A BATTELLE AMS CENTER KEY STAFF AND MASTER DISTRIBUTION LIST
APPENDIX B PERSONNEL RESPONSIBILITIES FOR THE AMS CENTER FOR
VERIFICATION TESTING ACTIVITIES
APPENDIX C AMS CENTER AMENDMENT AND DEVIATION FORMS
APPENDIX D DATA QUALITY INDICATOR DEFINITIONS AND EXAMPLES
APPENDIX E QAPP QA/R-5 ELEMENTS AND PREPARATION GUIDANCE
APPENDIX F ETV KICK-OFF MEETING CHECKLIST
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 6 of 48
ACRONYMS
ADQ Audit of data quality
AMS Advanced Monitoring Systems
ANSI American National Standards Institute
ASQC American Society for Quality Control
COA certificate of analysis
COC Chain of custody
DQI Data quality indicator
DQO Data quality objective
EEGB (Battelle) Energy and Environment
ENVS Battelle Environmental Solutions Product Line
EPA U.S. Environmental Protection Agency
ETV Environmental Technology Verification
IM Information Management
MQO Measurement quality objectives
PE Performance evaluation
QA Quality assurance
QAPP Quality assurance project plan
QC Quality control
QMP Quality management plan
RMO (Battelle) records management office
SBMS (Battelle) standards based management system
SOP Standard operating procedure
TSA Technical systems audit
VO Verification organization
VTC Verification test coordinator
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 7 of 48
1 INTRODUCTION
This document, the Quality Management Plan (QMP) for the Advanced Monitoring Systems (AMS)
Center, describes the quality system that will be employed by Battelle in administering the AMS Center.
This quality system is designed to be consistent with American National Standards Institute
(ANSI)/American Society for Quality Control (ASQC) E4-1994, the U. S. Environmental Protection
Agency (EPA) document Environmental Technology Verification (ETV) QMP (2008), and EPA/QA R-2
(2001). In addition, this QMP is compliant with the Battelle Environmental Solutions (ENVS) Product
Line QMP, which describes the organization's quality system.
1.1 Purpose
1.1.1 AMS Center Purpose
The purpose of the AMS Center is to verify commercially-available technologies for monitoring,
sampling, detection, and characterization of natural species and contaminants in a variety of matrices,
including air, water, and soil. As part of the larger ETV program, the AMS Center is designed to provide
technology users with objective, high quality performance data to support monitoring technology
selection decisions. Detailed information about the AMS Center is presented at
http://www.epa.gov/nrmrl/std/etv/center-ams.html. The AMS Center is one of six centers operating under
the ETV program.
1.1.2 Quality System Purpose
The purpose of the Quality System described in this QMP is to establish policies, processes, and
procedures that will ensure that the quality of data, products, and services provided by the AMS Center
meet or exceed the needs of its stakeholders. It also defines the quality management responsibilities,
functions, and activities that will support the AMS Center Manager in the implementation of the QMP.
1.2 Scope
The scope of this QMP encompasses all activities that Battelle, as an ETV verification organization (VO),
and any test participants, perform during the planning, testing, and reporting of AMS ETV tests.
The Quality System defined in this QMP applies to personnel involved in, and activities conducted for the
AMS Center. It contains the minimum requirements applicable to AMS Center activities. These include
personnel qualifications and training, procurement of items and services, documents and records,
computer hardware and software, planning, implementation of work processes, assessment and response,
and quality improvement provisions.
1.3 Background
1.3.1 ETV Program
The ETV program was established by EPA to evaluate the performance characteristics of innovative
environmental technologies across all media and to report objective performance information to the
permitters, buyers, and users of environmental technology. Credible, high-quality performance
information is one of the tenets of ETV; therefore, the highest appropriate level of quality assurance (QA)
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 8 of 48
is used throughout the program (ETV QMP, 2008). The ETV program is operated by the EPA Office of
Research and Development.
1.3.2 Battelle Memorial Institute
Battelle Memorial Institute (Battelle) is a 501(c)(3) charitable trust headquartered in Columbus, Ohio with
more than 22,000 employees in over 130 locations worldwide. Battelle is the world's largest,
independent research and development organization.
1.3.3 AMS Center
The AMS Center is managed within Battelle's Energy, Environment, and Material Sciences (EEMS)
Global Business which includes chemists, engineers, statisticians, and support personnel. Staff and
facilities will be drawn from EEMS and the other Battelle global businesses as needed to support the
AMS Center.
Staff assigned to support in the AMS Center include those with expertise in environmental monitoring,
stakeholder involvement, and outreach and communication.
Key Battelle facilities that are available for use on the AMS Center include comprehensive laboratory
analysis equipment, field sampling and analysis equipment, source simulators such as pilot plants,
environmental chambers, and real-world test sites.
1.4 Definitions
1.4.1 Verbs for Clarity
Shall, must: when the element is required and deviation from the specification will constitute
nonconformance with this QMP
Should, will: when the element is recommended
May: when the element is optional.
1.4.2 Applicable EPA ETV Definitions
The following definitions were extracted from the EPA ETV QMP or from other EPA QA references.
The EPA ETV QMP may be referenced for a complete summary of general terms at
http://www.epa.gov/etv/pubs/600r08009.pdf.
Generic Verification Protocol - (Also known as guideline document, generic test protocol, or protocol).
This document is developed, modified, or selected to promote uniform testing procedures by the VO for a
single class of technologies. Adequate documentation of a robust GVP may allow the development of
abbreviated individual QA project plans (QAPPs) which incorporate the GVP by reference. Generic
verification protocols may retain draft status until verification testing is performed, then finalized,
building upon the testing experience.
Stakeholders - Groups set up for each ETV center consisting of representatives of any or all of the
following groups: buyers and users of technology, technology developers/vendors, the consulting
engineers, the finance and export communities, government permitters, regulators, first responders,
emergency response, disaster planners, public interest groups, and other groups interested in the
performance of innovative environmental technologies. A list of AMS Center Stakeholder committee
members is available on the ETV website (http://www.epa.gov/etv).
Quality Assurance Project Plan (QAPP) - The plan developed by a VO for each individual test of a
technology or technology class. Therefore, the QAPP may include more than one technology. The QAPP
provides the experimental approach with clearly stated test objectives and associated quality objectives
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 9 of 48
for the related measurements. The QAPP may incorporate or reference existing GVPs or provide the basis
for refining draft GVPs. Guidance may be found at www.epa.gov/quality/qa_docs.html.
Vendor - An individual, company, or organization which submits a commercially-available
environmental technology for verification testing.
Verification Organization (VO) - The public and private sector organizations holding cooperative or
interagency agreements or contracts to assist EPA in implementing the ETV program. Battelle is the VO
for the AMS Center.
Verification Organization Center Manager - The person designated by the VO with the responsibility
to manage the Center and serve as the chief point of contact with the EPA. For the AMS Center, this is
the Battelle AMS Center Manager.
Verification Organization Quality Manager - The person designated by the VO to manage QA for the
ETV center on behalf of the VO manager. For the AMS Center, this is the Battelle AMS Center Quality
Manager.
Verification Report - Contains the technology description, how the tests were conducted, performance
verification test results, statistics and summary. The report is developed for each verified technology.
Verification Statement - A statement signed by the EPA laboratory director and the VO Product Line
manager. The statement contains the summary, generally 3 to 6 pages of the test results for a given
environmental technology.
Verification Test - The conducting of the performance evaluation of one or more similar environmental
technologies that is documented in a single QAPP and in one or more verification statements and
verification reports. Multiple verification tests, each with its own QAPP, may be conducted at different
times under a single GVP.
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 10 of 48
MANAGEMENT AND ORGANIZATION
2.1 Quality Policy
Battelle's quality policy is to provide services, products, and data of the highest quality that meet or
exceed our client's requirements and expectations. For the AMS Center, this policy is to implement best
practices in testing and assessment; ensure that organizations participating in testing have a quality system
consistent with this QMP; ensure that verification testing is conducted according to the QAPP; and that
verification reports are supported by quality-assured data. Battelle supports implementation of this policy
by supplying the resources, including staff, supplies, and support, needed to achieve the goals of each
verification test.
2.2 Organization and Responsibilities of the AMS Center
The organization chart for the AMS Center is provided in Figure 1.0 and shows key AMS Center staff
and their reporting lines. Descriptions of key AMS Center staff are provided below. The names,
mailing/email addresses, and phone/facsimile numbers of these Battelle AMS Center key staff are
included in Appendix A.
2.2.1 Battelle AMS Center Manager
The Battelle AMS Center Manager is responsible for meeting all technical, budget, and schedule goals for
the AMS Center. The AMS Center Manager serves as the primary point of contact for the EPA AMS
Center Project Officer. The AMS Center Manager also directs the activities of the leaders in three
organizational areas within the Center: Quality Assurance, Verification Testing, and Stakeholders. The
AMS Center Manager reports to the ENVS Product Line Manager within EEMS.
The Product Line Manager is responsible to provide the AMS Center Manager and the other key AMS
Center staff with direct support in securing and deploying Battelle resources for the AMS Center. The
ENVS Product Line Manager has ultimate responsibility for ensuring that necessary Battelle facility and
staff resources are available to support the AMS Center. The ENVS Product Line Manager signs the
verification statements for Battelle.
2.2.2 Battelle AMS Center Quality Manager
The Battelle AMS Center Quality Manager is independent of technical operations and reports directly to
the ENVS Product Line Manager. The AMS Center Quality Manager reports the results of assessments
of AMS Center activities to the AMS Center Manager and works with technical staff to address quality-
related issues, identify corrective action and quality system improvements. The AMS Center Quality
Manager is also the primary point of contact for the EPA AMS Center Quality Manager.
The Battelle AMS Center Quality Manager may appoint qualified QA Officers to perform assessments of
specific test activities. In these cases, the QA Officers report the results of their assessments to the
Battelle AMS Center Quality Manager. The responsibilities of Battelle AMS Center QA Officers are
defined in Appendix B.
-------
ETV Advanced Monitoring Systems Center
Quality Management Plan Version 8.0
Date: Aprils, 2011
Page 11 of 48
EPA AM S
Center Quality
Manager
Battelle
Stakeholder
Coordinator
Battelle
Stakeholder
Committees
EPA
AMS Center
Project Officer
Battelle
AMS
Center
Manager
Battelle
Verification
Testing
Leader
Verification Test Coordinators
Battelle
Energy,
Environment
and Material
Sciences
Global Business
President
Battelle
Environmental
Solutions
Product Line
Manager
Battelle
AMS Center
Quality
Manager
Battelle
AMS Center
Quality
Assurance
Officers
Figure 1. AMS Center Organization
(dotted lines indicate indirect reports)
2.2.3 Battelle AMS Center Verification Testing Leader
The AMS Center Verification Testing Leader has responsibility planning and leading verification tests,
including management of verification test coordinators (VTCs).
The AMS Center Verification Testing Leader reports directly to the AMS Center Manager regarding
activities, and can serve in the role of AMS Center Manager for tasks delegated by the AMS Center
Manager.
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 12 of 48
2.2.4 AMS Center Stakeholder Coordinator
The AMS Center Stakeholder Coordinator has primary responsibility for stakeholder recruitment,
communications, and stakeholder meeting facilitation. The AMS Center Stakeholder Coordinator reports
directly to the AMS Center Manager for the AMS Center activities.
2.2.5 Verification Organization Responsibilities
In accordance with EPA's ETV QMP dated January 2008, Battelle's responsibilities for the AMS Center
include the following:
establish, attend, and/or conduct meetings of stakeholder committees with representation from major
customer groups;
maintain communication with EPA to assure mutual understanding and conformance with EPA quality
procedures and expectations and ETV policies and procedures;
develop, review, and revise QAPPs in cooperation with technology vendors and stakeholders;
solicit technology vendors and verification test collaborators;
manage participation of and conduct verification activities;
assure that quality procedures are incorporated into all aspects of the AMS Center;
perform ETV activities within the documented quality system;
prepare ETV verification reports and statements at the completion of each technology verification;
appoint a quality manager, responsible for ensuring the AMS Center quality systems are in compliance
with the E4 standard and the EPA ETV QMP dated January 2008, and AMS Center staff complies with
this QMP.
2.2.6 Key Staff Responsibilities
Battelle is committed to operate an effective quality system that ensures compliance with all program
requirements. The responsibilities and authority of key EPA and Battelle AMS Center staff who have
responsibilities for verification testing activities described by this AMS Center QMP are listed in
Appendix B.
2.2.7 Stakeholder Responsibilities
The responsibilities of stakeholders for the AMS Center include the following:
participate in technical panel discussions (when available) and/or review an outline of the verification test
to provide input to the test design;
assist in development of the generic verification protocol when needed;
assist in prioritizing the types of technologies to be verified, focusing on these technologies with greatest
environmental and sustainability impacts;
review and provide input to Center-specific procedures and AMS Center documents including QAPPs,
verification reports, and verification statements, as requested;
participate in verification testing as collaborators, provide funding and/or in-kind support, or recommend
collaborators to the AMS Center;
assist in the definition and conduct of outreach activities appropriate to the technology area and customer
groups;
serve as information conduits to the particular constituencies that each member represents.
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 13 of 48
2.2.8 Vendor Responsibilities
The responsibilities of vendors who choose to participate in verification testing may include any of the
following. In addition, these test-specific responsibilities will be defined in the QAPP:
review and provide comments on the draft QAPP;
approve the final QAPP prior to test initiation;
provide technology(ies) for evaluation during the verification test;
provide all equipment/supplies/reagents/consumables needed to operate their technology(ies) for the
duration of the verification test;
supply a representative to train Battelle staff in operation of their technologies and provide written
consent for Battelle staff to operate their technologies during verification testing;
provide written instructions for routine operation of their technologies;
review and provide comments on the draft verification report and statement for their technology(ies).
2.3 Responsibilities and Authority
Battelle management is responsible for committing to a quality policy and for creating work environments
in which all personnel strive for the highest quality of services and products. Management shall also
provide the Battelle AMS Center Manager with the authority to ensure the following:
All applicable elements of the quality system as described in this QMP are understood and are
implemented in the AMS Center.
Adequate personnel and resources are available to plan, implement, assess, and improve services and
products relevant to the AMS Center.
Staff is (are) clearly designated to stop unsafe work and work of inadequate quality as affects the AMS
Center.
2.4 Technical Activities Supported by the AMS Center Quality System
This QMP applies to all verification testing performed under the EPA ETV AMS Center program.
Specifically, it applies to:
technical activities including planning, testing, and reporting;
any Battelle staff, facilities, and other resources used during testing;
reference laboratories and collaborators that perform activities in support of verification testing for the
AMS Center.
2.5 Management Assurance of Implementation
Battelle management is committed to ensuring that the quality system described in this QMP is
implemented for the AMS Center. The AMS Center QMP is in compliance with the ENVS QMP which is
approved by ENVS management. To ensure that staff understand and implement the quality system
requirements:
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 14 of 48
All staff involved in ETV testing are required to read this QMP. Reading is documented in Battelle's
training database, Battelle University3.
Dedicated management and technical staff are identified to ensure consistent application of the quality
system.
A quality manager has been designated with the resources needed to ensure that the QMP is
communicated, implementation verified, and that management is appraised of issues requiring corrective
action.
2.6 Dispute Resolution
If disputes arise during conduct of a test, the VTC is responsible to resolve the issue with the relevant
vendors, reviewers, and/or stakeholders. Every effort will be made to address concerns in a timely
manner. The AMS Center Manager is the arbiter of disputes that cannot be resolved by the VTC and also
for informing the EPA AMS Center Project Officer. If an audit finding or response creates a dispute that
cannot be resolved by the QA Officer and the VTC, the dispute will be elevated to the AMS Center
Quality Manager for resolution. The EPA AMS Center Project Officer and AMS Center Quality Manager
are the final arbiters of disputes that cannot be otherwise resolved.
"Battelle University is a training and training tracking tool. It contains a list of courses developed and offered by
Battelle and a reporting function that documents the training received by each staff member.
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 15 of 48
3 QUALITY SYSTEM AND DESCRIPTION
The Battelle quality system for the AMS Center is described in this QMP and conforms to the
specifications listed in:
ANSI/ASQ E4-2004, Specifications and Guidelines for Quality Systems for Environmental Data
Collection and Environmental Technology Programs;
EPA document: Environmental Technology Verification Program Quality Management Plan Version 3.0,
January, 2008;
EPA document: EPA Requirements for Quality Management Plans, EPA QA/R-2, March, 2001;
Battelle document Environmental Product Line Quality Management Plan, Revision 0, October, 2009.
3.1 Quality System Description
The AMS Center quality system is comprised of the following activities that support the quality policy:
quality system documentation (QMP);
training;
systematic projects planning ;
project-specific quality documentation (QAPPs, amendments, deviations, and SOPs);
management assessments;
project and data assessments.
Battelle's organizational quality system includes the following standards and guidelines that are applicable to
all project conducted within the ENVS product line:
Battelle Policy Manual: This document articulates the corporate level Battelle policies required for all
Battelle business and administrative practices. The implementation of the policies is intended to conform
to the requirements of the Battelle Memorial Institute Articles of Incorporation, relevant laws and
regulations, and Battelle-specific standards to preserve Battelle's reputation for scientific integrity,
objectivity and the highest standards of conduct.
The Project Manager Handbook: This document is a central repository of Battelle-specific guidance for
proper project initiation, planning, execution, control, and close-out using Battelle procedures and support
resources.
ENVS QMP: This document describes the quality system for the ENVS Product Line. It follows the
format and addressed the content requirements for EPA QA/R-2 (2002) (Section 3.3).
Records management and retention: The mission of Battelle's Records Management Office (RMO) is to
ensure that Battelle records are reliable, complete, cost effective, and accessible for as long as considered
necessary by Battelle and our clients. This includes providing guidance on how to create good records,
how long to keep records, and how to store and destroy records.
Battelle Standards Based Management System (SBMS): Battelle SBMS provides a single location for
policies, procedures and guidance that impact all staff. SBMS is designed to make Battelle policies and
procedures readily available to all staff in a concise and useful form. It includes the following document
types: procedure areas, policies, forms, supporting information, and management system descriptions.
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 16 of 48
3.2 Quality System Implementation Tools
Battelle uses several tools to implement the quality system for the AMS Center. These tools are listed below
with the QMP section that discusses the procedures, requirements, and responsibilities for each:
QMP (Section 6.1.1);
training (general Battelle requirements and test-specific training) (Section 4.0);
Battelle University (to track staff training records) (Section 4.2.1);
ENVS Corrective Action Logger (Section 10.6);
QAPPs (Sections 6.1 and 8.2);
TSAs (Section 10.2.2);
Audits of data quality (ADQs) (Section 10.2.3).
3.3 Battelle Quality System QMP
In addition to the AMS Center QMP, the ENVS Quality Systems Manager, Ms. Rosanna Buhl, develops a
QMP for the product line. Any projects conducted for the AMS Center must also comply with the
product line QMP.
The ENVS Quality Systems Manager is responsible for the development of the QMP.
The QMP is reviewed and approved by the ENVS Product Line Manager and Section Managers.
The QMP is distributed to each staff member. Documentation that each staff member has read the QMP
is entered by the staff member in Battelle University.
The QMP is reviewed by management and revised annually by the ENVS Quality Systems Manager if
needed to reflect changes to the scope, organization, quality system, or Battelle corporate requirements.
3.4 Roles and Responsibilities
Every staff member is responsible for implementing the QMP, applicable SOPs, and QAPPs, and
verifying the quality of their work.
Battelle Managers (Battelle AMS Center Manager, VTC, and the Battelle AMS Center Quality Manager)
are responsible for ensuring the implementation of the QMP for the program and for each verification
test.
VTCs managers are ultimately responsible for the quality of work performed for their projects.
Appendix B provides detailed roles and responsibilities.
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 17 of 48
PERSONNEL QUALIFICATIONS AND TRAINING
4.1 Training Policy
It is AMS Center policy that staff members have the knowledge, skill, and any professional certifications
needed to perform their AMS Center assignments. If formal training is available or required, such training
must be documented.
4.2 Processes for Ensuring Qualifications and Training
For the AMS Center, Battelle personnel qualifications and training must be suitable for technical work
performed in support of verification testing activities. Each verification test QAPP will define the
specific qualifications required for key personnel and any specialized training. Training and qualification
are achieved through formal education, class-time, seminars, and on-the-job training. Staff must have
demonstrated ability in an assignment prior to working independently. Assessing Staff Qualifications
Staff qualifications are assessed as part of the verification test team assembly. The AMS Center Manager
assigns VTCs on a test-specific basis. VTCs are selected based on their technical expertise and project
management training and experience. Staff that have served as VTCs on previous AMS Center
verification tests are considered first due to their experience with implementation of the AMS Center
QMP. The VTC selects technical leads, field staff, and laboratory facilities and staff to support each
verification test based on their technical qualifications and prior experience.
4.2.1 Staff Training
Staff training is primarily accomplished through on-the-job training, Battelle University, seminars, and
internal or external training sessions. During on-the-job training, staff perform work under the
supervision of qualified personnel.
Job proficiency is assessed by demonstrated skill as witnessed by a qualified trainer/staff member.
Minimum requirements for each job assignment may include professional training, appropriate degrees or
years of experience, and demonstrated expertise.
For each job title, Battelle's Human Resources office defines the function, responsibilities, authority,
dimensions, and requirements. These criteria establish the minimum requirements for training and
demonstration of proficiency for each staff member.
For some activities, such as the use of instruments or equipment, proficiency is demonstrated by the
ability to perform instrument inspections, operation, calibration, and maintenance; objective measures
such as instrument performance can be used to assess proficiency.
Proficiency for personnel who are assigned as AMS Center QA Officers include: demonstrated ability to
identify critical QA/QC issues, accurately assess the importance of those issues, and prepare written
reports that clearly discuss the issues and appropriate resolution. Section 10.3 defines the assessor
qualifications.
Battelle University provides Battelle staff members, managers, and leaders to access to over 260 personal,
team, and functional training sessions that provide learning in a variety of formats (including web-based,
self-paced, class room sessions, or webinars). Staffer managers can identify training needed to perform a
current function or to enhance skills. Professional external training is available via links to Harvard
Manage Mentor. Battelle University also houses all the ENVS SOPs; staff can select and read SOPs that
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 18 of 48
are needed to meet project requirements. As noted in Section 4.2.2, a record is maintained of any training
completed through Battelle University.
Seminars are presented by Battelle to enhance specific skills. These are offered on-site by Battelle for
organization-wide requirements, such as general computer software use (E-mail, spreadsheets) and to
enhance specific skills, such as project management or sample custody. Off-site training, project/program
meetings, and technical society membership are available for specific disciplines contributing to the staff
member's overall job proficiency.
To ensure that Battelle's AMS Center personnel are familiar with current QMP requirements they will be
notified via AMS Center SharePoint site announcement when the QMP is revised; a summary of changes
will be provided.
The need for retraining will be based on changes in job requirements (assignments), modification of
existing procedures, or quality system requirements for periodic refresher courses. Generally, the need
for retraining will be determined by the staff member and the appropriate level of management (project or
Associate Managers). To maintain staff proficiency, opportunities are provided by Battelle annually and
as needed.
4.2.2 Documentation of Technical Training
As described in AMS Center QAPPs, relevant training that demonstrates staff qualifications to perform
the verification test activities must be documented. Documentation will be maintained in Battelle
University whenever possible. Historical records and ancillary training records that cannot be entered into
Battelle University will be maintained in the staff member's qualification/training file. Battelle personnel
working on AMS Center verification tests are responsible for maintaining their records of training and
experience, which shall include the following record:
education history, which can include formal qualifications or certification relevant to technical, QA, or
management disciplines;
work experience, which can be either academic background or on-the-job performance in technical and/or
management areas (i.e., performance on task, project, or program assignments);
experience in the application of QA/quality control (QC) requirements in technical performance or data
verification;
on-the-job training in specific skill documented by the qualified training/staff member (Specific
performance based on national certification requirements or performance evaluation samples can be
recorded with certificates or other documentation.);
expertise in advanced technical activities based on experience and demonstrated competence. (Staff
expertise is documented in the Battelle expertise database (link) and biosketches, which capture project-
related experience).
Verification test collaborators working on behalf of Battelle in support of the AMS Center and/or
individual test operations are expected to provide the VTC with records of the following:
educational background and/or degree(s) relevant to technical areas represented in the AMS Center;
work experience related to the technology category undergoing verification.
This information will be reviewed by the Battelle AMS Center QA Officer during the TSA.
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 19 of 48
4.2.3 Responsibilities for Training
Associate Managers have primary responsibility for providing resources for required staff training.
Project Managers are responsible for ensuring that staff has appropriate training for project-related
assignments.
Staff is responsible for completing required training, maintaining their training records, and alerting their
Associate Manager and Project Manager if they do not have the training needed for their assigned tasks.
Training documents may be reviewed by the Battelle AMS Center QAO prior to the kick-off meeting or
as part of the TSA, as appropriate.
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 20 of 48
5 PROCUREMENT AND ACCEPTANCE OF ITEMS AND SERVICES
5.1 Policy
It is AMS Center policy that the quality of purchased items and services meet the end use requirements.
The technical and quality requirements for procured items and services are generally based upon value
(cost, durability, maintainability), performance (specification compliance, operating conditions,
calibration capacity), delivery (timeliness, ease of ordering), customer support (responsiveness, technical
ability), past experience with a particular vendor, and completeness and coherence of instructions (clarity,
accuracy).
5.2 Procurement Documents
Technical and quality requirements for items and services procured for a specific verification test should
be included in the QAPP. These requirements will typically be specified under materials and/or
measurement system equipment (QAPP Section B8, Inspection/Acceptance Requirements for Supplies
and Consumables). The VTC or technical lead will review purchase orders and contracts for critical
supplies and equipment to ensure that they define the level of quality needed.
Purchase orders and contracted services must be complete, accurate, and clearly describe, as appropriate:
the item or service needed;
any associated technical and quality requirements (such as purity, calibration requirements, etc.);
quality system elements for which the supplier is responsible, e.g., internal review of analytical results for
accuracy;
how the supplier will verify conformance to Battelle's requirements, e.g., internal review of technical
activities vs. the QAPP requirements for QC samples.
5.3 Responses to Solicitations
If any aspect of verification testing is contracted to another organization, such as an independent reference
laboratory, the proposal or qualifications submitted by that organization must be reviewed by the VTC to
ensure that the supplier has a documented quality system consistent with this document and that it has the
qualifications to perform the work defined in the QAPP. At a minimum, the VTC should verify the
following:
documentation of the organization's quality system in a current, detailed quality manual;
SOPs or protocols exist for the critical aspects of testing (e.g., sample analysis);
limits of detection and/or quantitation are defined for critical quantitative measurements;
procedures for equipment and instrument calibration have defined frequencies and acceptance criteria;
QC samples are defined with frequencies and acceptance criteria;
methods for independent data verification and validation are defined;
include compliance to the test-specific QAPP as part of the purchasing agreement, where possible.
Where possible, laboratory and other services should meet registration or certification requirements
applicable to the verification. The laboratory should be provided with pertinent sections of the QAPP or
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 21 of 48
asked to provide input to the analytical sections to ensure that the QAPP accurately reflects laboratory
capabilities and practices.
5.4 Acceptance of Purchased Items and Services
Purchased items must be inspected when received to ensure that they are not defective and that they are of
the right type and quality to meet the intended use. Purchased services (e.g., laboratory analysis,
subcontractor reports) must be reviewed to ensure that the quality meets the requirements of the project or
intended use. If so defined in the QAPP, a TSA will be conducted during project activities to assess
performance in real time.
5.4.1 Test Equipment
Testing equipment purchased or rented for activities affecting quality shall be tested.
Equipment must be calibrated using independent standards to ensure that the level of accuracy and
precision defined in the QAPP can be achieved.
Critical equipment performance must be verified prior to use in the verification test (e.g., performance
evaluation [PE] audits). If the equipment does not meet QAPP criteria initially then the equipment should
be recalibrated prior to further use. If the equipment is found to operate outside of the QAPP criteria then
it must either be tagged and removed from service or returned to the supplier for repair/replacement, as
necessary.
Initial verification, routine calibration, and maintenance records will be included in individual verification
test records. These records should be included in the laboratory data package provided with the data.
5.4.2 Testing Materials
Test materials procured for activities affecting quality (e.g. reference standards or gases) shall be
accompanied with a Certificate of Analysis (COA) where appropriate. The COA will be examined to
ensure that the listed specifications are within the QAPP limits. The COA will be retained and included
in the verification test records.
5.4.3 Services
Methods to accept procurement of services (i.e. subcontractors, installation, repair, or maintenance work,
etc.) include technical verification of the data produced, surveillance and/or audit of the activity being
performed, or review of objective evidence for conformance to procurement document requirements.
Analytical data should be accompanied by a QC narrative that summarizes the analytical results and any
QC failures or technical deviations from the laboratory SOPs or the QAPP.
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 22 of 48
DOCUMENTS AND RECORDS
6.1 Documents
ANSI/ASQ E4-2004 states that documents may be any media that contains information describing,
defining, specifying, reporting, or certifying activities, requirements, procedures, or results. Documents
developed or used by the AMC Center include both uncontrolled and controlled documents.
Uncontrolled documents maintained by the Battelle AMS Center Manager may include the following:
minutes of stakeholder meetings;
cooperative agreement records;
verification reports;
verification statements;
Battelle quality assessment reports.
These documents are maintained by the Battelle AMS Center Manager and posted to the AMS Center
SharePoint site and website, as appropriate. Controlled documents are discussed in Section 6.2.
6.2 Controlled Documents
Some documents generated by or for the AMS Center require control to ensure that only the most current
versions are available to document users. The AMS Center will treat the following documents as
controlled:
QMP for the ETV Advanced Monitoring Systems Center (this document);
Standard Operating Procedures (SOPs);
QAPPs, including amendments and deviations;
Generic Verification Protocols.
Document control is maintained by implementing the following procedures:
formal signature approval block within the document (QMP and SOPs, only);
an effective date;
a pagination system;
a specific version number or designation as a permanent part of the document;
retention of the document with original signed page(s) in a limited access storage area (QMP, QAPPs, and
SOPs);
a distribution list;
notification of document publication or revision as an AMS Center announcement and email to applicable
AMS Center distribution list (Appendix A) (QMP and SOPs, only);
availability of only the current version of each document on the AMS Center SharePoint site;
revision only by the originator or as assigned by the AMS Center Manager or Quality Manager or this
QMP.
Controlled document identification will be assigned by the person primarily responsible for the document
(typically the originator/author). The Battelle AMS Center Quality Manager will maintain a distribution
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 23 of 48
list for the QMP and any program or project-specific SOPs and is responsible for ensuring that these
individuals are notified if the documents are updated. The VTC has this responsibility for QAPPs,
amendments, and deviations. Table 1 defines records management responsibilities for the AMS Center.
6.2.1 AMS Center QMP
The AMS Center QMP describes Battelle's quality system for the AMS Center. The format and content
is based on the following documents:
ANSI/ASQ E4-2004, Specifications and Guidelines for Quality Systems for Environmental Data
Collection and Environmental Technology Programs;
EPA document: Environmental Technology Verification Program Quality Management Plan Version 3.0,
January, 2008;
EPA document: EPA Requirements for Quality Management Plans, EPA QA/R-2, March, 2001;
Battelle document Environmental Product Line Quality Management Plan, Revision 0, October, 2009.
The AMS Center QMP is prepared by the Battelle AMS Quality Manager, and reviewed by the Battelle
AMS Center Manager, the EPA AMS Center Project Officer, and the EPA AMS Center Quality Manager.
The QMP review assesses accuracy, conformance to the guidance documents listed above, and
appropriate level of detail. Once all comments are addressed, the document is signed by these four AMS
Center key personnel. The signature page indicates consensus and commitment to implementing the QMP
for each verification test.
The initial approved QMP was identified as Version 1.0; revisions have been designated as Version 2.0,
Version 3.0, etc. The version number and effective date appear on the upper right corner of each
document page.
The approved QMP will be distributed to all staff electronically via e-mail with a link to Battelle
University and the AMS Center Share Point site. All staff listed in the distribution (Appendix B) must read
the document, verify training in Battelle University, and perform assignments in accordance with the
policies defined in this document. The AMS Center Quality Manager will ensure, as part of testing kick-
off meetings, that staff new to the ETV program (e.g., newly assigned VTCs) are directed to read the
QMP.
To ensure that the QMP remains current, it is reviewed annually by the AMS Center Quality Manager.
The results of the review will be documented in an email to the AMS Center Manager that indicates if a
revision is needed and includes the "marked up" QMP indicating sections that require revision. The
document will be revised if needed to reflect current procedures. The revised document is then approved
by the original approvers, and redistributed to the full distribution list. The Battelle AMS Center Quality
Manager is responsible for maintaining the original hardcopy and Microsoftฎ Word version of the current
document and all historical versions of the QMP.
6.2.2 Standard Operating Procedures
It is AMS Center policy that SOPs be in place for all routine procedures that affect the quality of products
or services. The term "SOP" includes textual documents, flow-charts, checklists, blue prints, etc. SOPs
help to ensure the integrity, reproducibility, and quality of data and must be clearly written with sufficient
detail that a qualified individual can perform the procedure independently. Procedures that are not
routine, or that are unique to a project, may be described in the QAPP or in written protocols attached to
the QAPP.
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 24 of 48
In general, SOPs are authored and maintained by the technical groups at Battelle, collaborators, or
reference laboratories that perform testing and analysis in support of verification testing. These
organizations are expected to have procedures for the preparation, review, approval, release, and revision
of their SOPs. If ETV-specific SOPs are required, they will be prepared according to Battelle ENVS SOP
1-001. SOPs must be managed as controlled documents (Section 6.1).
SOPs must be readily available "at the bench" to project staff who are expected to follow them.
Staff must have documented training in the application of SOPs they will implement for testing activities.
SOPs that describe testing and analysis must be maintained as part of the project files on the ETV
Share Point site.
6.2.3 Quality Assurance Project Plan
A QAPP is prepared for each verification test. The preparation, review for conformance, approve, issue,
use, and revision of QAPPs is discussed in Section 8.2.
6.3 Records
ANSI/ASQ E4-2004 states that a record is a document that furnishes objective evidence of the quality of
items or activities and that has been verified and authenticated as technically complete and correct.
Records may include photographs, drawings, magnetic tape, and other data recording media. Typical
AMS Center records include test study files and test records. Requirements for maintaining these records
are discussed in the subsections that follow.
6.3.1 Verification Test Study Files
Detailed, systematic records must be maintained for each project. Project records shall be detailed
enough to track project progress, identify decision points, and support conclusions. Project data and
records must be capable of withstanding challenges to their validity, accuracy, and legibility. The VTC is
responsible for meeting these requirements at a minimum:
Hard copy documents and records that are part of the project study files should be maintained in
organized project files.
Electronic project study files must be maintained on the Battelle AMS Center SharePoint site; no study
files may exist only on personal computers.
The VTC is responsible for setting up the project folder with main folders and subfolders needed to
effectively manage project records. At a minimum, the project folder should contain folders for project
management records, work in progress, data, draft or pre-delivery reviews, and final deliverables.
The QAPP will specify the specific study file locations and requirements.
The VTCs have the option of setting up external SharePoint sites for data sharing with EPA and other
project collaborators.
6.3.2 Verification Test Records
Verification test records collected in support of AMS Center testing must be collected using scientifically
valid methods and retained securely. Raw (original) datab collected in the field or laboratory should be
b Raw data are defined as any original factual information from a measurement activity or study recorded in a
laboratory notebook, worksheets, records, memoranda, notes, or exact copies thereof that are necessary for the
reconstruction and evaluation of the report of the activity or study. Raw data may include photography, microfilm
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 25 of 48
recorded such that samples collected and data generated are complete and traceable throughout their
history.
Data should be recorded in standardized formats, e.g., data collection forms, bound and paginated
laboratory and field logbooks, laboratory record books, spreadsheets, computer records, and output from
instruments (both electronic and hardcopy). The QAPP must define how test data will be documented.
All verification test records shall carry minimum identification pertaining to title, responsible person or
author, and date.
All manual entries shall be entered using ink and initial and dated by the individual recording the entry.
Changes to original (raw) data should not obliterate the original entry and should be corrected using a
single line and annotated with the new data, and the date/initials of the person who modified the record.
A short explanation will be added to non-obvious corrections.
Electronic data collected by field or laboratory instruments should be backed up daily or transcribed daily
onto a hard copy data form and verified 100% by another person.
Instrument logs should be used document use and maintenance. Calibration records should be maintained
as part of the study file.
Laboratory and field records must be completed, reviewed in real time, and provided to the VTC as soon
as practically possible.
Once a verification test has been completed, the VTC must complete the AMS Center Project Closeout
list located on the AMS Center Share Point site when sending project study file documents for archive to
the Battelle Records Management Office (RMO) (Section 6.4).
6.4 Maintaining Document Version Control
It is critical that version control be maintained for project documents and records, including data
spreadsheets. Thus, a naming convention that uniquely identifies each document revision is required
unless the project SharePoint site is used for document collaboration. The AMS Center default naming
convention is as follows; the VTC may develop alternate procedures as appropriate to meet testing needs:
Internal working document versions are named with the file name and the date updated.
Review versions are re-named by adding the initials of each reviewer to the document.
When a document is ready for release outside of Battelle it is renamed as either draft, final, or revised
final with the date of release. Initials and internal naming conventions are removed
Internal revisions in response to comments are named logically using the format above to again track
changes through the revision process.
6.5 Compliance with Records and Documents
Testing performed by the AMS Center must conform to the QMP, QAPP, and applicable SOPs. A
deviation occurs when testing does not comply with the requirements of these documents. Once a
deviation has been identified during testing, it must be communicated to the VTC, Battelle AMS Center
Manager, Battelle AMS Center Quality Manager, EPA AMS Center Project Officer, and EPA AMS
Center Quality Manager within 24 hours and documented in a formal deviation to EPA within 2 weeks.
or microfiche copies, computer printouts, magnetic media, including dictated observations, and recorded data from
automated instruments. If exact copies of raw data have been prepared (e.g., tapes which have been transcribed
verbatim, data verified accurate by signature), the exact copy or exact transcript may be submitted (Department of
Defense [DoD], 2009). Sections 6.1 and 6.3 provide ANSI/ASQC definitions for documents and records.
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 26 of 48
Deviations must be fully documented including, date and description of deviation, and impact on the
verification test. Amendment and deviation forms are in Appendix C.
6.6 Records Management
All AMS Center documents and records needed to reconstruct test activities and verify that reported data
were collected in a manner consistent with this QMP and AMS Center requirements will be catalogued
and maintained for an established retention period. The Battelle RMO will retain, as a permanent record,
documentation of the transfer or destruction of Battelle's AMS Center documents and records. It is
Battelle policy that all program, project study files, and business records must be archived by the RMO in
accordance with the requirements established in SBMS.
The VTC is responsible for assembling and submitting hard copy and electronic verification test records
to the Battelle RMO for archival.
The RMO SBMS defines procedures for labeling, inventorying, transferring, retrieving, and destroying
records. Battelle SOPs and LRBs are maintained permanently.
The retention period for AMS Center documents begins at the end of the verification test period of
performance (i.e., EPA approval of the verification report). Program files and QA records will be
maintained by Battelle's RMO for 10 years. Final verification reports will be maintained by Battelle's
RMO for 20 years. Disposition of the records after the retention period will differ by QA Category as
follows:
For verifications designated by the EPA as QA Category III, all records will be disposed at the
retention period (10 years project study files and QA records; 20 years for reports).
For verifications designated by the EPA as QA Category II, all records held by the Battelle RMO
will be submitted to the US EPA at the end of the retention period. Battelle will ship the records
to the following address:
Andrew W. Breidenbach Environmental Research Center
U.S. Environmental Protection Agency
26 W. Martin Luther King Dr.
MS 208
Cincinnati, OH 45268
Attention: AMS Center Project Officer
Battelle provides the EPA AMS Center Project Officer with test documents and records (described in
Sections 6.1 and 6.2) throughout testing. Battelle provides electronic copies of the QAPP and reports for
the EPA AMS Center Project Officer to upload to the ETV website. Battelle also uploads QA audit
reports to the ETV database. Additional documents and records will be provided to AMS upon request.
6.7 Compliance
The AMS Center QMP and QAPPs are prepared to meet the format and content requirements of EPA
QA/R-2 (2001), and EPA QA/R-5 (2002), respectively, as well as the current version of the ETV Program
QMP. The review and approval process defined in this QMP and EPA procedures are intended to ensure
that the QMP and QAPPs comply with EPA standards. Approval of these documents by EPA establishes
compliance.
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 27 of 48
6.8 Sample Handling
It is critical that documentation prior to, during, and after field operations should be adequate to enable
historical reconstruction of all events resulting in final data, including sufficient detail so that decision
logic may be traced. Unique sample numbers and rigorous sample transfer procedures are key to data
traceability.
6.8.1 Sample Identification Numbers
Unique sample numbers must be assigned to each collected sample to enable sample tracking through an
entire process to ensure samples are not switched accidentally, lost, or reported with the wrong data. A
unique sample is designated as one that has the same:
source (station location);
date and time;
collection technique (note this would encompass matrix);
preservation;
analyte tests to be run on each container or group of containers.
6.8.2 Sample Custody
When samples are transported from the field, regardless of transportation method, a sample transmittal or
chain-of-custody (COC) form must accompany the samples. The form should list each sample present in
the shipping container. Samples are considered to be in a person's custody if:
The samples are in a person's actual possession.
The samples are in a person's view after being in that person's possession.
The samples were in a person's possession and then were locked or sealed up to prevent tampering.
The samples are in a secure area.
6.9 Confidentiality
As specified in each AMS Center Vendor Agreement, vendor technology information marked proprietary
must not be disclosed, intentionally or unintentionally, to anyone except other authorized relevant Battelle
staff. Appropriate markings are to be used on all sensitive materials to assist in their protection.
6.10 Document Distribution
Once records have been reviewed and approved as required, distribution will be made as listed in Table 1.
AMS Center documents specifically requiring EPA approval before release include:
AMS Center QMP;
Generic Verification Protocols;
QAPPs;
ETV verification reports;
ETV verification statements.
-------
ETV Advanced Monitoring Systems Center
Quality Management Plan Version 8.0
Date: Aprils, 2011
Page 28 of 48
Table 1. Records Management Responsibilities for the AMS Center
Record Type
Cooperative
Agreement
Reporting
Records (e.g.,
quarterly and
annual progress
reports)
AMS Center
Quality
Management Plan
Minutes of
Stakeholder
Meetings
QAPP (including
SOPs,
amendments and
deviations)
Generic
Verification
Protocol
Raw data
Preparation/
Updating
Battelle AMS
Center
Manager
Battelle AMS
Center
Quality
Manager
Battelle AMS
Center
Stakeholder
Coordinator
Battelle VTCs
Battelle
Battelle
Review
Battelle Verification Testing
Leader
Battelle AMS Center Quality
Manager
Battelle AMS Center Manager
EPA AMS Center Project
Officer
EPA AMS Center Quality
Manager
EPA AMS Center Project
Officer
Stakeholders
Battelle AMS Center Manager
Battelle AMS Center Quality
Manager
EPA AMS Center Quality
Manager
EPA AMS Center Project
Officer
Assigned Stakeholders2
Peer Reviewers2
EPA AMS Center Project
Officer
Battelle AMS Center Quality
Manager
Assigned Stakeholders
Internal Technical Peer
Review
Approval
Battelle AMS Center
Manager's line manager
EPA AMS Center Project
Officer
EPA AMS Center Quality
Manager
Battelle AMS Center
Manager
Battelle AMS Center Quality
Manager
NA1
Vendors
EPA AMS Center Project
Officer
EPA AMS Center Quality
Manager
EPA AMS Center Project
Officer
EPA AMS Center Quality
Manager
NA
Finals Distributed to:
EPA AMS Center
Project Officer
Appendix A
Stakeholders
EPA AMS Center
Project Officer
Appendix A
EPA AMS Center
Project Officer
Verification Test
records. First day's
data distributed for
EPA Review
Appendix A
-------
ETV Advanced Monitoring Systems Center
Quality Management Plan Version 8.0
Date: Aprils, 2011
Page 29 of 48
Record Type
ETV Verification
Report
ETV Verification
Statement
Battelle Audit
Reports
Preparation/
Updating
Battelle
Verification
Test
Coordinator
Battelle
Verification
Test
Coordinator
Battelle AMS
Center QAO
Review
Battelle AMS Center Manager
Battelle AMS Center Quality
Manager
EPA AMS Center Project
Officer
EPA AMS Center Quality
Manager
Vendor
Peer Reviewers
Battelle AMS Center Manager
Battelle AMS Center Quality
Manager
EPA AMS Center Project
Officer
EPA AMS Center Quality
Manager
Vendor
ETV Program Director
Peer Reviewers
Battelle AMS Center Manager
Battelle Verification Test
Coordinator
Approval
EPA AMS Center Project
Officer
EPA Laboratory Director
Battelle ENVS Product Line
Manager
EPA AMS Center Project
Officer
EPA AMS Center Project
Officer
Battelle AMS Center Quality
Manager
Finals Distributed to:
Appendix A
Appendix A
Appendix A
1NA = Indicates that approval is not applicable.
2These records will be provided to stakeholders and peer reviewers if deemed appropriate by the VTC.
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 30 of 48
7 COMPUTER HARDWARE AND SOFTWARE
Hardware and software used in support of AMS Center verification testing must be adequate to meet the
project requirements. Hardware and software capability needs should be assessed during the planning
process and specific needs defined in the QAPP. Hardware and software will be installed, maintained,
and used according to Battelle policy and procedures, as defined in the ENVS Product Line QMP.
The selection and use of hardware and software must conform to the requirements of the Battelle
Operations and Systems Security Information Management (IM) office. IM uses software patch
management and software version upgrade processes to ensure that the baseline software on a computer
connected to the Battelle network is kept up-to-date with the latest security patches and upgrades.
7.1 Hardware
All computer hardware at Battelle contains Intel based Pentium or Xeon processors running a Microsoft
operating system. Personal computers are leased on a 3-year replacement schedule to take advantage of
advances in technology, to ensure complete compatibility with current standards, and to provide adequate
resources to staff with the heaviest computational needs.
Computer hardware will be upgraded as needed to improve performance and provide complete
compatibility with current Battelle standards or program requirements. The decision to upgrade computer
hardware is made when a project that requires specific computer capabilities is received. Next, an
assessment of impact is completed. This assessment includes a review of current computer programs and
the impact or upgrading hardware on data accessibility.
7.2 Software
Each personal computer (PC) at Battelle is set up with standard complement of Microsoft software (e.g.,
Word, Excel, Access, PowerPoint, and Outlook) with capabilities of running other commercial software
(e.g., WordPerfect, Quattro, Acrobat, Lotus, etc.) and delivery of data in various standard formats.
Microsoft Windows 2007 is currently being installed on new leased computers during set-up. The need
other, specialized software to support a verification test will be identified in the QAPP.
Most software used at Battelle is acquired commercially and is loaded and tested by Battelle's IM
department as specified by the publisher. Software used for data management activities may include
Microsoft Excel or Access. Standard word processing software (e.g. Word) is used to create reports.
Independently-developed software is not used within the ETV AMS Center; only commercial products
are used.
7.2.1 Change Control
Change control involves tracking software and hardware versions used for specific environmental data
collection and management activities so that data can be reproduced using the version used to create it
(e.g., instrument software versions) and so that if problems or "bugs" are identified in specific hardware
or software versions, the affected data can be identified and fixed. Battelle's IM department tracks the
hardware configuration and software versions loaded on each PC during set-up.
The purpose of retesting and recalibration is to verify that a new system is compatible with the previous
system and that it will accommodate previous data entry and data reduction programs without impact to
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 31 of 48
existing data. In most cases testing will consist of review of existing documents or data to verify that
formats and output are not changed. For more complex applications, such as modeling or 3D algorithms,
a data set may be re-entered, tested, or plotted and the outputs compared. In these cases, testing results
should be documented in the project files. The performance of hardware and software may require
retesting if:
Hardware components are changed.
Software configuration is modified.
New or updated software is installed.
The VTC and technical lead for a verification test must be aware of hardware and software change control
and potential impacts of version differences. They are responsible for determining if and how retesting is
required and how retesting will be accomplished and documented.
7.2.2 Purchased Hardware and Software
Any specialized hardware or software purchased for a specific AMS Center applicable must be evaluated
to ensure that it meets the intended use requirements and that it complies with any contractual
requirements and standard. The evaluation process will be defined in the QAPP and/or documented in the
project files.
7.2.3 Validation Policy
Since all hardware and software used for the ETV AMS Center is commercially available, wide public
use and continued market viability is considered proof of software dependability; therefore, validation is
not considered necessary. However, verification of data analysis techniques within each program (e.g. the
use of formulas and macros) is required. For example, spreadsheet formulas and output shall be
independently reviewed, verified, and documented.
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 32 of 48
8 PLANNING
Systematic, timely, and effective planning is necessary to assure that data and information collected for
AMS Center verification testing meet the objectives and quality for their intended use.
8.1 AMS Center Systematic Planning
The overall planning process for the AMS Center involves the identification of potential technologies for
testing, establishing the test objectives, determining the testing logistics, identifying the data needed to
meet the test objectives, and defining the criteria for assessing the quality and usability of test data. This
planning process will establish the framework to define requirements for testing QA/QC, data collection,
and data analysis and evaluation. An overview of the EPA ETV verification process is shown in Figure 2.
Verification test planning shall be coordinated by Battelle among the participating organizations including
EPA, the stakeholders, the vendors, and any testing organizations and laboratories participating in the test.
Battelle, with the concurrence and oversight of the EPA AMS Center Project Officer, shall identify the
planning roles of the participants, and shall conduct planning activities by shared communication via e-
mail, teleconference, video conference, and in-person meetings, as appropriate, and within the constraints
of budget.
8.1.1 Stakeholders
Stakeholder committee(s) or technical panels with representatives of appropriate technology interest
groups are jointly established by the EPA AMS Center Project Officer and the AMS Center Stakeholder
Coordinator (Section 2.2.4). Individual stakeholders are selected for these committee(s) based on their
expertise and interest in environmental monitoring and their availability and willingness to participate.
A joint meeting of the EPA AMS Center Project Officer, Battelle, and each stakeholder committee or
technical panel will be held at least once annually, with meetings minutes recorded, reviewed, and
circulated to the stakeholders and the EPA AMS Center Project Officer. The meeting can be conducted in
person or by teleconference. The purposes of the stakeholder or technical panel meetings include:
identify, revise, and/or clarify the technical and quality goals of the work to be accomplished;
determine testing priorities;
discuss test design;
define and review verification plans;
identify verification test collaborators.
-------
ETV Advanced Monitoring Systems Center
Quality Management Plan Version 8.0
Date: Aprils, 2011
Page 33 of 48
Identification of Technology
Categories for Testing
(Stakeholder/EPA/Battelle)
Recruitment of Vendors/Collaborators
(Battelle)
J
Draft Test QAPP
(Battelle/Vendors/Stakeholders/Collaborators
Vendor Review of Test QAPP
(Battelle)
Peer and QA Review of
Revised Test QAPP
Coordination with Host-Facility/
Subcontractors/Collaborators
Final Test QAPP
(Battelle)
Finalize Host/Subcontractor/
Subcontractor Roles
(Battelle)
To Testing
Figure 2. Systematic Planning of Verification Tests
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 34 of 48
8.1.2 Systematic Planning of Verification Tests
The AMS Center planning process will establish the verification test details, including:
goals, data quality objectives, and data quality indicators;
project schedule, resource needs, milestones, and requirements;
type and quantity of data needed and how the data will be used to support the data quality objectives;
data quality performance criteria;
QA/QC activities to assess the quality performance criteria;
data collection methods and logistics;
data analysis and evaluation procedures.
The test details will be documented in the verification test QAPP.
8.1.3 Data Quality Objectives
The design process for verification testing establishes the test data quality objectives (DQOs). These
ensure that the collected data are of sufficient type, quality, and quantity to answer the study questions
(typically expressed relative to the ability to estimate an unknown parameter within specified bounds or
make a correct decision within a certain degree of confidence).
Once the DQOs are established, the experimental design will be developed. The QAPP will define the
verification test to be conducted, the baseline parameters, the number of replicate tests, and the controls.
8.1.4 Data Quality Indicators
Data quality indicators (DQIs) will be defined in the QAPP for each test objective. DQIs are a set of
measurable characteristics0 that address the quality of data at the field and lab analytical level. DQIs have
some influence on determining whether the DQOs have been met, as they help define the level of quality
in the data. The QAPP must define the DQIs appropriate for the verification test objectives and the
measurement quality objectives (MQOs; the actual acceptance criteria) placed on the DQIs. The MQOs
will be used during data assessment to determine whether the quality of a data set is acceptable relative to
the DQIs. Table 2 illustrates the relationship between DQOs, DQIs, and MQOs. Appendix D provides
definitions of DQIs and the types of QC samples used to measures them.
The test objectives and DQIs will establish the criteria for the selection of field and laboratory procedures,
methods, and equipment. DQIs are defined in AMS Center QAPPs for reference data as well as for
ancillary measurements that support verification test data when possible. DQIs are typically not required
in AMS Center QAPPs for vendor technology data, since the vendor is responsible for specifying the
quality measurements to be made to ensure the integrity of their technology's outputs.
Considerations include reliability under the intended field conditions or sample matrices, detection limits,
specificity, and sensitivity. For all test activities critical to the achievement of the DQOs, the QAPP will
detail:
equipment for each field activity and measurement;
analytical methods for each laboratory procedure;
0 DQIs are typically defined as the PARCC parameters: precision, accuracy, representativeness, comparability, and
completeness.
-------
ETV Advanced Monitoring Systems Center
Quality Management Plan Version 8.0
Date: Aprils, 2011
Page 35 of 48
sampling and data collection procedures;
calibration, operation, and maintenance requirements;
QC samples and procedures to be implemented in the field and laboratory and MQOs for each DQI.
Whenever possible these details should be described in SOPs that are developed by the sampling and
testing team or reference laboratory. SOPs are discussed in Sections 6.1.2 and 9.2.
Table 2. The Relationship between DQOs, DQIs, and MQOs
Data Quality Objective
(DQO) -ป.
Qualitative and quantitative
study objectives
How 'good' does the
study data have to
be?
How many samples
are needed to
determine
Eg.,
50 samples are
needed to achieve
desired level of
confidence (ฑ30%)
that the attribute is
correctly characterized
Data Quality Indicator
(DQI)->
Quantitative:
Precision
Accuracy
Sensitivity
Qualitative:
Representativeness
Comparability
Completeness
E.g.,
Precision
Accuracy
Measurement Quality Objective (MQO)
Project specific acceptance criteria for the DQIs
Eg.,
Laboratory duplicates precision < 10%
RPD
Blank spike accuracy ฑ15%
8.2 Quality Assurance Project Plans
A QAPP is prepared for each verification test by the Battelle VTC. The QAPP format and content is
based on EPA QA R-5 EPA, 2001. Elements of the QAPP are defined in Appendix E. The QAPP
describes the details of the test design required to meet the data quality objectives established during the
planning process.
Verification tests are designed using the systematic planning process shown in Figure 2.0. The design
takes into account time, scheduling, and resource constraints. All relevant activities pertaining to
environmental data operations must be detailed in the QAPP, including performance specifications, the
appropriate controls, and the statistical methods used to establish both the design and evaluation of test
results, as appropriate.
8.2.1 Sample Collection Procedures
The QAPP must define the procedures for samples collected in the field to ensure handling, storage,
cleaning, packaging, shipping, and preservation of field and laboratory samples are adequate to prevent
damage, loss, deterioration, artifacts, or interferences. Topics to be covered include:
sampling methods, sample types, numbers, quantities, handling, packaging, shipping, and custody (if
sampling is performed);
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 36 of 48
sample locations, storage conditions, and holding times;
protection of the health and safety of testing personnel and the public (unless a stand-alone health and
safety plan is prepared);
procedures for the minimization and disposal of waste generated.
8.2.2 Documentation Requirements
The QAPP will define what records and observations will be documented during testing and where and
how those records and observations will be recorded. Documentation must follow the requirements of
Section 6.2.1.
8.2.3 Reference Methods
Verification testing that involves the collection of analytical data by a technology should incorporate
parallel analysis of samples by a reference laboratory using standard methods or benchmark
measurements. For these tests, the results of the reference or benchmark measures will typically be used
to assess the performance of the technology.
8.2.4 Conformance to the QAPP
All verification test activities must conform to the requirements of the QAPP.
Changes to the approved QAPP that are made during the test are deviations. Deviations are documented
using the deviation form (Appendix C). The notification and documentation period for deviations is
defined in Section 6.3.
Changes to the approved QAPP that are made before testing begins, or between rounds or phases of
testing, are amendments. Amendments are documented using the amendment form (Appendix C).
Significant changes to the QAPP may warrant a reversion of the QAPP. The EPA AMS Center Quality
Manager makes the determination as to whether the changes are significant. Revisions are reviewed and
approved according to Section 6.1.3.
Generation of verification test data will not be initiated until the approved QAPP is in place. If
verification test data are generated prior to full QAPP approval, a QMP deviation must be prepared and
approved. The deviation form in Appendix C is used to document this deviation.
8.3 Generic Verification Protocols
The generic verification protocol is meant to promote uniform testing for a technology category, and is
therefore a more general document than a test-specific QAPP. The Battelle AMS Center Manager will be
responsible for assuring that generic verification protocols are prepared and transferred to the EPA AMS
Center Project Officer and stakeholders for review. The issues that may be addressed in the verification
protocol are the following:
general description of the Center;
responsibilities of all involved organizations;
experimental design;
equipment capabilities and description;
description and use of field test sites;
description and use of laboratory test sites;
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 37 of 48
QA/QC;
data handling;
requirements for other documents;
health and safety;
references.
8.4 Existing Data
Existing data are defined as data from databases, data resulting from previous projects, the scientific
literature, or other sources. If existing data are used to support verification testing, examination to
determine whether the data are suitable for their intended to support verification testing. Existing data
can be used for informational purposes to support development of the verification test design. If existing
data are to be used as verification test data for an AMS Center test, then the data must meet the
requirements specified in Appendix C of the ETV QMP for existing data.
8.5 Roles and Responsibilities
The Battelle AMS Center Manager is responsible for directing the activities of the Center in collaboration
with the EPA AMS Center Project Officer. The Battelle AMS Center Quality Manager is responsible for
developing and maintaining the AMS Center QMP (this document). The VTC is responsible for
preparation of QAPPs, amendments, and deviations in compliance with the AMS Center QMP. QAPPs,
amendments, and deviations are reviewed and approved by the Battelle AMS Center Manager, Battelle
AMS Center Quality Manager, EPA AMS Center Project Officer, and EPA AMS Center Quality
Manager.
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 38 of 48
IMPLEMENTATION OF WORK PROCESSES
9.1 Performance of Verification Test Activities
Technology performance verification tests must be implemented according to approved QAPPs and other
supporting documents referenced in the QAPP (e.g., SOPs, instruction manuals). When a prescribed
sequence for the work is defined during the planning stages, work must follow that sequence.
9.1.1 Kick-off Meetings
A kick-off meeting will be held prior to the start of each verification test to review procedures for the test
with all verification testing staff, the Battelle AMS Center Manager, the Battelle AMS Center Quality
Manager, the EPA AMS Center Project Officer, and the EPA AMS Center Quality Manager. The kick-
off meeting should be conducted after the QAPP has been approved, signed by the vendor, and distributed
to the team. The VTC leads the kickoff meetings. The purpose of the kick-off meeting is to describe the
test background, objectives, schedule, logistics, QA/QC requirements, and communication procedures.
During the kick-off meeting staff may seek clarification of test details or identify logistical concerns.
During the kick-off meeting, the EPA AMS Center Project Officer and the EPA AMS Center Quality
Manager may expand on any discussion item as needed to ensure that EPA's concerns and insights are
communicated to the participants. The kick-off meeting checklist is provided in Appendix F. In some
cases, a separate meeting may be held with additional staff (e.g. collaborators, external personnel,
reference laboratory, etc.) if warranted.
9.1.2 Communicating Verification Test Requirements
A copy of the approved QAPP and supporting documents will be distributed to test personnel prior to
testing. These documents must be immediately available at locations where testing, sampling, and
analysis in support of the verification test are being performed.
Any changes to the QAPP required during testing must be documented by deviation. Deviations are
distributed to Battelle AMS Center Manager, Battelle AMS Center Quality Manager, the EPA AMS
Center Project Officer, and the EPA AMS Center Quality Manager. If the changes described in the
deviation are for activities not yet completed then the deviation should also be distributed to appropriate
test personnel.
QAPP amendments must be distributed to anyone who received the original QAPP.
Deviations must be retained in the verification test records and summarized in the verification test report.
Frequent deviations from established procedures should result in a retrospective review of the written
document and possible revision. Amendments and deviations will include all the information displayed
on the form shown in Appendix C, including impact to the test data.
All test activities must be documented as described in Section 6.2.1 and the QAPP.
9.2 Identifying the Need for Special Procedures
The QAPP and documents referenced therein must define the procedures for verification testing. The
level of detail must be adequate for a qualified individual to perform the procedure independently. In
some cases, the level of detail needed to adequately describe a procedure is best documented in a
specialized, detailed SOP or protocol.
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 39 of 48
If the procedure is routine then an SOP should be developed by the procedure 'owner.' As described in
Section 6.1.2, SOPs are controlled documents and formally approved and tracked.
If the procedure is test-specific then a protocol is adequate. Protocols are test-specific documents
prepared by the technical lead or delegate that are approved by the technical lead, VTC, and the AMS
Center Quality Manager. Protocols must be version-controlled in some manner.
A special procedure requiring a detailed SOP or protocol is required if the procedure:
is complex, involves many steps, and must be completed follow a regime of timing and sequences (e.g.,
analytical methods);
is lengthy and is best captured in a separate document;
involves the quantitative preparation of chemical solutions that impact data quality (e.g., calibration
solutions);
involves the setup, use, and operation of complex equipment or instruments that will impact data quality
and that are not easily operating by following simple manufacturer instructions (analytical instruments).
The VTC is responsible for identifying the need for special written documents to describe a procedure.
The AMS Center Quality Manager is responsible for ensuring that all technical procedures described in
the QAPP are adequately described in a written procedure.
9.3 Document Control
It is critical that test personnel have access to the current, approved versions of each document required
for verification testing and that obsolete documents are not available for inadvertent use. Section 6.1
describes the management of controlled documents for the AMS Center. The procedures that will be
implemented by the AMS Center to ensure test-specific document control include the following:
The final, approved QAPP is distributed to verification test participants prior to testing.
Any QAPP amendments and deviations (that describe QAPP changes to be implemented) are distributed
to the full QAPP distribution list once approved.
Instrument manuals will be maintained with the instruments.
QAPPs, amendments, and deviations (that describe QAPP changes to be implemented) must be physically
available at each technology verification testing site.
SOPs or protocols developed for testing will be distributed to personnel who will be conducting the
procedures.
When a QAPP, amendment, deviation (that describes QAPP changes to be implemented), manual, SOP,
or protocol is revised, it will be formally distributed to each test participant. Test participants are
responsible for immediately removing previous document versions from their work area or saving them in
the project files as a historical record. The VTC is responsible for maintaining all versions of all test
documents to maintain a history of project activities.
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 40 of 48
10 ASSESSMENT AND RESPONSE
Assessments are planned, scheduled, conducted reported, and tracked to closure by the AMS Center
Quality Manager or appointed QA Officers to determine the suitability and effectiveness of the quality
system overall and the quality of the testing performed for each verification test. Appendix B delineates
the responsibilities of the AMS Center Quality Manager and QA Officers.
10.1 Quality System Assessment
The Battelle AMS Center Manager and Battelle AMS Center Quality Manager assess the adequacy of the
quality system formally during the review and update of the QMP (Section 6.1.1). As part of the review
they consider elements are that are adequate to achieve quality results, problems that need further control,
and requirements that are obsolete or superseded with other requirements. This same assessment process
is conducted during the review of each assessment report, as part of responses to comments by the EPA
AMS Center Project Officer and EPA AMS Center Quality Manager, and during the development of
QAPP amendments or deviations. Systematic issues that require modification to procedures are
announced to the AMS Center team using the Share Point Announcement feature. Further, lessons learned
are tabulated by the Battelle AMS Center Quality Manager and published on the AMS Center SharePoint
site.
An external quality system audit of the Battelle quality system is expected to be performed at least once
by the EPA AMS Center Quality Manager. In addition, an independent technical systems audit (TSA)
will be performed by the EPA AMS Center Quality Manager or designee, at least one time per year for
the AMS Center.
10.2 Assessment Procedures
The assessment process involves planning, implementation, documenting the assessment, and preparing
the assessment report. Assessments schedules must be conducted so that real-time intervention can be
implemented if needed. Section Cl of each QAPP will include a detailed description of the TSA and
ADQ schedules. Three types of assessments are typically conducted for each verification test. The
assessments are discussed below. Section of D of each QAPP will provide the test-specific assessment
details.
10.2.1 Performance Evaluation Audit
PE audits are a quantitative evaluation of a measurement system. PE audits should be conducted
whenever a reference method is available or whenever a technology will measure a parameter for which a
reference sample is available.
The type and frequency of PE are specified in the QAPP for each verification test.
The value or composition of reference materials must be certified or verified prior to use, and the
certification or verification must be adequately documented.
The Battelle AMS Center QAO will review results of PE audits.
10.2.2 Technical Systems Audits
A TSA is a qualitative on-site evaluation of sampling and/or measurement systems associated with a
particular verification test. The objective of the TSA is to assess and document the conformance of on-
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 41 of 48
site testing procedures with the requirements of the QAPP and associated SOPs. The TSA may assess
test facilities, equipment maintenance and calibration procedures, reporting requirements, sample
collection, analytical activities, and QC procedures.
The Battelle AMS Center Quality Manager or designated QA Officer conducts a TSA at least once
during each verification test.
A TSA checklist based on the QAPP is prepared prior to the assessment by the Battelle AMS Center QA
Officer, reviewed by the Battelle AMS Center Quality Manager and Battelle AMS Center Manager, and
submitted to the EPA AMS Center Project Officer and EPA AMS Center Quality Manager. EPA
approval of the checklist is not required prior to the TSA but any comments will be incorporated.
For each TSA, an assessment kickoff meeting should be conducted with the verification testing team
when possible and appropriate so that the team understands the assessment logistics. The agenda should
include at least the following information:
o Schedule of assessment(s),
o Proper completion of data records,
o Notification to affected parties,
o Specific assessment requirements (personnel lists, equipment lists, and availability of
QAPPs),
o Follow-up procedures for corrective action, including debriefing and discussion of possible
resolutions,
o Corrective action guidelines to facilitate completion of the reported assessment.
At the close of the TSA, an immediate informal debriefing will be conducted. A formal debriefing that
includes the Battelle AMS Center Quality Manager or designed QA Officer, the Battelle AMS Center
Manager, the VTC, the on-site team that was audited, the EPA AMS Center Project Officer, and EPA
AMS Center Quality Manager should be conducted when possible.
The EPA AMS Center Quality Manager conducts an independent TSA once per year, as applicable, for
the AMS Center.
The results of TSAs will be documented in a formal audit report. The format of the report is located on
the AMS Center SharePoint site.
The TSA report schedule is as follows:
o The draft TSA report with the completed checklist will be submitted to the VTC and EPA
within 10 days of the TSA.
o The VTC audit response is due 10 working days from delivery of the TSA report.
o The final TSA, with audit responses, is due to EPA within 10 days of receiving the response.
The final report with VTC responses accepted by the test QA Officer will be signed, scanned, and
uploaded to the ETV database.
10.2.3 Audit of Data Quality
An ADQ is a quantitative evaluation of the verification test datad. The objective of the ADQ is to
determine if the test data were collected according to the requirements of the QAPP and associated SOPs.
The ADQ assesses data accuracy, completeness, quality, and traceability.
d The ADQ verifies data. Data verification is defined as evaluating the completeness, correctness, and conformance
of the data set vs. the QAPP and method requirements.
-------
ETV Advanced Monitoring Systems Center
Quality Management Plan Version 8.0
Date: Aprils, 2011
Page 42 of 48
The ADQ is conducted after data have been 100% verified by the VTC or designated project personnel.
Battelle has established internal SharePoint sites with automated notifications for all active verification
tests so the QA Officer is notified when data are posted and available for audit.
The Battelle AMS Center QA Officer conducts the ADQ.
An ADQ checklist based on the QAPP is prepared prior to the assessment by the Battelle AMS Center
QA Officer, reviewed by the Battelle AMS Center Quality Manager, Battelle AMS Center Manager, and
submitted to the EPA AMS Center Project Officer and EPA AMS Center Quality Manager. EPA
approval of the checklist is not required prior to the ADQ but any comments will be incorporated.
The amount of data reviewed during the ADQ, and the reporting frequency, is dependent on whether the
verification test is designated as Category II or Category III by the EPA AMS Center Project Officer.
The Category level is defined in the QAPP. The final ADQ will assess the accuracy of the verification
report vs. the collected and compiled data. The audit requirements are summarized below in Table 3.
Table 3. AMS Center ADQ Audit Requirements
ADQ Elements
Reference Method Calibration
Reference Method QC
First batch of posted data
Report to VTC and EPA
Subsequent data
Verification Report
Category II
100%
100%
25%
10 days from data receipt
25%
25%
Category III
100%
100%
10%
15 days from data receipt
10%
10%
The reporting frequency for ADQs will be defined in QAPP based on the frequency of data reporting, the
test duration, and the presence/absence of data quality issues identified during the audits.
Problems that could impact data quality are immediately communicated to the VTC and the Battelle AMS
Center Manager.
The results of the ADQ will be documented in a formal audit report. The format of the report is located
on the AMS Center SharePoint site.
o The draft ADQ report with the completed checklist will be submitted to the VTC and EPA
within 10 days of the ADQ.
o The VTC audit response is due 10 working days from delivery of the ADQ report.
o The final ADQ, with audit responses, is due to EPA within 10 days of receiving the response.
The final report with VTC responses accepted by the test QA Officer will be signed, scanned, and
uploaded to the ETV database.
10.2.4 Data Validation
Data validation assesses the overall quality of a data set based on the MQOs. Data validation is initially
conducted by the VTC during the data review process. The VTC review includes verifying that:
the raw data records are complete, understandable, well-labeled, and traceable;
all data identified in the QAPP has been collected;
instrument calibration and QC criteria were achieved;
data calculations are accurate.
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 43 of 48
The VTC review may deem a data set unusable, questionable, or semi-quantitative, based on the results of
the QC data and achievement of the DQIs.
Data validation is also conducted during the ADQ when the AMS Center QA Officer reviews the data vs.
QAPP requirements and assessed overall data quality. The ADQ verifies a percentage (Table 3) of the
reported data vs. raw data, including any calculations. In addition, during the ADQ a checklist based on
the QAPP assesses, as appropriate
data completeness;
sample handling, holding times, and integrity;
instrument calibration;
quality control; and
documentation.
Any limitations on the data and recommendations for limitations on data usability are documented in the
data audit report.
10.2.5 Data Assessment
The VTC assesses data usability during the review of test data. This assessment typically includes a
statistical and scientific evaluation of the data to determine the validity of the test design and the
performance of the technology vs. the performance measures specified in the QAPP design process.
The draft and final Verification Report is prepared by the VTC. The report assesses how the technology
performed under the test conditions according to the verification parameters defined in the QAPP.
Verification reports are peer-reviewed by external reviewers and verification statements are signed by an
EPA laboratory director and Battelle management.
Any assessment findings that impact the quality of verification test data will be summarized in the
verification test report
Any limitations on the data and recommendations for limitations on data usability are documented in the
ETV verification report.
10.3 Qualifications of Assessors
Personnel who perform the AMS Center assessments must be QA professionals with specific training in
QA/QC principles and procedures.
Assessors must be independent of the work being reviewed, free of conflict of interest, knowledgeable in
the area being assessed, and trained in assessment techniques.
The principal Battelle assessor will be the Battelle AMS Center Quality Manager, who will have an
extensive QA laboratory and field inspection, technical, and management experience, and who will be
familiar with the AMS Center assessment requirements.
Should the need arise, the Battelle AMS Center Quality Manager will designate other individual to be QA
Officers for specific verification tests or to perform scheduled assessments, based upon that person's
technical skill and knowledge of the technology being tested and testing schedule.
The Battelle AMS Center Quality Manager is responsible for the assessments performed by other
personnel and will review and approve all audit reports. The Battelle AMS Center Manager is ultimately
responsible for the quality of the AMS Center performance and will approve all VTC audit responses.
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 44 of 48
10.4 Authority of Assessors
By virtue of their designation as QA personnel at Battelle, the Battelle AMS Center Quality Manager and
QA Officers have the responsibility and authority to:
identify, document, and report problems affecting the quality of verification results;
identify noteworthy practices and encourage their implementation as continuous improvement activities;
propose recommendations for resolving these quality problems;
independently confirm implementation and effectiveness of solutions;
access records and appropriate managers.
recommend that work during a verification test be stopped if safety and quality are threatened.
o The Battelle AMS Center Manager and VTC must ensure compliance with all applicable
federal, state, and local safety policies during the performance of verification testing.
o The Battelle AMS Center Manager shall then direct the AMS Center staff accordingly.
o Any AMS Center staff member can contact the VTC if they suspect that personal health or
test objectives during the conduct of verification testing are being compromised. The VTC is
responsible for immediately notifying Battelle AMS Center Manager.
o The EPA AMS Center Quality Manager is delegated to notify the EPA AMS Center Project
Officer who will notify the Battelle AMS Center Manager to facilitate a stop work order if
work of inadequate quality is discovered.
o Documentation is required of any stop work order and the corrective action implemented and
shall be maintained as part of the Battelle quality records, with a copy provided to the EPA
AMS Center Project Officer and EPA AMS Center Quality Manager.
10.5 Reports to Management
The results of all AMS Center QA assessments are documented in written reports to the VTC and the
Battelle AMS Center Manager. Assessment results are categorized as acceptable or not acceptable.
Unacceptable results are further categorized as findings or observations. Findings are non-conformances
at the project level that will have a significant adverse effect on quality. Observations are non-
conformances at the project level that will not have a significant adverse effect on quality.
The Battelle AMS Center Manager receives the initial assessment reports and is thus apprised of any
issues identified during the audit.
The Battelle AMS Center Manager reviews and approves the VTC responses and thus ensures that
responses are thorough, fully address the audit findings and observations, and thoughtfully assess any
impact to testing.
The EPA AMS Center Project Officer and EPA AMS Center Quality Manager receive the draft and final
assessment reports.
Any comments received from management that do not change the intent of an audit finding or observation
will be addressed by the author.
Distribution of assessment reports is described in Section 10.3 and Appendix A.
As needed, issues requiring management action are reported during management meetings and project
reviews.
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 45 of 48
10.6 Corrective Action
Corrective action is implemented in response to any situation that compromises the quality of testing or
data generated by the AMS Center. The need for corrective action can be identified by any AMS Center
personnel and implemented with the prior approval of the Battelle AMS Center Manager, Battelle AMS
Center Quality Manager, or VTC.
Corrective action is required for all assessment findings and observations.
The VTC is responsible for determining appropriate corrective action to address an issue. The corrective
action should minimize the chance that a problem adverse to quality will re-occur. The corrective action
will be documented by the VTC on the assessment report.
The VTC is responsible for ensuring that corrective actions are implemented as documented in the
assessment report.
Implementation of corrective actions must be verified by the Battelle AMS Center Quality Manager or
QA Officer who prepared the assessment report to ensure that corrective actions are adequate and have
been completed.
Verification of corrective actions can be by re-assessment or examination of documentation.
The assessment report cannot be finalized until each corrective action has been identified and verified.
Staff may request corrective action through the on-line Corrective Action Logger
http://wwwi.duxbury.battelle.org/CAL/ (SOP 4-035). This tool enables entry of a request for corrective
action, sends an email notification to the person responsible, their manager, and the QA Manager, and
tracks responses and approvals.
The corrective action process should include an assessment of the root cause of a problem so that
effective changes can be implemented to minimize reoccurrence. Once the root cause determination is
verified, appropriate actions can be planned, documented, and implemented by the AMS Center staff.
Any finding that is a QAPP deviation must be documented as described in Section 8.2.4.
10.7 Dispute Resolution
If an audit finding or response creates a dispute that cannot be resolved by the QA Officer and VTC, the
dispute will be elevated to the Battelle AMS Center Quality Manager and Battelle AMS Center Manager.
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 46 of 48
11 QUALITY IMPROVEMENT
The AMS Center is committed to a process of continuous quality system improvement. While every
member of the AMS Center team is encouraged to contribute to quality improvement initiatives, the
Battelle AMS Center Manager and Battelle AMS Center Quality Manager are specifically responsible for
identifying opportunities to improve the quality system. The purpose of quality system improvement is to
ensure that conditions adverse to quality are prevented, identified, corrected, documented, and tracked.
11.1 Annual QMP Review
The QMP for the AMS Center will be conducted annually by the Battelle AMS Center Quality Manager
and technical and management staff in order to identify and incorporate improvements to the quality
system (Section 6.1.1). Action items identified during the review will be documented by inclusion in the
revised QMP.
11.2 Problem Identification and Resolution
All staff are encouraged to identify problems and offer solutions to problems in the following quality
areas:
adequacy of the quality system, as defined in the QMP;
consistency of the quality system;
implementation of the quality system to specific verification tests;
correction of quality system procedures;
completeness of documented information;
quality of data;
quality of planning documents, such as the QAPPs;
implementation of the work process.
Suggests are received by the Battelle AMS Center Manager and Battelle AMS Center Quality Manager.
No formal tracking system has been developed because suggestions are typically implemented in near-
real time. The process for identifying and implementing improvements includes the following:
o Improvements to QAPPs, reports, VSs, and audit reports are made on the next new document of
each type, which then becomes the template for future documents.
o Suggestions for improving sampling designs or test logistics are communicated during AMS
Center team meetings.
o The Battelle ETV SharePoint site uses the Announcement feature to communicate improvements,
lessons learned, and new instructions to the Battelle AMS Center team.
o Any staff member may request a corrective action (Section 10.6) if an on-going problem or
systematic issue is identified.
11.3 Assessments
TSAs and ADQs serve as tools to determine cause and effect relations of significant problems that might
require testing protocol, management system, or quality system changes. Monitoring and evaluation by
the AMS Center Quality Manager, for example, may indicate trends or common and recurring problems
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 47 of 48
for an entire technology evaluation. In this case, the situation is immediately communicated to the
Battelle AMS Center Manager and an appropriate corrective action identified.
11.4 Customer Satisfaction and Feedback
Battelle conducts customer satisfaction surveys to obtain feedback and identify areas for improvement.
Action items are reviewed by all levels of management each quarter and improvement opportunities are
identified. Quality processes are continually monitored and both short-term and long-term quality issues
are identified through customer feedback and client involvement, peer review and internal lessons
learned, and program review.
11.5 Communication of Quality System Improvements
Quality system improvements identified through any of the venues described above are announced to the
AMS Center team using the SharePoint Announcement feature.
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Page 48 of 48
12 REFERENCES
ANSI/ASQ. 1994. Specifications and Guidelines for Quality Systems for Environmental Data Collection
and Environmental Technology Programs. E4-1994.
ANSI/ASQ. 2004. Quality Systems for Environmental Data and Technology Programs - Requirements
with Guidance for Use.
Battelle. 2009. Environmental Product Line Quality Management Plan. Revision 0. October 2009.
Environmental Protection Agency (EPA). 2008. Environmental Technology Verification Program
Quality Management Plan. Version 3.0. January 2008.
Environmental Protection Agency (EPA). 200la. EPA Requirements for Quality Management Plans.
EPAQA/R-2. EPA/240/B-01/002. Office of Environmental Information. March.
Environmental Protection Agency (EPA). 2001b. EPA Requirements for Quality Assurance Project
Plans. EPAQA/R-5. EPA/240/B-01/003. Office of Environmental Information. March.
Environmental Protection Agency (EPA). 2006a. Guidance on Systematic Planning Using the Data
Quality Objectives Process. EPA QA/G-4. EPA/240/B_06/001. Office of Environmental
Information. February. Available at: http://www.epa.gov/quality/qs-docs/g4-final.pdf
-------
APPENDIX A
KEY EPA AND BATTELLE AMS CENTER STAFF AND DISTRIBUTION LIST
-------
KEY EPA AND BATTELLE AMS CENTER STAFF AND DISTRIBUTION LIST
ROLE
Frequency of
Preparation or
Review
EPA AMS Center
Project Officer
EPA AMS Center
Quality Manager
Battelle Center
Manager
Battelle Quality
Manager
Verification
Testing Leader
Stakeholder
Coordinator
NAME1
Dr. John
McKernan
Ms. Amy
Dindal
Ms.
Rosanna
Buhl
Dr.
Thomas
Kelly
Ms.
Rachel
Sell
CONTACT INFORMATION
26 W. Martin Luther King Dr., MS 208
Cincinnati, OH
45268
Ph: 513-569-7415
Fx: 513-569-7158
McKernan.John@epa.gov
1801 Waldorf Dr.
Royal Palm Beach, FL 3341 1
Ph: 56 1-422-01 13
Fx: 614-458-6697
dindala@battelle.orq
397 Washington St.
Duxbury, MA 02332
Ph: 781-952-5309
Fx: 781-934-2124
buhl@battelle.ora
505 King Ave.
Columbus, OH 43201
Phone: 614-424-3495
Fax: 614-458-3495
kellyt@battelle.orq
4009 Rondeau Dr.
San Jose, CA
95124-4724
a.
a
Annual
/
/
S
/
S
S
Master Distribution List
QAPPs,
Amendments, &
Deviations
As needed
S
S
S
S
General AMS
Center
Announcements2
Daily as needed
S
S
S
S
Verification Test
Correspondence,
E-mails, & Data3
Daily as
needed
S
S
S
S
a
O
< -e
~ฐ 2
c Q-
<<*
CO
^
^
^
^
Verification
Reports &
Statements
At
completion
of testing
^
^
^
^
o w
13
t' >
& <">
I a
If
3 B.
2 o
ft tOp
^ ^
" M
JT' r4-
O ft
0 3
po w
o O
ft
It
-------
ROLE
VTC
Verification Test
QA Officer
Verification Test
Technical Staff
Vendors
Stakeholders
NAME1
Test-
specific
Test-
specific
Test-
specific
Test-
specific
Test-
specific
CONTACT INFORMATION
Ph: 408-728-7008
Fx: 6 14- 458-0728
sellr(o)battelle.ora
Defined in QAPP
Defined in QAPP
Defined in QAPP
Defined in QAPP
Defined in QAPP
a.
a
Master Distribution List
QAPPs,
Amendments, &
Deviations
,
/
/
QAPP Only
QAPP Only
General AMS
Center
Announcements2
,
/
/
Verification Test
Correspondence,
E-mails, & Data3
,
/
/
ง
< -e
~ฐ a.
CO
J
/
"m *" &
/
s
'
o w
13
t' >
& o
I a
If
3 B.
2 o
-------
APPENDIX B
PERSONNEL RESPONSIBILITIES FOR AMS CENTER VERIFICATION TEST ACTIVITIES
-------
ETV Advanced Monitoring Systems Center
Quality Management Plan Version 8.0
Date: Aprils, 2011
Appendix B-l
Personnel Responsibilities for AMS Center Verification Test Activities
AMS Center Team
Member
Responsibilities
Authority
EPA AMS Center
Project Officer
Review the draft QAPP.
Approve the final QAPP.
Review and approve deviations to the approved final QAPP.
Appoint a delegate to review and approve deviations to the
approved final QAPP in his absence, so that testing progress
will not be delayed.
Review the first day of data from the verification test and
provide immediate comments if concerns are identified.
Review the draft verification report and statement.
Oversee the EPA review process for the verification report and
statement.
Coordinate the submission of verification report and statement
for final EPA approval.
Approve QAPPs, audit reports,
deviations, and verification
reports/statements.
Authorized to suspend work if data
quality or project objectives are
jeopardized.
EPA AMS Center
Quality Manager
Review the draft QAPP.
Review the first day of data from the verification test and
provide immediate comments if concerns are identified.
Perform at her option an external ISA and/or audit of data
quality during the verification test.
Prepare and distribute an assessment report summarizing
results of the external audit.
Notify the EPA AMS Center Manager of the need for a stop
work order if the external audit indicates that data quality or
safety is being compromised.
Review the draft verification report and statement.
Approve the QAPP, audit reports,
deviations, and verification
statements.
Authorized to suspend work if data
quality or project objectives are
jeopardized.
Battelle AMS Center
Manager
Ultimate responsibility for all aspects of the AMS Center,
including meeting all technical, budget, and schedule goals for
the Center.
Ensure that necessary Battelle resources, including staff and
facilities, are committed to the verification test.
Conduct and oversee activities to establish and maintain active
stakeholder committees
Maintain communication with EPA's AMS Center Project
Officer and QM.
Operate ETV activities within the documented quality system
Oversee vendor solicitations
Ensure that confidentiality of sensitive vendor information is
maintained.
Manage oversight and conduct of verification activities
Assure that quality procedures are incorporated and
implemented during verification activities
Assign VTC for each test
Approve the Battelle release of
QAPPs, audit reports, deviations,
and verification reports/statements.
Authorized to suspend work for
cause if data quality or staff safety
are threatened.
Authorized to delegate specific
responsibilities to other qualified
individuals
-------
ETV Advanced Monitoring Systems Center
Quality Management Plan Version 8.0
Date: Aprils, 2011
Appendix B-2
Personnel Responsibilities for the AMS Center for Verification Testing Activities
AMS Center Team
Member
Responsibilities
Authority
Battelle Verification
Testing Leader
Coordinate planning, performance, and data reviews of
technology verification testing consistent with the AMS Center
QMP requirements
Coordinate review of applications from technology vendors
wanting to have their technology verified
Work with stakeholders and EPA to identify and prioritize
technologies for verification
Schedule verification tests
Provide recommendations to the AMS Center Manager for
verification teams to perform specific technology verification
test/data reviews.
Review QAPPs, and amendments and deviations to QAPPs
Prepare and review verification reports/statements
Oversee/assist in problem resolution involving verification tests
Serve as AMS Center Manager
when designated to perform this
role
Battelle Stakeholder
Coordinator
Stakeholder recruitment, communications, and stakeholder
meeting facilitation.
Battelle AMS Center
Quality Manager
Ensure that the AMS Center quality system is compliant with
EPA-specified standards
Prepare a draft and final QMP that describes the quality
system to be implemented by the AMS Center.
Review all quality system documentation as specified in the
QMP and QAPPs
Interact with AMS Center management and technical
personnel to ensure that QA/QC procedures are understood
Conduct QA-related training as needed.
Advise the Battelle AMS Center Manager of any systematic
QA/QC issues and oversee corrective actions
Review draft and final QAPPs, amendment, deviations, and
SOPs.
Attend the verification test kick-off meeting and lead the
discussion of the QA elements of the kick-off meeting
checklist.
Assign QA Officer to each project.
Approve release of the QAPP, audit
reports, and deviations.
Authorized to recommend that work
be suspended if data quality is
threatened.
Authorized to delegate specific
responsibilities to other qualified
individuals
-------
ETV Advanced Monitoring Systems Center
Quality Management Plan Version 8.0
Date: Aprils, 2011
Appendix B-3
Personnel Responsibilities for the AMS Center for Verification Testing Activities
AMS Center Team
Member
Responsibilities
Authority
Battelle AMS Center
QA Officer
Prior to the start of verification testing, verify the presence of
applicable training records, including any vendor training on
test equipment.
Maintain real-time communication with the VTC on QA
activities, audit results, and concerns.
Prepare ISA and ADQ audit checklists; conduct TSAs and
ADQs, and prepare audit reports for every verification test as
specified in the QMP and QAPP. (If other staff are designated
as verification test QA Officers, review audit checklists, audit
reports, and responses.
Communicate to the VTC and/or technical staff the need for
immediate corrective action if an audit identifies QAPP
deviations or practices that threaten data quality.
Verify that audit responses for each audit finding and
observation are appropriate and that corrective action has
been implemented effectively.
Provide a summary of the QA/QC activities and results for the
verification reports.
Review verification reports and verification statements
Notify the Battelle AMS Center
Manager to issue a stop work order
if assessments indicate health,
safety, or quality concerns
-------
ETV Advanced Monitoring Systems Center
Quality Management Plan Version 8.0
Date: Aprils, 2011
Appendix B-4
AMS Center Team
Member
Responsibilities
Authority
Battelle Verification
Test Coordinators
Direct the team (Battelle testing staff and vendor) in performing
the verification test in accordance with the QAPP.
Provide sufficient test oversight to ensure that the QMP and
QAPP requirements are implemented
Perform PE audits of reference laboratories and calibrated
equipment for each verification test, as appropriate
Serve as the primary point of contact for vendor
representatives and collaborators.
Maintain instrumentation (vendor and/or reference
instrumentation) in accordance with the QMP, QAPP, SOPs,
and the manufacturer's instructions
Provide technical support to verification testing as needed, and
interact with the Battelle AMS Center QA Officer and Quality
Manager during inspections and implementation of corrective
actions when needed
Maintain verification test records that adequately capture test
activities and raw data to demonstrate the quality of data
collected
Maintain all verification test documents and records on the
AMS Center SharePoint site
Maintain real-time communication with the Battelle AMS
Center Manager and EPA AMS Center Project Officer and QM
on any potential or actual deviations from the QAPP.
Perform QA/QC activities specified in the QMP, QAPP, and
applicable SOPs, including QC measures and activities
required for sample analyses
Document results of QC analyses and include them with
sample results and historical data files
Provide test data, including data from the first day of testing, to
the Battelle AMS Center Manager for distribution to the EPA
AMS Center Project Officer and QM.
Review testing data and associated QC data at the frequency
specified in the QAPP to determine if quality goals and
objectives have been met. Designate an appropriate Battelle
technical staff member to review data generated by the VTC.
Respond to TSAs and ADQs within 10 days of receipt.
Perform corrective action at the direction of the Battelle AMS
Center Manager and Battelle AMS Center Quality Manager in
response to ISA and ADQ audit reports
Prepare draft and final verification reports and verification
statements
Authorized to establish contracts
with reference laboratories
Assign technical and operational
staff to the project.
Allocate budget among tasks as
identified in the QAPP.
Approve all labor, materials,
equipment, and subcontractor
charges to the project.
Authorized to suspend work for
cause if data quality or staff safety
are threatened.
Authorized to delegate specific
responsibilities to other qualified
individuals.
-------
ETV Advanced Monitoring Systems Center
Quality Management Plan Version 8.0
Date: Aprils, 2011
Appendix B-5
Personnel Responsibilities for the AMS Center for Verification Testing Activities
AMS Center Team
Member
Responsibilities
Authority
Battelle Testing
Staff
Assist in planning for the test, and making arrangements for
the receipt of and training on the technologies.
Support the VTC in the preparation of the test/QA plan and
reports, as necessary
Attend the verification test kick-off meeting, as requested.
Assist vendor staff as needed during technology receipt and
training.
Conduct verification testing using each participating
technology, following all aspects of the ETV AMS Center QMP
as well as the QAPP for the verification.
Record qualitative observations about the maintenance and
operation of the technologies being tested .
Ensure that the data from the technologies and reference
laboratories are reviewed according to the QAPP schedule,
compiled, recorded, and transmitted to the VTC.
Support the VTC in the preparation of the QAPP and reports,
as necessary
Support the VTC in responding to any issues raised in
assessment reports and audits related to technical
performance, statistics, or data reduction as needed.
Coordinate activities of any laboratory performing the
reference measurements
Confirm that the reference laboratory is conducting analyses
according to the QAPP
-------
APPENDIX C
AMS CENTER AMENDMENT AND DEVIATION FORMS
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Appendix C-l
QAPP AMENDMENT
QAPP TITLE AND DATE:
AMENDMENT NUMBER:
EFFECTIVE DATE:
PART TO BE CHANGED/REVISED:
CHANGE/REVISION:
REASON FOR CHANGE:
ORIGINATED BY:
Battelle Verification Test Coordinator
DATE
APPROVED BY:
Battelle AMS Center Manager Battelle AMS Center Quality Manager
DATE DATE
Required Distribution - All individuals/organizations listed on distribution for the applicable
QAPP, including but not limited to:
Battelle AMS Center Management
Battelle AMS Center Testing Staff
Battelle AMS Center Quality Manager
Subcontractors (if any)
Verification Test Collaborators (if any)
EPA/ETV AMS Center Project Officer
EPA/ETV AMS Center Quality Manager
Vendors
Distribution must be documented
-------
ETV Advanced Monitoring Systems Center
Quality Management Plan Version 8.0
Date: April 5, 2011
Appendix C-2
QAPP DEVIATION
QAPP TITLE AND DATE:
DEVIATION NUMBER:
DATE OF DEVIATION:
DESCRIPTION OF DEVIATION:
CAUSE OF DEVIATION:
IMPACT OF DEVIATION ON THE TEST:
CORRECTIVE ACTION:
ORIGINATED BY:
Battelle Verification Test Coordinator
DATE
ACKNOWLEDGED BY:
Battelle AMS Center Manager
Battelle AMS Center Quality Manager
DATE
DATE
Required Distribution - All individuals/organizations listed below:
Battelle AMS Center Management
Battelle AMS Center Quality Manager
Distribution must be documented
-------
APPENDIX D
DATA QUALITY INDICATOR DEFINITIONS AND EXAMPLES
-------
ETV Advanced Monitoring Systems Center
Quality Management Plan Version 8.0
Date: Aprils, 2011
Appendix D-l
DATA QUALITY INDICATOR DEFINITIONS AND EXAMPLES
Data Quality
Indicator
Meaning
QC Measures
Precision
Agreement among repeated
measurements under identical, or
substantially similar conditions
Field duplicates or splits
Lab duplicates/replicates
Can be within same organization or among
organizations using the same or different
methods
Bias
Systematic or persistent distortion
of a measurement process that
causes errors in one direction
Instrument calibration standards
Lab QC spikes
Matrix spikes & duplicates)
Accuracy
Overall agreement of a
measurement to a known value -
includes a combination of random
error (precision) and systematic
error (bias) in both sampling and
analysis operations
Matrix-specific standard or certified
reference materials
Spiked matrix samples
Representativeness
The degree to which data
accurately and precisely
represent a characteristic of a
population or condition
No specific QC tools to measure
Evaluate if samples were collected and
measurements made in such a way that
they reflect the population of interest (as
specified in the QAPP)
Comparability
Measure of confidence that one
data set can be compared to
another and combined for the
decision(s) to be made
Split samples; existing data
Compare population targeted by sampling
techniques; sample collection, handling,
preparation, & analysis procedures; holding
times, stability issues, QA protocols
Completeness
The amount of valid data needed
to be obtained from a
measurement system
# of valid results vs. the number determined
to be necessary during project planning (as
specified in the QAPP)
Specificity
Correct identification of the
parameter you are targeting
Retention times
Ion abundance ratios
Confirmation analyses
Peak shape
Detection and
Quantitation
The ability to
Determine if it is there or not
Distinguish between responses
representing different
concentrations of interest
Method Detection Limit (MDL) or equivalent
Signal to noise ratios
Calibration range
Analysis of samples at/near quantitation
limit
Well below action level
aFrom: EPA, 2010. Data Validation, Verification and Usability Quality Management Training Module. Presented at
Great Lakes Restoration Initiative Quality Technical Conference, Chicago IL. Presented by: Marion Kelly, US EPA
Office of Water; Dale Rushneck, Interface, Inc.;; William Telliard, US.EPA (Retired); Lynn Walters, CSC. December
7,2010.
-------
APPENDIX E
QAPP QA/R-5 ELEMENTS AND PREPARATION GUIDANCE
-------
QAPP QA/R-5 ELEMENTS AND PREPARATION GUIDANCE
U.S. EPA QA/R-5 QAPP
Element
GROUP A: PROJECT
MANAGEMENT
A1 Vendor Approval Page
A2 Table of Contents
A3 Distribution List
A4 Verification Test
Organization
A5 Background
Description for Section
The elements in this group address the basic area of project management, including the project history and
objectives, roles and responsibilities of the participants, etc. These elements ensure that the project has a
defined goal, that the participants understand the goal and the approach to be used, and that the planning
outputs have been documented.
Includes the vendor name, company, and date.
Table of Contents
Distribution List, typically includes vendor, EPA peer reviewers, collaborators and other stakeholders, and
Battelle testing staff.
This section should identify and define responsibilities for the EPA and Battelle managers and QA managers,
the VTC, stakeholders, and testing staff. Their involvement in the test should be described. Include an
organization chart that shows lines of authority, responsibility, and communication. Subcontractor task
leaders should be included.
This section should describe the technology need and technology description.
o w
3
>
p
OQ
ft
O
ft
d-
ft en
"i '-<
52 to
o f?
0 3
po en
O O
ft
-------
QAPP QA/R-5 ELEMENTS AND PREPARATION GUIDANCE
U.S. EPA QA/R-5 QAPP
Element
A6 Verification Test
Description and
Schedule
A7 Quality Objectives and
Criteria for Measurement
Data
A8 Special
Training/Certification
A9 Documents and Records
GROUP B: DATA
GENERATION AND
ACQUISITION
B1 Experimental Design
B2 Sampling Methods
B3 Sample Handling and
Custody
Description for Section
Define the test schedule
Test sites/locations
Health and safety considerations
This section should define the test Data Quality Objectives (DQOs), the data quality indicators (DQIs) that will
be used to assess data quality, and the Measurement Quality Criteria (MQOs) that define the criteria by which
data acceptability will be used assessed.
Special Training/Certification
This section should describe the controlling documents for the project and other documents that will be
generated during the project. The section should also define the records that will be kept during the project,
how they will be maintained, and their final disposition.
The elements in this group address all aspects of project design and implementation. Implementation of these
elements ensure that appropriate methods for sampling, measurement and analysis, data collection or
generation, data handling, and QC activities are employed and are properly documented.
Outline the experimental design, including test procedures, sampling design and rationale, sampling
frequencies, matrices, and measurement parameters of interest; define supporting documents (e.g., SOPs).
Any statistical analysis planned for the data should be described.
Sampling Methods (collection and approach) details, including applicable SOP citations.
This section should describe the sample handling and custody procedures that will be implemented for the
collection of environmental samples.
Sample Handling and Custody (Describe procedures for sample labeling, shipment, chain-of-custody
forms, procedures for transferring and maintaining custody of samples).
Sample Identification numbers and labels
o w
3
>
p
OQ
ft
O
ft
d-
ft en
"i '-<
52 to
o f?
0 3
OO VI
o O
ft
-5 >
11
&ซ
W
to
to
o
-------
QAPP QA/R-5 ELEMENTS AND PREPARATION GUIDANCE
u
B4
B5
B6
B7
B8
B9
.S. EPA QA/R-5 QAPP
Element
Reference Method
Quality Control
Instrument/Equipment
Testing, Inspection, and
Maintenance
Instrument/Equipment
Calibration and
Frequency
Inspection/Acceptance of
Supplies and
Consumables
Nondirect Measurements
Description for Section
This section should define the reference method against which technology results will be assessed. It should
define how, when, and from where data will be obtained. It should identify any constraints on the data
collection process. It should address, where appropriate:
Analytical Methods (identify analytical methods and equipment for the study, including method
performance requirements and applicable SOPs) MDLS, method details, including applicable SOP citations.
This section should specify the activities during data collection that will provide the information used to assess
data quality (i.e., field or laboratory QC operations, audits, technical assessments). Specifically, it should
address:
QC (Describe QC procedures that should be associated with each sampling and measurement
technique. List required checks and corrective action procedures).
This section will describe the maintenance procedures required for equipment or instruments used to collect or
measure environmental data. Details should include:
Instrument/Equipment Testing, Inspection, Maintenance, Frequency, and Acceptance Criteria. It is usually
acceptable to reference a specific SOP, rather than provide details in the QAPP. However, if specific
maintenance is critical to an operation the project leader may choose to highlight those procedures in the text.
This section should describe the calibration of equipment or instruments used to collect and/or measure
environmental data. Details should include:
Instrument/Equipment Calibration, Frequency, and Acceptance Criteria. It is acceptable to reference SOPs for
routine calibration procedures. However, the criteria for critical measurements should be defined in the QAPP.
Inspection/Acceptance of Supplies and Consumables (Define how and by whom the sampling supplies and
consumables will be accepted for use in the project).
Nondirect Measurements (existing data) (define the criteria for the use of nonmeasurement data, such as data
that come from databases or literature).
o w
3
>
p
OQ
ft
O
ft
d-
2 o
ft en
>-t '-<
52 to
o f?
0 3
OO VI
o O
ft
-5 >
11
&ซ
W
OJ
to
o
-------
QAPP QA/R-5 ELEMENTS AND PREPARATION GUIDANCE
U.S. EPA QA/R-5 QAPP
Element
B10 Data Management
GROUP C: ASSESSMENT
AND OVERSIGHT
C1 Assessment and
Oversight
C2 Reports to Management
GROUP D: DATA
VALIDATION AND
USABILITY
D1 Data Review,
Verification, and
Validation
D2 Validation and
Verification Methods
Description for Section
This section should describe how data collected during and after testing will be documented, managed,
stored, and controlled. Outline the data management scheme including the path and storage of the data and
the data record-keeping system. Identify all data handling equipment and procedures that will be used to
process, compile, and analyze the data. Describe data reporting conventions, including the use of data
qualifiers and units).
The elements in this group address the activities for assessing the effectiveness of the implementation of the
project and associated QA and QC activities. The purpose of assessment is to ensure that the QA Project Plan
is implemented as prescribed.
This section should describe the assessment activities that will be implemented for the project. This will
typically include PEs, TSAs, and ADQs.
This section will identify the frequency, content, and distribution of reports issued to keep management
informed of the results of audits and assessments.
The elements in this group address the activities that occur after the data collection or generation phase of the
project is completed. Implementation of these elements ensures that the data conform to the specified criteria,
thus achieving the project objectives.
Describe the types of data review, verification, and validation that will be implemented for the study.
Define the data validation and verification procedures and the criteria used to accept or reject the data based
on quality.
o w
3
>
p
OQ
ft
O
ft
d-
ft en
"i '-<
52 to
o f?
0 3
OO VI
o O
ft
-5 >
it
&^;
w
to
o
-------
QAPP QA/R-5 ELEMENTS AND PREPARATION GUIDANCE
U.S. EPA QA/R-5 QAPP
Element
D3 Reconciliation with User
Requirements
Description for Section
This section will describe how the data for the project will be analyzed, evaluated, and assessed against their
intended use and the performance criteria established above. It will describe specific procedures, including
formulas, to be used in routinely assessing data precision and accuracy, representativeness, comparability,
and completeness of the specific measurement parameters involved. The purpose of this section is to describe
how results will be evaluated to determine if performance criteria have been satisfied.
o w
3
>
p
OQ
ft
2 o
ง i-
O f
0 3
o en
ft
&.
W
-------
APPENDIX F
ETV KICK-OFF MEETING CHECKLIST
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Appendix F-l
ETV VERIFICATION TEST KICK-OFF MEETING
PURPOSE
To prepare verification testing staff for an upcoming test and review critical logistical, technical, and
administrative aspects of the test.
STAFF TO ATTEND
Verification Test Coordinator
Verification testing leader (if appropriate)
Battelle AMS Center Manager
Battelle AMS Center Quality Manager
EPA AMS Center Project Officer
EPA AMS Center Quality Manager
Vendor(s), if significant involvement with implementation of QAPP procedures
All testing staff involved in all phases of test (need to have kick-off meeting on-site with staff from
partnering organizations/subcontractor if necessary)
TIMING AND LENGTH
The kick-off meeting should be scheduled prior to the start of testing. It should be near the start of the test
but allow time for the test coordinator to address any lingering issues.
A tentative date/time should be set for the kick-off meeting a month in advance of the start of the test, if
possible, to allow the meeting to be scheduled when the critical staff mentioned above are all available.
It may be necessary to schedule a second kick-off meeting for tests that are conducted at multiple
locations if all staff cannot attend the original meeting.
PROJECT MANAGEMENT (internal only)
Review roles/responsibilities of all staff attending meeting.
Work authorization distributed to all staff?
All staff have project number and subaccount number(s)?
All testing staff have budget for their time on each subaccount involved?
Stakeholders, EPA/ETV program manager, and EPA/QA staff pre-notified of testing schedule and start
date?
Review test schedule.
Formal distribution of final, signed hard-copy QAPP made to all Battelle staff, subcontractor (if any),
vendors, and EPA? (The kickoff meeting should not be conducted unless the QAPP is final, signed by
vendors and distributed).
Documentation: All pertinent forms should be signed, copied for Amy, and brought to the meeting.
o Peer review forms on QAPP. Ideally includes one EPA reviewer/two non-EPA peer
reviewers.
o Final QAPP signed by all vendors? Where are original signature pages stored? All staff
attending meeting should be told to bring their copy of QAPP to kick-off meeting prior to
meeting or final QAPP should be distributed at meeting.
o All vendor agreements signed/checks received/copies sent to Amy for project files?
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Appendix F-2
If Battelle, partner, or subcontractor staff is operating technology, has vendor trained those staff in
operating the technology and signed formal training form certifying training and vendor's acceptance of
Battelle generated results? Where is original signed form located/filed?
Subcontractors (if used):
o Subcontract fully signed? Provide copy to Amy for project files.
o Due dates well defined in subcontract?
o Get test data from subcontractor/partner real time or at close of test.
Communication: If testing is being performed off-site, will regular communication with the staff at the
test site be maintained? If so, how?
Has a 508- compliant pdf file been generated of the final QAPP?
QUALITY ASSURANCE
PE Audit:
o Was/will a PE audit be performed? Who is/was responsible for obtaining the PE and
providing it to the testing organizations?
o What are QC limits? Were they achieved?
o Corrective action if QC limits are not met? Who to contact?
Staff training records:
o Documentation to support experience and qualifications.
o Technology training - Needs to be documented on the Vendor Training/Operation Consent
form.
o Reference laboratory staff qualifications/experience.
Technical Systems Audit (TSA):
o Define which phases and locations of testing will be reviewed (e.g., field, laboratory,
reference laboratory) and the primary focus of the audits.
TSA of reference laboratory will focus on compliance with the TQAP, reference
method, and good laboratory practices.
TSA of technologies will focus on testing vs. test design and data acquisition
procedures.
o Define the TSA schedule.
o Determine if EPA will also conduct a TSA during the test and their schedule.
o Results of the TSA will be documented in a report by QAO and submitted to the VTC and
EPA (after internal review) within 10 business days after completion of the data review.
Deviations :
o Review deviation/amendment procedures at meeting - what to do in the middle of a test if
QAPP cannot be followed - who to notify/what forms to file.
o Verification Test Coordinator will prepare deviation reports for any departure from the TQAP
during the verification.
o Testing staff will contact VTC if there are any questions re test procedures. VTC will consult
with Dindal, Buhl, McKernan, and Henderson. Verbal agreement on deviations will be
obtained, followed by a written deviation.
o For deviations or amendments, obtain the requisite approvals, send deviation to EPA for
review/approval, and distribute to the project team.
o Distribute the approved report as specified in the AMS Center QMP.
Data Quality Audit (ADQ):
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Appendix F-3
o The first batch of data (define what is considered the first data batch based on TQAP) will be
audited within 10 business days of receipt and assessed using a project-specific checklist.
o Describe audit schedule for remaining data based on TQAP.
o Battelle QAO will audit 100% of the calibration and quality control data and 10% of
Category III data or 25% of Category II verification data acquired in the verification test.
o Results of the ADQ will be documented in a report by QAO and submitted to the VTC and
EPA (after internal review) within 10 business days after completion of the data review.
DOCUMENTATION
All testing personnel will use data sheets and/or Laboratory Record Book (LRB) for the project.
Samples and/or direct readings within each test series must be uniquely labeled when collected. The
sample IDs should be documented in the test records so there is a direct link between each sample/data
collected during the test and the analytical results. The VTC will define the sample ID scheme.
All subsequent analysis must reference (or be traceable to) the unique sample ID.
Field and/or laboratory personnel should fill out data sheets and LRB pages in real time, as needed - each
signed and dated.
Data must undergo a 100% validation and verification by technical staff (i.e. VTC, or designee) before it
will be assessed as part of the data quality audit.
If both an LRB and data sheets are used for documentation, the VTC should define the information that
will be recorded in each.
Copies of the TQAP and other methods cited in TQAP should be available to testing staff and in
laboratory where test will be performed.
All records should be signed and dated by the person entering the records.
If samples are to be transported between labs, or between Battelle/partner/ subcontractor, staff should
have training in the COC SOP. Distribute an example COC form and discuss how to complete it.
Define reagents or standards that should be supported by certificates of analysis; these should be included
in the verification test binder.
TECHNICAL
Emphasize to testing staff to document anything and everything that is observed about the technologies,
particularly if there are unusual sample results (e.g., sample color).
Will the samples be blind and randomly distributed to the operators?
Are provisions made to handle daily preparation of solutions/standards, if necessary?
Take digital photos of all test activities.
DATA/REPORTING
Review data recording forms or sheets at meeting or discuss how/where will data be recorded for each
testing activity.
How are data going to be converted electronically? Are data saved in technology undergoing verification
and then exported to Excel? Or will data be recorded manually by the operators? If so, how will
transcription errors be avoided?
Data review - who will be doing two week review for each data set collected? If Battelle staff not on-
site, how will data be transmitted to Battelle for two-week review?
-------
ETV Advanced Monitoring Systems Center Date: April 5, 2011
Quality Management Plan Version 8.0 Appendix F-4
Who is Battelle verification report author? Distribute and review report schedule. Reporting should
begin at the same time as testing.
------- |