United States -
Environmental Protection
Agency
Office of Research and
Development
Washington, DC 20460
EPA/600/R-98/064
May 1998
Environmental Technology
Verification Program
Quality and Management
Plan for the Pilot Period
(1995-2000)
-------
-------
EPA/600/R-98/064
May 1998
ENVIRONMENTAL TECHNOLOGY
VERIFICATION PROGRAM
QUALITY AND MANAGEMENT
PLAN FOR THE PILOT PERIOD
(1995-2000)
National Risk Management Research Laboratory
National Exposure Research Laboratory
Office of Research and Development
U.S. Environmental Protection Agency
Cincinnati, Ohio 45268
The most current version of the document can be found on the ETV web site (www.epa.gov/etv).
Printed on Recycled Paper
-------
-------
US Environmental Protection Agency
Environmental Technology Verification Program
QUALITY AND MANAGEMENT PLAN
FOR THE PILOT PERIOD
May 1998
APPROVALS
Signatur/f// Date
Signature
Date
E. Timothy Oppelt, Director
National Risk Management Research Laboratory
Gary J. Foley, Ph.D., Director
National Exposure Research Laboratory
Penelope Hansen, Coordinator
Environmental Technology Verification Program
Frank T. Princiotta, Director
NRMRL Air Pollution Prevention and Control
Division
Signature
Signature
Date
Date
Date
Robert M. Clark, Ph.D., Director
NRMRL Water Supply and Water Resources
Division
Subhas K. Sikdar, Ph.D., Director
NRMRL Sustainable Technology Division
Jay J. Messer, Ph.D., Acting Director
NERL Environmental Sciences Division
Signature '
Date
Larry T. Cupitt, Ph.D., Director
NERL Human Exposure and Atmospheric Sciences
Division
m
-------
ACKNOWLEDGMENTS
The first draft of this document was developed by a team of writers consisting of the following
quality assurance staff members from the U.S. Environmental Protection Agency Office of
Research and Development National Risk Management Research Laboratory and National
Exposure Research Laboratory: Sam Hayes, Lora Johnson, Ann Kern, and Jeff Worthington.
Subsequent revisions included significant input in the form of comments from members of the
Environmental Technology Verification Team. Verification partners also provided comments.
The team of writers was joined by the following EPA staff in development of this final document:
Nancy Adams, Penelope Hansen, Linda Porter, and Shirley Wasson.
IV
-------
TABLE OF CONTENTS
DOCUMENTS AND GENERAL TERMS | viii
ABBREVIATIONS AND ACRONYMS jxii
INTRODUCTION xiii
PART A
MANAGEMENT SYSTEMS xvii
1.0 MANAGEMENT AND ORGANIZATION 1
1.1 ETV quality policy 1
1.2 Organization structure 1
1.3 ETV customer identification and ETV customer needs/expectations/work
objectives 5
1.4 Potential verification partners 6
1.5 Management resolution for verification partner quality constraints 7
1.6 Resources 7
1.7 Authority to stop work for safety and quality considerations 8
2.0 QUALITY SYSTEM AND DESCRIPTION 8
2.1 Authorities and conformance to E4 quality standard . 8
2.2 Quality system documents 9
2.3 Quality system scope 9
2.4 Quality expectation for products and services 9
2.5 Quality procedures documentation 10
2.6 Quality controls 10
2.7 Management system reviews (MSRs) 11
3.0 PERSONNEL QUALIFICATION AND TRAINING 11
3.1 Personnel training and qualification procedures 11
3.2 Formal qualifications and certifications 12
3.3 Technical management and training 12
3.4 Retraining 13
3.5 Personnel job proficiency 13
4.0 ETV VERIFICATION PARTNER SELECTION 14
4.1 Planning and control of selection process 14
4.2 Technical and quality requirements 15
4.3 Quality specification/conformance 15
-------
4.4 Peer review of assistance agreements 15
4.5 Conformance of verification testing efforts 16
5.0 RECORDS 16
5.1 Scope 16
5.2 Preparation, review, approval, and distribution 16
5.3 Records storage and obsolete records 17
6.0 COMPUTER HARDWARE AND SOFTWARE 19
6.1 General procedures 19
6.2 Scope of ETV computer hardware/software procedures 20
6.3 Configuration testing 20
6.4 Measurement and testing equipment configurations 20
6.5 Change assessments - configurations, components, and requirements 21
6.6 ETV website roles and responsibilities 21
7.0 PLANNING 21
7.1 Systematic planning process 21
7.2 Planning document review 23
8.0 IMPLEMENTATION OF WORK PROCESSES 23
8.1 Implementation 23
8.2 Procedures 24
8.3 Oversight 24
9.0 ASSESSMENT AND RESPONSE 24
9.1 Numbers and types of assessments 24
9.2 Procedures 25
9.3 Personnel qualifications, responsibility, and authority 25
9.4 Response 27
10.0 QUALITY IMPROVEMENT 27
10.1 Annual review for quality improvement 27
10.2 Detecting and correcting quality system problems 27
10.3 Cause and effect relationship 28
10.4 Root cause 28
10.5 Quality improvement action 28
VI
-------
PART B
COLLECTION AND EVALUATION OF
ENVIRONMENTAL DATA 29
1.0 PLANNING AND SCOPING 30
1.1 Systematic planning of the verification test 30
1.2 Systematic planning for verification testing 31
2.0 DESIGN OF TECHNOLOGY VERIFICATION TESTS 32
2.1 Design process 32
3.0 IMPLEMENTATION OF PLANNED OPERATIONS 36
3.1 Implementation of planning 36
3.2 Services and items 37
3.3 Field and laboratory samples 37
3.4 Data and information management 38
4.0 ASSESSMENT AND RESPONSE 38
4.1 Assessment types 38
4.2 Assessment frequency 38
4.3 Response to assessment 39
5.0 ASSESSMENT AND VERIFICATION OF DATA USABILITY 39
5.1 Data validation 39
5.2 Existing data 39
5.3 Reports reviewed 40
REFERENCES 41
APPENDDC A . .. ! A-l
APPENDIXB B-l
APPENDIX C .. C-l
VJl
-------
DOCUMENTS AND GENERAL TERMS
Annual progress report
The report developed on an annual basis to report implementation of the ETV program.
Directors of Quality Assurance
Quality assurance directors for the EPA ORD laboratories, the National Exposure
Research Laboratory and the National Risk Management Research Laboratory.
E4
The ANSI/ASQC national consensus standard which is the Agency standard applicable to
assistance agreements. The standard is entitled, E4-1994, Specifications and Guidance for
Quality Systems for Environmental Data Collection and Environmental Technology
Programs.
ETV assistance agreement
The contractual record developed by the EPA and signed by the verification partner.
ETV coordinator
The EPA person designated by EPA ORD to lead the ETV team.
ETV verification statement
A summary statement, developed by the verification partner and approved by the EPA
pilot manager, which reports individual technology performance.
ETV verification report
The report of the result of an individual technology test.
EPA line management
The management structure to whom each EPA pilot manager reports; i.e. branch chief,
division director, laboratory director.
EPA pilot manager
The EPA person designated by EPA line management to serve as the lead for an individual
ETV pilot.
EPA pilot quality managers
The EPA quality assurance person designated by EPA line management to manage quality
assurance efforts on behalf of the pilot manager.
vni
-------
EPA review/audit reports
The "quality records" developed by EPA as a result of conducting assessments during
ETV implementation.
ETVteam
EPA employees actively working on the ETV program; the ETV coordinator, pilot
managers, and the directors of quality assurance are core members
ETV test objective
The stated objective(s) of each technology test. Verification partners use the DQO
process to establish test objectives and test measurement quality criteria.
ETV webmaster
The person designated by EPA line management with responsibility for establishing and
maintaining the ETV website.
Evaluation contractor
The contractor selected to collect information on pilot performance.
Generic verification protocol
Those protocols developed, modified, or selected to promote uniform testing for a single
pilot operated by the verification partner. Adequate documentation of a robust protocol
may allow the development of abbreviated individual test/QA plans which incorporate the
generic verification protocol by reference.
Laboratory director
The directors of EPA ORD laboratories, the National Exposure Research Laboratory and
the National Risk Management Research Laboratory.
Management system review
The qualitative assessment of data collection operation and/or organization(s) to evaluate
the adequacy of the prevailing quality management structure, policies, practices, and
procedures for obtaining the type and quality of data needed are obtained.
Office of Research and Development Assistant Administrator
The administrative lead person directing the EPA's Office of Research & Development.
Quality and Management Plan for the Pilot Period
The specific policies and procedures for managing quality related activities in the ETV
program.
IX
-------
Raw data
All data and information recorded in support of analytical and process measurements made
during planning, testing, and assessing environmental technology including support
records such as: computer printouts, instrument run charts, standards preparation records,
field log records, technology operation logs, and monitoring records. ETV test files (all
records including raw data) and technical data and associated quality control data which
support the data summarized and the conclusions made in each ETV verification report.
Records
All books, papers, maps, photographs, machine readable materials, or other documentary
materials, regardless of physical form or characteristics, made or received by the EPA or a
verification partner or their designated representative for the ETV program.
Stakeholder groups
Groups set up for each pilot area consisting of representatives of any or all of the
following verification customer groups: buyers and users of technology, developers and
vendors, the consulting engineering, the finance and export communities, and government
permitters and regulators.
Standard operating procedures
Procedures describing routine verification activities including sample collection, analytical
testing, and associated verification process.
Test/QA plan
The plan developed by a verification partner for each individual test of a technology or
technology class. Therefore, the test/QA plan may include more than one technology.
The test/QA plan provides the experimental approach with clearly stated test objectives
and associated quality objectives for the related measurements. The test/QA plan may
incorporate or reference generic verification protocols.
Test measurement
Those critical measurements that must be made during the course of a technology test to
evaluate achievement of the ETV test objective.
Verification
Establishing or proving the truth of the performance of a technology under specific,
predetermined criteria or protocols and adequate data quality assurance procedures.
Verification partners
The public and private sector organizations holding cooperative or interagency agreements
to assist EPA in implementing the ETV program.
-------
Verification partner manager
The person designated by the verification partner to'manage the pilot and serve as the
chief point of contact with the EPA.
Verification partner quality manager
The person designated by the verification partner to manage quality assurance for the pilot
on behalf of the verification partner manager.
Verification partner quality management plan
The procedures for quality related activities developed and implemented by the verification
partner to assure quality in the work processes and services developed for ETV. If the
verification partner has a current quality system that accommodates ETV's needs,
additional quality system elements do not need to be developed.
VP review/audit reports
The "quality records" developed by the verification partner as a result of conducting
assessments during ETV implementation.
Verification Strategy
The ETV Program Verification Strategy, first published in February 1997, outlines the
pilot period goals, programmatic operating principles, pilot selection criteria, key
definitions, budgets, and implementation activities that are molding the ETV program, as
well as the challenges that are emerging and the decisions that need to be addressed in the
future.
XI
-------
ABBREVIATIONS AND ACRONYMS
ADQ
ANSI
ASQorASQC
AWBERC
CBD
COAG
ETV
ETVQMP
FTE
GAD
IAG
ISO
MSR
NRMRL
NERL
ORD
OSHA
PEA
QA
QC
RFA
SOP
SOW
ISA
VP
WQMP
audits of data quality
American National Standards Institute
American Society for Quality
Andrew W. Briedenbach Environmental Research Center building
Commerce Business Daily
cooperative agreements
environmental technology verification
ETV quality and management plan
full time equivalent
Grants Administration Division
interagency agreement
International Standards Organization
management systems review
National Risk Management Research Laboratory
National Exposure Research Laboratory
EPA's Office of Research and Development
Occupational Safety and Health Administration
performance evaluation audits
quality assurance
quality control
request for application
standard operating procedure
statement of work
technical systems audit
verification partner
verification partner quality management plan
XII
-------
INTRODUCTION
Background
The Environmental Technology Verification Program (ETV) has been established by the
Environmental Protection Agency (EPA) to evaluate the performance characteristics of innovative
environmental technologies across all media and to report this objective information to the
permitters, buyers, and users of environmental technology. ETV has evolved in response to the
following mandates:
• A directive to EPA by the President in his 1995 environmental technology
strategy, Bridge to a Sustainable Future, to "work with the private sector to
establish a market-based verification process... which will be available nationally
for all environmental technologies within three years."
• Goals articulated in the Vice-President's Reinventing Government: A Performance
Review which directed EPA to begin a comprehensive environmental technology
verification program no later than October 1995.
• Congressional appropriation language contained in the FY96 and FY97 budgets,
that the Agency fund technology verification activities at the $10 million level in
each year.
To comply with these directives, EPA's Office of Research and Development (ORD) has
established a five year pilot program to evaluate alternative operating parameters and determine
the overall feasibility of a technology verification program. ETV began in October 1995 and will
be evaluated through October 2000, at which time the Agency will prepare a report to Congress
containing the results of the pilot program and recommendations for its future operation.
Program Description
Developers of innovative environmental technology report numerous impediments to
commercialization. Among those most frequently mentioned is the lack of acceptance of vendor
performance claims. It is believed that objective, independently acquired, high quality
performance data and operational information on new technologies will significantly facilitate the
use, permitting, financing, export, purchase, and general marketplace acceptance of such
technologies. The purpose of ETV is to provide such data and information to the customer
groups that require them in order to accelerate the real world implementation of improved
technology. Better technology will more thoroughly, rapidly, and efficiently protect the
environment. It is important to stress that the product of ETV is high quality data and
information, not technology approval or endorsement.
xm
-------
The thesis that independent performance verification more rapidly moves new technology into use
will be tested by EPA's five year pilot program. ETV funds and operates twelve pilot projects,
each operated by third party organizations under the auspices of EPA. These "partner
organizations" include private sector testing, evaluation and research companies, state technology
evaluation programs, federal laboratories, and industry associations. For the most part, each pilot
is focused on a different environmental, industry, or technology sector (e.g., air pollution control
technology, drinking water systems, field monitoring devices, industrial coatings products). By
design, all pilots are operated in a somewhat different manner in order to test various methods for
both technical and operational efficiency and effectiveness in verifying performance. Management
techniques are in place to assure that constant evaluation of alternative methods occurs and
results in continuous improvement of processes throughout the pilot period.
Because credible information is the ultimate product of ETV, the highest appropriate quality
assurance procedures will be used throughout the program. The EPA's Office of Research and
Development implements an Agency-wide quality system to assure that activities conducted in
EPA research laboratories, other EPA research facilities or locations, or at facilities being
operated on behalf of or in cooperation with the EPA are supported by data of known and
acceptable quality for their intended use. Individual research laboratories develop laboratory-
specific quality management plans. The National Risk Management Laboratory (NRMRL) and
the National Exposure Research Laboratory (NERL) are implementing ETV in conformance with
such plans.
Program and Quality Management Documents
Several documents define the overall operation of the program during its pilot stage. The first to
be published (February 1997) was the Environmental Technology Verification Strategy. This
document lays out goals, customer and key word definitions, basic operating principles, pilot
selection criteria, and the programmatic and budgetary vision of the pilot program. The Strategy
is intended to evolve over time and is now (May 1998) undergoing review to evaluate the need
for modification and amplification.
xiv
-------
The second major program management document being used by ETV to guide its operation is
the ETV Quality and Management Plan (QMP) which follows this introduction. Under
development for over a year, the ETV QMP uses the structure, policies and standards of the
American National Standard ANSI/ASQC E4-1994, "Specification and Guidelines for Quality
Systems for Environmental Data Collection and Environmental Programs." This document^
...describes a basic set of mandatory specifications and non-mandatory guidelines by which
a quality system for programs involving environmental data collection and environmental
technology can be planned, implemented and assessed1.
As of February 1996, all cooperative agreements entered into by EPA concerning environmental
technology must be in conformance with the provisions of E4. This requirement is expected to be
extended to EPA contracts in 19982.
The ETV Quality and Management Plan
Based upon the structure and standards of E4, the ETV QMP lays out the definitions, procedures,
processes, inter-organizational relationships, and outputs that will assure the quality of both the
data and the programmatic elements of ETV during the pilot period. Part A of the ETV Quality
and Management Plan contains the specifications and guidelines that are applicable to common or
routine quality management functions and activities necessary to support the ETV program. Part
B of the ETV Quality and Management Plan contains the specifications and guidelines that apply
to test-specific environmental activities involving the generation, collection, analysis, evaluation,
and reporting of test data.
EPA's verification program is organizationally complex, involving numerous outside organizations
through its extensive stakeholder process, partner organizations who bear most of the quality
assurance responsibilities, and testing and consulting companies hired by partner organizations to
conduct field and laboratory work. Within EPA, the program is coordinated through ORD's ETV
Team consisting of staff from ten Branches located in six Divisions of two Laboratories, NRMRL
and NERL, along with quality assurance staff in each of the laboratories' physical locations.
Finally, EPA program offices and regions are increasingly involved in outreach activities, as are
states and other Federal agencies through the White House Environment and Technology
Working Group. The ETV QMP is designed to play a major role in clearly delineating the roles
and responsibilities of all of these diverse and important players.
1 American Society for Quality Control; American National Standard: Specifications and
Guidelines for Quality Systems for Environmental Data Collection and Environmental Technology
Programs: Milwaukee, Wisconsin; 1994
2 All ETV pilots are executed under cooperative agreements with the exception of the Site
Characterization and Monitoring Pilot and the Industrial Coatings and Coatings Equipment Pilot
which utilize Interagency Agreements with the Departments of Energy and Defense.
xv
-------
All partner organizations will use this document and its parent, the E4 standard, to create quality
management plans that assure appropriate levels of data collection, quality outputs, and customer
responsiveness. These plans will be submitted to EPA for review and approval by pilot managers
and quality assurance staff. It is not the purpose of this document to require that partner
organizations create wholly new operating procedures solely for use under ETV. Most
organizations selected by EPA as cooperators already have many of the procedural and process
elements required by E4 incorporated into their existing management systems. Other
requirements found in this document will be new or different. Cooperators should address all
appropriate elements of the ETV QMP either specifically in their ETV plan or include appropriate
and adequately detailed references to existing documents.
The ETV QMP will be reviewed on an annual basis throughout the pilot period (and beyond if the
program is extended) to incorporate lessons learned from the experiences of the pilots and
feedback from customer groups. The addition of new policies and elimination or modification of
ineffective procedures will be discussed with all participants and modifications to partner QMP's
may be required.
The ETV QMP follows the general outline of the ANSI/ASQC E4-1994document.
xvi
-------
PART A
MANAGEMENT SYSTEMS
Part A of the ETV Quality and Management Plan contains the specifications and guidelines that
are applicable to common or routine quality management functions and activities necessary to
support the ETV program.
Part B of the ETV Quality and Management Plan contains the specifications and guidelines that
apply to test-specific environmental activities involving the generation, collection, analysis,
evaluation, and reporting of test data.
xvu
-------
-------
1.0 MANAGEMENT AND ORGANIZATION
1.1 ETV quality policy
The Office of Research and Development shall establish and implement a quality policy to ensure that the Environmental
Technology Verification (ETV) program produces the type and quality of program outputs needed and expected by ETV
clients.
The EPA Office of Research and Development's (ORD) quality policy for the
Environmental Technology Verification (ETV) program is established as follows:
The quality system for the overall ETV program seeks to be consistent with
industry consensus standards. Each verification partner shall implement a valid
and approved quality system. As of February 1996, the Agency's required quality
system for cooperative agreements is ANSI/ASQC E4. Each verification test will
be performed according to planned and documented, pre-approved test/QA plans.
All technical statements in ETV verification reports shall be supported by the
appropriate data.
1.2 Organization structure
The relevant organizations, functional responsibilities, levels of accountability and authority, and lines of communication
shall be formally defined in the quality system and approved by the EPA laboratory directors responsible for the quality of
work performed by or on behalf of each EPA laboratory.
The overall organizational structure of the ETV program is depicted in Figure 1.1. The
structure graphically presents lines of accountability, authority, and communication.
Listed below are the general functional responsibilities for the major organizational units
specified in Figure 1.1.
1.2.1 Assistant Administrator for ORD and the USEPA Administrator responsibilities:
• provides overall program direction
• serves in a program leadership role with Congress, other agencies of the Executive
Branch, and the general public
1.2.2 ORD laboratory directors responsibilities:
• approve and implement annual program budgets and resource allocations
• allocate laboratory personnel and other resources to accomplish ETV's goals,
including appointment of the ETV coordinator
• approve all ETV verification statements
• review and approve ETV Verification Strategy and ETV Quality and Management
Plan (QMP)
• ensure that appropriate assessments (i.e., see Table 9) are implemented
ETV QMP
Page: 1 of 41
Version: 1.0
Date: May 26,1998
-------
•M
s
-------
1.2.3 Division directors and branch chiefs responsibilities:
• allocate appropriate division and branch personnel and other necessary resources
to support pilots located in the division/branch
• provide oversight of pilot outputs and products prior to public release
1.2.4 ETV coordinator responsibilities:
• leads the ETV team by providing communication opportunities, e.g., periodic
conference calls, meetings, and training
• coordinates the overall ETV program, including design of multi-year strategies,
operating principles, implementation activities, and annual budgets
• communicates ETV team and program activities, progress, outputs, and
recommendations to EPA, Congress, agencies in the Executive Branch, customer
groups, and the general public
• maintains an up-to-date ETV website containing materials relevant to the program
and to each pilot
• manages overall ETV program outreach activities to ensure that customer groups
are knowledgeable about the existence and use of ETV generated data
• collects data on operational parameters and program outputs to continuously
evaluate the ETV program and make recommendations to management and the
Congress on its present and future operation.
• reviews, approves, and assists in revision of ETV QMP
• ensures ETV QMP is implemented in the ETV program
1.2.5 ETV team responsibilities:
• establish mutually acceptable program-level strategies and protocols
• participate in development of an overarching ETV outreach strategy
• communicate pilot-specific progress, issues, difficulties, and lessons learned
• meet to discuss program objectives, seek collegial guidance, and evaluate success
• review ETV QMP
1.2.6 EPA pilot managers responsibilities:
« select and oversee verification partners
« communicate requirements for and oversee the verification partner's quality system
• arrange for peer review of verification partner proposals and ETV verification
reports
• attend and/or conduct regular meetings with stakeholders
« oversee production and approval process of ETV verification reports and ETV
verification statements
• assist with ETV outreach activities for the assigned ETV pilot
• participate in ETV team activities
ETV QMP
Page: 3 of 41
Version: 1.0
Date: May 26,1998
-------
1.2.7 Verification partners responsibilities:
• establish, attend, and/or conduct meetings of stakeholder
• maintain communication with EPA to assure mutual understanding and
conformance with EPA quality procedures and expectations and ETV policies and
procedures
• manage the oversight and conduct of verification activities
• assure that quality procedures are incorporated into all aspects of each ETV pilot
• develop, conduct, and/or oversee test/QA plans in cooperation with technology
vendors
• solicit technology vendor proposals or vendor products
• operate ETV activities within their documented and approved quality management
plan
• prepare ETV verification reports on technology tests
• prepare a three-to-five page ETV verification statement at the completion of each
technology verification
appoint a quality manager, responsible for ensuring that the VP's organization and
suppliers/contractors to the VP organization have quality systems in compliance
with this QMP, and that the VP complies with their documented quality system.
1.2.8 Stakeholders group responsibilities may include the following:
• assist in development of generic verification protocols
• assist in prioritizing the types of technologies to be verified
• review pilot-specific procedures and selected ETV verification reports emerging
from the ETV pilot
• assist in the definition and conduct of outreach activities appropriate to the
technology area and customer groups
• serve as information conduits to the particular constituencies that each member
represents
1.2.9 ETV directors of quality assurance responsibilities:
• develop and implement the ETV quality system at the direction of the ETV
coordinator and in coordination with the ETV team
• document the ETV quality system in the ETV QMP
• review, and update, if necessary, the ETV QMP annually in cooperation with the
ETV coordinator
• work with ETV pilot quality managers to ensure implementation of ETV QMP
• provide current copies of the ETV QMP to the appropriate participants in the
ETV program
• communicate quality issues and information to the ETV team in a timely manner
• conduct internal management system reviews (MSR)
ETVQMP
Page: 4 of 41
Version: 1.0
Date: May 26,1998
-------
1.2.10 EPA pilot quality managers responsibilities:
• communicate quality system requirements, quality procedures, and quality issues to
the assigned EPA pilot manager and verification partner
• review and comment on verification partner's quality system description to verify
conformance to the quality provisions of this document
• perform MSR of each pilot's quality system to verify conformance to the quality
provisions of this document
• perform technical systems audits (TSAs) and performance evaluations (PEs) of
pilots, as appropriate
• provide assistance to pilot personnel in resolving quality assurance issues
Tables available on the ETV website (www.epa.gov/etv) present a current listing of the
pilots that are either underway or soon to be awarded. Included in the table are the EPA
pilot managers, EPA pilot quality managers, verification partner managers, and verification
partner quality managers. The tables contain their names, geographic locations, ORD
laboratory or company affiliations, and phone numbers.
1.3 ETV customer identification and ETV customer needs/expectations/work objectives
The ETV coordinator, pilot managers, and partners are responsible for coordinating the identification of customers and
communicating the needs of the internal and external customers to ensure that ETV-work products satisfy their needs.
1.3.1 As identified in the ETV Verification Strategy, external customers (i.e., outside EPA)
include, but are not limited to:
• public and private sector buyers and users of technology
• developers and vendors of technology
• the consulting engineering community who recommend technologies to buyers
• federal, state and local government permitting/regulatory agencies
• international marketers and the financial community
• Congress
In a general sense, needs and expectations of external customers include:
• ETV verification reports and ETV verification statements supported by objective
and reliable data, provided in a timely manner
• a justifiable approach to selecting technologies for testing
• a practical approach in testing which provides efficient, timely, and cost-effective
technology tests
• user-friendly documents (e.g., easy to read and to implement)
• technology operation consistent with statements in the ETV verification report
ETVQMP
Page: 5 of 41
Version: 1.0
Date: May 26,1998
-------
For each pilot, needs and expectations of external customers are defined and documented
in the minutes of stakeholders meetings. The process to define these pilot-specific needs
and expectations includes:
• discussions between the EPA pilot managers, verification partners and
stakeholders
• development by EPA pilot managers, verification partners and stakeholders of pilot
objectives for use prior to testing
NOTE: Not all pilots are structured the same. For example, the independent pilot uses an
advisory committee, an expert review, and an alliance group in determining pilot-specific
needs and expectations.
1.3.2 Internal customers of the ETV program are those EPA staff responsible for execution of
the ET V.program in accordance with the expectations of Congress and the
Administration. These customers include EPA and ORD senior managers who expect
confbrmance with management and quality policies of the Agency
Other EPA staff, such as EPA technical experts in the regions and headquarters, will
benefit incidentally from the program in the following areas:
• data of known and useful quality
• expedited use of improved environmental technologies
• ETV testing accomplished on a wide variety of technologies
• user-friendly documents (e.g., easy to read and to implement)
• development of appropriate verification testing protocols
1.4 Potential verification partners
EPA line management shall oversee the selection ofETV verification partners.
Each ETV pilot seeks to evaluate a wide variety of verification partnerships, using both
interagency and cooperative agreements. When cooperative agreement holders conduct
the verification testing, they are competitively solicited using the Request for Application
(RFA) process, whereby notice of EPA's intent to issue an RFA is published, typically in
the Commerce Business Daily (CBD).
ETVQMP
Page: 6 of 41
Version: 1.0
Date: May 26,1998
-------
1.5 Management resolution for verification partner quality constraints
When necessary, appropriate EPA management shall negotiate acceptable measures of quality and success when constraints
of time, costs, or other problems affect the verification partner's capability to fully satisfy customer's needs and expectations.
When constraints of time, costs, or other problems significantly affect the verification
partner's capability to fully satisfy the ETV's quality system needs and expectations, EPA
pilot managers negotiate with the verification partner by the following procedure to
establish acceptable measures of quality and success:
• If any one of the above problems occur, the verification partner notifies the EPA
pilot manager.
• The EPA pilot manager negotiates and documents an acceptable agreement with
the verification partner.
• If agreement cannot be reached between the EPA pilot manager and the
verification partner, the negotiation discussion will be elevated to the appropriate
EPA branch chief, division director, and/or ETV coordinator.
• If agreement cannot be reached between EPA management and the verification
partner, appropriate actions will be performed (e.g., increased funding for the
benefit of the program, non-funding of extensions, or non-support for verifications
statements).
• If negotiation of issues reaches a stalemate, the Grants Administration Division
(GAD) will be notified for possible legal action.
1.6 Resources
The laboratory directors shall provide adequate resources to the ETV directors of quality assurance, EPA pilot managers and
EPA pilot quality managers to enable them to plan, implement, assess, and improve the overall ETV program and quality
system effectively.
Laboratory directors take the following actions to achieve the above policy:
• provide FTE allotment of EPA pilot managers
• provide FTE allotment of QA and other support personnel at each laboratory's
geographical location
• provide sufficient travel funds for each pilot for an appropriate level of oversight
and external assessments
• provide for maintenance of communication lines between ORD laboratory
directors, the ETV team, and the ETV coordinator
ETVQMP
Page: 7 of 41
Version: 1.0
Date: May 26,1998
-------
1.7 Authority to stop work for safety and quality considerations
The verification partner shall stop unsafe work and work of inadequate quality, or shall delegate the authority to do so to
others.
The following procedures are necessary to stop unsafe work and work of inadequate
quality:
The verification partners shall ensure compliance with all federal, state, and local
health and safety policies during the performance of the pilot tests. This includes
obtaining appropriate permits.
The verification partner's quality system shall identify one or more individuals who
may issue a stop work order in the event that unsafe work or work of inadequate
quality is identified.
EPA pilot managers and EPA pilot quality managers shall contact the authorized
indivjdual(s) in the event that work of inadequate quality is discovered.
• In extreme circumstances, the EPA pilot managers may ask GAD to intervene if
the verification partner does not implement their approved quality managment
plan.
2.0 QUALITY SYSTEM AND DESCRIPTION
A quality system shall be planned, established, documented, implemented, and assessed as an integral part of an ETV management system for
environmental technology verification programs defined by ETV quality policy.
Development and subsequent endorsement of this plan by the ETV coordinator and EPA
line management are evidence that the ETV quality system is planned, established,
documented, implemented, and assessed as an integral part of an EPA ETV management
system.
2.1 Authorities and conformance to E4 quality standard
The ETV quality system shall address applicable parts ofE4 and shall include the organizational structure, policies and procedures,
responsibilities, authorities, resources, and guidance documents.
The authority for developing appropriate quality systems for ETV is USEPA Order
5360.1. The requirement for assistance agreement holders is found in Federal Register,
CFR Parts 30 & 33, February 15, 1996.
This plan complies with ANSI/ASQC E4-1994, Specifications and Guidance for Quality
Systems for Environmental Data Collection and Environmental Technology Programs,
the Agency standard applicable to assistance agreements. E4 is comparable to the
International Standards Organization (ISO) 9000 standards series, as shown in the
comparison table provided in Annex B-5 to E4.
The ETV quality system addresses each applicable individual "specification" provided in
the published quality standard, ANSI/ASQC E4-1994, using the policies and procedures in
this plan, as appropriate.
ETVQMP
Page: 8 of 41
Version:' 1.0 '
Date: May 26, 1998
-------
Verification partners develop quality system descriptions to be consistent with both
ANSI/ASQC E4-1994 (and/or ISO 9001) and/or this document.
2.2 Quality system documents
The EW quality system shall be described in a QMP that is reviewed and approved by the ETV coordinator and EPA line management.
The ETV quality system is described in this quality and management plan.
• The ETV team develops and implements the quality system.
• The ETV coordinator, ORD laboratory directors, and appropriate ORD division
directors review and approve the ETV QMP and subsequent revisions to the plan,
as policy for the ETV program.
• Verification partners' quality systems (which are consistent with ANSI/ASQC E4-
1994) are described in a written ETV pilot quality management plan, and are.
reviewed and approved by verification partner management, the EPA pilot
manager, and EPA pilot quality manager. Subsequent revisions are reviewed in a
similar manner.
2.3 Quality system scope
The ETV quality system description shall identify in general terms those items, programs, or activities to which it applies.
This quality system description applies to the following:
• the EPA ETV program
• selection and oversight of verification partners
• review and approval of verification partners' quality management plans
• ETV products (e.g., test/QA plans, reports, ETV verification statements)
• planning, implementation, and assessment activities supporting ETV verification
activities
2.4 Quality expectation for products and services
The ETV quality system shall include provisions to ensure that products or results of the environmental programs defined by the ETV program
are of the type and quality needed and expected by ETV clients.
The preeminent products of the ETV program are the environmental technology
verification reports and statements issued by EPA and the verification partner. Provisions
to ensure that these products and other results of the ETV program are of the quality
expected include:
Products are reviewed as described in part A, section 5.0,.
MSRs and technical assessments are conducted as described in part A, section 9.0.
Technical assessments may include field and laboratory audits, performance
evaluation audits, and audits of data quality.
ETV QMP
Page: 9 of 41
9
Version: 1.0
Date: May 26,1998
-------
2.5 Quality procedures documentation
Following approval of the ETVQMP, management elements of the quality system shall be implemented as described.
Verification partners must operate the ETV pilots under a written and EPA-approved
quality management plan that is based on E4 and/or the provisions of this plan.
• Environmental technology pilots initiated prior to approval of the ETV QMP will
be requested to provide quality system elements consistent with requirements of
their assistance agreements with EPA. The ETV coordinator requests that existing
pilots comply with the requirements set forth in this plan within 6 months of the
date of this document.
For pilots initiated after the approval of the ETV QMP, verification partners
provide evidence of compliance before verification activities are started, as
required by the assistance agreement signed by the verification partner.
The EPA pilot manager is responsible for obtaining a copy of the verification partner's
quality management plan, as specified in the RFA, for their own review and forwarding the
document to the EPA pilot quality manager for review and approval prior to planning
technology tests.
2.6 Quality controls
The ETV quality system description shall define when and how controls are to be applied to specific technical or technology testing efforts and
shall outline how these efforts are planned, implemented, and assessed.
2.6.1 ETV program controls include:
• existing EPA policies and procedures for selection and administration of
verification partner efforts in ETV
an approved ETV QMP
• quality management, assurance, and control procedures as part of the assistance
agreement
2.6.2 Pilot-specific controls include:
• test/QA plans, developed and approved prior to testing
• oversight by the EPA pilot quality managers of the implementation process and
follow-up to any finding of nonconformance
• technical operations assessment
• specified quality control requirements in the assistance agreement
Pilot-specific procedures for planning, implementation, and assessment are described in
the verification partner's quality system. Procedures for planning, implementing, and
assessing the overall ETV quality system are detailed in part A sections 7.0, 8.0, and 9.0
and in part B.
ETVQMP Version: 1.0
Page: 10 of 41 10 Date: May 26,1998
-------
2.7 Management system reviews (MSRs)
At regular intervals (at least annually) the ETY quality system shall be reviewed and its description updated, if appropriate, to reflect changes in
the organization as well as changes in ETVquality policy. •'••-' ;
The ETV directors of quality assurance perform an internal MSR of the program in
accordance with the process as outlined in part A section 9.0.
3.0 PERSONNEL QUALIFICATION AND TRAINING
3.1 Personnel training and qualification procedures
Personnel performing -work shall be trained and qualified based on appropriate requirements prior to the start of the -work or
activity.
3.1.1
3.1.2
3.1.3
EPA pilot managers are selected based on:
• educational background and/or a degree that is directly relevant to the pilot
technology area
• work experience specific to the pilot technology area
• experience in program management
• participation in required training for project officer responsibilities on assistance
agreements, as documented in training records
EPA pilot quality managers are selected based on:
• educational background and/or a degree relevant to the technology tests and
programs
• work experience specific to QA of technology tests and programs
• experience in quality management
Key participants working directly for or on behalf of the verification partner in support of
the pilot and/or individual test operations are selected by the verification partner and
evaluated by the EPA during the RFA process. RFA evaluation criteria for key personnel
will vary, but typically include a consideration of the following:
• educational background and/or a degree(s) relevant to technical areas represented
in the pilot
• work experience related to the technology areas represented in the pilot
• experience in quality management
The verification partner's documented quality management plan will address training and
qualification procedures for pilot personnel.
ETVQMP
Page: 11 of 41
11
Version: 1.0
Date: May 26, 1998
-------
3.2 Formal qualifications and certifications
Tlte need to require formal qualification or certification of personnel performing certain specialized activities shall be
evaluated and implemented -where necessary.
ETV program management, quality management, and pilot management requires no
formal qualification or certification other than where applicable:
• EPA Project Officer Training and Assistance Training (or Work Assignment
Manager training as appropriate)
• Appropriate OSHA courses
Formal qualification or certification of personnel performing specialized activities for each
pilot or for specific test/QA plans is addressed on a pilot-specific or test/QA plan-specific
basis. Verification partners maintain records of the qualification or certification of such
personnel.
NOTE: Requirements for formal qualifications or certification may be based on applicable
federal, state, or local requirements associated with a particular test. Examples of possible
certifications include but are not limited to drinking water plant operators certification,
professional engineering registration, and certification of industrial hygienists.
3.3 Technical management and training
Appropriate technical and management training, -which may include classroom and on-the-job, shall be performed and
documented.
EPA line management is responsible for appropriate technical and management training
for staff working on the ETV program. Such training will be documented in each
individual's training file.
Verification partners are responsible for personnel training and qualification procedures
for each pilot or for specific test/QA plans. Verification partners maintain the training
records (available for review by EPA).
The ETV team will be trained at meetings occurring at least twice a year to develop policy
and share information and lessons learned. The directors of quality assurance provide
training on the requirements of the ETV QMP during the periodic workshops organized
by the ETV coordinator.
ETV QMP Version: 1.0
Page: 12 of 41 1-^ Date: May 26,1998
-------
3.4 Retraining
When job requirements change, the need for retraining to ensure continued satisfactory job proficiency shall be evaluated.
The need for retraining EPA staff is evaluated on an annual basis by the appropriate line
management.
Verification partners are responsible for retraining for each pilot or for specific test/QA
plans.
3.5 Personnel job proficiency
Evidence of personnel Job proficiency shall be documented and maintained for the duration of the technology test or activity
affected, or longer if required.
3.5.1 EPA pilot managers - The existing performance standards of the EPA pilot managers may
already include tasks consistent with the following items. These items should be
considered for specific identification in the performance standards:
• active participation in the ETV team; communicating pilot issues, lessons learned,
required reports, and appropriate assistance to members of the ETV team and
management
• developing RFAs and/or management of cooperative agreements/IAGs
• facilitating stakeholders group activities
• ensuring development of and contributing to generic verification protocols and
test/QA plans
• providing a leadership role to ensure technologies are selected consistent with the
ETV Verification Strategy
• serving as a communication link between EPA and the verification partner, in
particular, providing information and documents to support the ETV website
• reviewing draft and final ETV verification reports and other pilot documents
• reporting to program management the completeness and validity of the ETV
verification statement prior to report issuance
• ensuring the timely delivery of complete and consistent ETV products and services
NOTE: Evaluations are the responsibility of the appropriate supervisor and are not a
record of the ETV program.
ETVQMP
Page: 13 of 41
13
Version: 1.0
Date: May 26,1998
-------
3.5.2 EPA pilot quality managers -The existing performance standards of the EPA pilot quality
managers may already include tasks consistent with the following items. These items
should be considered for specific identification in the performance standards:
• verification partner pilot quality management plan review
• management system reviews
• technical system audits, performance evaluation audits, and audits of data quality
• verification reports and statements reviews
• providing complete and timely pilot audit reports
NOTE: Evaluations are the responsibility of the appropriate supervisor and are not a
record of the ETV program.
3.5.3 Verification partner staff- Verification partners document and maintain records (such as
annual performance reviews) of personnel job proficiency for work performed directly in
support of the verification partner's ETV activities.
NOTE: Evaluations are the responsibility of the verification partner and are not a record
of the ETV program.
4.0 ETV VERIFICATION PARTNER SELECTION
4.1 Planning and control of selection process
Funding of assistance associated with the ETV program shall be planned and controlled to ensure that the quality of
verification tests is known, documented, and meets technical requirements and acceptance criteria of the clients.
The ETV program is designed to investigate ways to facilitate the verification and use of
environmental technology and exists solely for the benefit of industry and the user
community. Two pilots operate under interagency agreements; however, the ETV
program primarily operates by securing the cooperation of verification partners through
open competition utilizing the agency's assistance agreement program. Agency
procedures for advertising, reviewing, and awarding assistance agreements are followed
during the selection process. The procedures governing this process are available from
GAD and are not discussed in this section. The procedures used to plan and control the
selection of verification partners for ETV are listed below.
Planning to select verification partners requires:
• assessing and prioritizing environmental technology categories for use in pilot
projects (i.e., defining the scope of each pilot project in terms of technology areas
to be tested)
• establishing ANSI/ASQC E4-1994 as an applicable quality standard
• issuing RFA's
• selecting the appropriate verification partner based on their experience and
proficiency
ETVQMP Version: 1.0
Page: 14 of 41 14 Date: May 26,1998
-------
Controlling the selection process to ensure the quality of verification tests includes:
• implementing controls stipulated in EPA policies and procedures for assistance
agreements
« establishing specific language in each RFA requiring development and
implementation of a quality system consistent with the ETV quality system, and
ANSI/ASQC E4-1994.
• reviewing the applicant's proposed quality system to verify it meets the RFA
requirement and provides for quality of verification tests which will be known,
documented, and meet technical requirements
4.2 Technical and quality requirements
Assistance solicitation documents shall contain information clearly describing the technical and quality requirements
associated -with the verification testing.
Technical and quality requirements expressed in the RFA include technical evaluation
criteria for technical skills and experience of staff members, and demonstrated experience
in the development of quality systems relevant to ETV. Cooperative agreements require
that a verification partner, once selected, develops and submits for approval to EPA, a
pilot quality management, plan consistent with E4, prior to conducting technical activities.
4.3 Quality specification/conformance
Assistance solicitation documents shall specify the ETV quality requirements for -which the verification partner is responsible
and how the verification partner's conformance to client's requirements shall be verified.
ETV quality requirements for which the verification partner is responsible are specified in
the RFA and in this ETV QMP. During verification partner selection, the applicants'
proposals and written responses to the requirements are reviewed for conformance to the
RFA specifications. After a verification partner is selected, the EPA pilot quality manager
reviews and the EPA pilot manager approves written quality system documents (e.g., pilot
QMP) for conformance to the EPA and ETV quality policies and procedures.
4.4 Peer review of assistance agreements
Assistance award documents shall be reviewed for accuracy and completeness by qualified personnel prior to award.
Peer review is an integral part of EPA's project planning, implementation, and assessment
process. RFA packages are internally peer reviewed prior to their issuance. Responses to
the RFA undergo a peer review process which supports the award of the assistance
agreement.
ETV QMP
Page: 15 of 41
15
. Version: 1.0
Date: May 26, 1998
-------
4.5 Conformance of Verification Testing Efforts
Appropriate measures shall be established to ensure that the verification testing efforts satisjy all terms and conditions of the
assistance agreement. Verification partners shall have a demonstrated capability to meet all terms and conditions.
Once a verification partner has been selected, measures to ensure continued conformance
to terms and conditions in the assistance agreement are implemented as described in part
A sections 8, 9, and 10.
5.0 RECORDS
5.1 Scope
Procedures shall be established, controlled, and maintained for identifying, preparing, reviewing, approving, revising,
collecting, indexing, filing, storing, maintaining, retrieving, distributing, and disposing of pertinent quality documents and
records. Such procedures shall be applicable to all forms of documents and records, including printed and electronic media.
Measures shall be taken to ensure that users understand the documents to be used. Records requiring control shall be
identified.
Records to which this policy applies include:
- ETV Verification Strategy
- ETV QMP (this document)
- cooperative agreement and interagency agreement records
- the verification partners' quality management plans
- minutes of stakeholder meetings
- generic verification protocols (how a given type of technology is verified)
- Test/QA plans (procedures for an individual test, including SOPs)
- raw data (all written and electronic data generated when tests are conducted)
- ETV verification reports (comprehensive reports on a technology verification project)
- ETV verification statements (summary statement for an individual technology test)
- annual pilot progress reports
- EPA reviews and audit reports
- verification partner reviews and audit reports
Information in this section applies to both electronic and printed records, as well as
original records developed on behalf of the ETV program that are required to demonstrate
the quality of information and data provided in ETV verification reports.
5.2 Preparation, review, approval, and distribution
Sufficient records shall be specified, prepared, reviewed, authenticated, and maintained to reflect the achievement of the
required quality for completed work and/or to fulfill any statutory requirements. Documents used to perform -work shall be
identified and kept current for use by personnel performing the work. Documents, including revisions, shall be reviewed by
qualified personnel for conformance with technical requirements and quality system requirements and approved for release by
authorized personnel.
Table 5.1 lists the pertinent quality records for ETV, the person(s) responsible for
preparing and updating these records, the reviewers, those given approval authority for
ETV QMP Version: 1.0
Page: 16 of 41 16 Date: May 26,1998
-------
each record type, and the distribution plan. Where a procedure is not applicable (e.g., a
document is not subject to approval), N/A is entered in Table 5.1. All reviewers and
approving officials receive copies of the records they review/approve; the Distribution
column in Table 5.1 lists only those individuals who receive final copies, in addition to the
reviewers and approving official. For revised documents, these same review, approval,
and distribution pathways are followed. Unless otherwise noted, material placed on the
ETV website is available for public inspection, comment, and use.
5.3 Records Storage and Obsolete records
Obsolete or superseded documents shall be identified and measures shall be taken to prevent their use, including removal from
the workplace and from the possession of users -when practical. Maintenance of records shall include provisions for
retention, protection, preservation, traceability, and retrievableness. While in storage, records shall be protected from
damage, loss, and deterioration. Retention times for records shall be determined based on contractual and statutory
requirements, or, if none stated, as specified by the EPA coordinator and EPA line management.
Obsolete records should be clearly marked as such. These records may be retained in the
workplace for historical reference, or they may be removed to archival storage. ETV will
follow ORD'S Records Management Policy, Part 003 (see Appendix A), which addresses
requirements for indexing, filing, maintaining, retrieving, and disposing of documents and
records from all extramural financial agreements. The current minimum requirement is
that all records be kept for seven years after the final payment on a cooperative agreement
or interagency agreement.
ETVQMP
Page: 17 of 41
17
Version: 1.0
Date: May 26,1998
-------
I
O
-t I
2
VO
J1
>
18
-------
6.0 COMPUTER HARDWARE AND SOFTWARE
l
6.1 General procedures
Computer software and computer hardware configurations used in the ETVprogram shall be
installed/tested/used/maintained/controlled/documented to meet users'requirements and shall conform to this quality policy
and applicable consensus standards and/or data management criteria.
At the program level, ETV does not expect to develop software. At the pilot level, if
verification partners intend to develop software to support their ETV process (or an
individual test/QA plan), the partner should have procedures in place as specified here. If
the verification partner uses only commercial software for office operations (e.g.. word
processing software, spreadsheet software), it is unlikely that the partner would need
specific procedures for assessing software quality. Part A, sections 6.2 through 6.6, apply
only to software and software/hardware configurations developed specifically for the ETV
program.
The following are the ETV program procedures which ensure that each pilot controls the
quality of all computer hardware/software configurations for the program.
The EPA pilot manager and the verification partner discuss and agree upon the
computer hardware and software requirements of the pilot and/or specific test/QA
plan.
• Once decisions are finalized, the verification partner supplies evidence of meeting
all requirements before data collection, reduction, or validation procedures begin.
For software developed for ETV programs, the verification partner tests all
applications and configurations using a test data set or by running a shakedown
test of the system to ensure all applications/configurations are operating to
specifications. The verification partner must show evidence of a system to
maintain, control, and document such software and hardware configurations. This
includes, but is not exclusive of; resources to correct any hardware/software failure
with minimal downtime to the program, tracking upgrades/revisions to software or
configuration changes, documenting software names, versions, and copyright
dates, and complete documentation of the code. Complete documentation of code
includes the written code with comments structured in a modular form.
ETVQMP
Page: 19 of 41
19
Version: 1.0
Date: May 26, 1998
-------
6.2 Scope of ETV computer hardware/software procedures
Computer software and computer hardware/software configurations covered by ETV's quality policy include, but are not
limited to:
• operation or process control of environmental technology systems (including automated data acquisition and
laboratory instrumentation)
• and data bases containing environmental data
Computer software and computer hardware/software configurations covered by this
quality and management plan include all agreed upon, pilot-specific applications or
configurations. These include, but are not limited to:
• evaluating and reducing environmental data
• reporting environmental data
• data bases containing environmental data
6.3 Configuration testing
Computer hardware/software configurations shall be tested prior to actual use and the results shall be documented and
maintained.
On a pilot level, the verification partner conducts tests of the computer hardware/software
configuration using a standard set of testing conditions.
NOTE: Verification partner is required to have a system to document all testing of
computer hardware/software configurations, as required by part A section 6.1. A test data
set or a standard set of testing conditions should be developed on a pilot- or test/QA plan-
specific basis. Maintenance testing should be easily tracked and retrievable.
6.4 Measurement and testing equipment configurations
Computer hardware/software configurations integral to measurement and testing equipment that are calibrated for a specific
purpose do not require further testing unless:
• the scope of the software usage changes OR
• modifications are made to the hardware/software configuration
On a pilot level, verification partners perform the following procedures (as provided in the
verification partner's quality system).
• Whenever computer hardware/software configurations integral to measurement
and testing equipment are calibrated for a specific purpose, further testing is not
normally performed unless the scope of the software usage changes or
modifications are made to the hardware/software configuration.
In the event either of the above mentioned changes occurs, the verification partner
retests the changes as described in part A sections 6.1 and 6.3. Retesting is
documented to the same extent as the original application/configuration.
ETVQMP
Page: 20 of 41
20
Version: 1.0 •
Date: May 26,1998
-------
6.5 Change assessments - configurations, components, and requirements
Changes to hardware/software configurations, components, or program requirements shall be assessed to determine the impact
of the change on the technical and quality objectives of the ETVprogram supported.
The verification partner is responsible for assessing the changes, determining the need for
testing, and reporting the assessments to the pilot manager.
6.6 ETV website roles and responsibilities
The ETV -website shall be operated in such a way that it serves all ETV participants and customers through prompt and
accurate posting of ETV information and documents.
The pilot managers, or alternate(s) designated in writing by the pilot manager, are
responsible for sending the following information to the ETV webmaster:
• general fact sheets and brochures
• stakeholders lists (and updates)
• meeting announcements and summaries
• generic testing protocols (indicating draft or final)
• test/QA plans (indicating draft or final)
• CBD announcements
• ETV verification statements
• upcoming meetings/speeches/announcements
7.0 PLANNING
7.1 Systematic planning process
A systematic planning process shall be established, implemented, controlled, and documented to:
• identify the customers), and their needs and expectations
• identify the technical and quality goals that meet the needs and expectations of the customer
• translate the technical and quality goals into specifications that shall produce the desired result
• consider any cost and schedule constraints -within which technology test activities are required to be performed
• identify acceptance criteria for the results or measures of performance by which the results shall be evaluated and
customer satisfaction shall be determined
7.1.1 Systematic Planning Process Established
The planning process for ETV is conducted as follows:
EPA establishes the number and type of ETV pilots necessary to comply with the
Presidential mandate to cover all environmental technologies within three years.
• EPA lays out basic program operation parameters in two documents, the ETV
Verification Strategy and ETV QMP.
Based upon the ETV Verification Strategy, the ETV coordinator, in consultation
with the Agency's Innovative Technology Council, the ETV team, and EPA line
management, designs an annual budget.
Appropriate personnel are appointed from within the ORD laboratories to fill ETV
positions. Selection and information on qualification is presented in part A
sections 3.1 and 3.2.
ETV QMP
Page: 21 of 41
21
Version: 1.0
Date: May 26,1998
-------
• Resources and planning to support EPA staff assigned to the pilot, such as training
and travel, are provided by the division and branch management. Technical
training is discussed in part A sections 3.3 and 3.4.
• At the beginning of each pilot, verification partners are selected in conformance
with part A section 4.0. EPA's requirements for the appropriate extramural
agreement (e.g., cooperative agreement, interagency agreement) must also be met.
• After selection, the verification partner, in consultation with the EPA pilot
manager, establishes a stakeholders group that contains representatives of
customer groups of concern to that pilot area. [Note: in order to evaluate the
efficacy of stakeholders groups, at least one pilot will not establish such a group.]
The EPA pilot manager and the verification partner develop plans for pilot
verification tests and present them to the stakeholders for review, comment, and
advice.
The EPA pilot manager, the EPA pilot quality manager, the verification partner,
and the stakeholders group hold at least one joint meeting annually to:
identify, revise, and/or clarify the technical and quality goals of the work to
be accomplished
translate the technical and quality goals into written specifications that will
be used to produce the desired result
consider any cost and schedule constraints within which test activities are
required to be performed
develop qualitative measures of performance by which the results will be
accepted
determine testing priorities and evaluate customer satisfaction
• These four parties appoint a recorder to document the minutes of each meeting.
• Minutes of each meeting are taken and then are distributed to participants for
comment. Minutes of stakeholders meetings are incorporated within the records
management scheme described in part A section 5.0.
7.1.2 Implementation of the systematic planning process
Planning is accomplished through frequent meetings among participants and through
posting initial planning documents and stakeholders meeting minutes on the ETV website.
Procedures for planning at the pilot and test level are addressed in Part B. Procedures for
implementing the planning process are detailed below:
• Customer Identification - ETV customers are identified in part A section 1.
• Technical and quality goals identification - These are identified during planning
meetings with senior management, conference calls with ETV participants, and
meetings with EPA quality professionals and technical staff.
• Technical and quality goal specifications - The ETV coordinator works with the
EPA pilot managers, EPA directors of quality assurance, and other quality
professionals to translate technical and quality goals of the overall program into
theETVQMP.
ETVQMP Version: 1.0
Page: 22 of 41 22 Date: May 26,1998
-------
• Cost and schedule constraints - These are discussed during planning meetings with
senior management and considered yearly for allocation to each pilot.
• Measures of performance - The ETV coordinator develops measures of
performance for the program which are evaluated by the ETV team and
verification partners throughout the course of the program. Appendix B contains
the most current measures of performance.
• Customer satisfaction evaluation - The program evaluates all pilots on an annual
basis to collect data for a final report on the pilot phase. This evaluation includes
customer satisfaction measures as appropriate.
7.1.3 Systematic planning process control
Planning process controls include:
• development and implementation of written procedures (verification test protocols
and test/QA plans)
• requirement of minutes of stakeholders group meetings
• review of verification partner work efforts by the EPA pilot manager and EPA
pilot quality manager
7.1.4 Systematic planning process documents
Planning process documentation includes the ETV Verification Strategy, the ETV QMP,
the verification partners' QMPs, and test/QA plans.
7.2 Planning document review
All planning documentation shall be reviewed and approved for implementation by authorized personnel before the specific
work commences. Such documentation includes but is not limited to test/QA plans and generic verification test protocols.
Planning document review is discussed in part A section 5.2.
8.0 IMPLEMENTATION OF WORK PROCESSES
8.1 Implementation
Work shall be performed according to approved planning and technical documents.
The planning for the implementation of the EPA management and quality work processes
is contained in part A section 7.0. The individual ETV pilot work is performed according
to planning documents written by the pilot. All technology verification work shall occur
according to protocols and test/QA plans developed and agreed upon by EPA, the
verification partner, and the vendor. The authors, reviewers, and approvers of these
documents are specified in part A section 5.0, Table 5.1.
ETV QMP
Page: 23 of 41
23
Version: 1.0
Date: May 26, 1998
-------
The approved protocols and test/QA plans shall be present on the site of testing, and the
work shall be implemented in accordance with them. During the work phase,
modifications to plans and procedures shall be documented, and the modifications shall be
incorporated into the final protocols and test/QA plans. The authors, reviewers, and
approvers of changes to these documents are the same as for the original documents and
are specified in part A section 5.0, Table 5.1.
Verification partners are responsible for implementing their work processes in accordance
with their quality systems.
8.2 Procedures
Procedures shall be developed, documented, and implemented for appropriate routine, standardized, special, or critical
operations. Operations needing procedures shall be identified. The form, content, and applicability shall be addressed, and
the reviewers and approvers shall be specified.
Procedures for the overall operation of the ETV program are contained in the ETV
Verification Strategy, the ETV QMP and in other appropriate EPA policies (e.g.,
contractual, records management). The individual ETV pilots shall identify and document
those operations in their pilots requiring procedures as discussed in Part B. Procedures
shall be written in a format that can be readily comprehended by the user and shall contain
sufficient detail and clarity to ensure that results are achieved effectively. Appropriate
operations documents, authors, reviewers, and approvers are specified in part A section
5.0, Table 5.1.
8.3 Oversight
Implementation of work shall be accomplished with a level of management oversight and inspection commensurate with the importance of the program
and the intended use of the results, and shall include the routine measurement of performance against established technical and quality specifications.
EPA line management has responsibility for oversight of verification work processes as
discussed in part A section 1. Verification partner oversight and responsibilities for the
verification work processes are given in the individual pilot QMPs.
9.0 ASSESSMENT AND RESPONSE
9.1 Numbers and types of assessments
Assessments shall be planned, scheduled, and conducted to measure the effectiveness of the implemented management and
quality systems. Several types of assessments are available for this purpose. Management shall determine during the planning
stage the appropriate types of assessment activities. Assessments shall include an evaluation to determine and verify whether
technical requirements, not just procedural compliance, are being implemented effectively.
The assessments shown in Table 9.1 and the minimum frequency are commensurate with
the importance of the ETV program and the intended use of the verification results.
Management assessments shall be used to measure the effectiveness of the implemented
management systems and technical systems. Performance assessments shall be used to
ETV QMP
Page: 24 of 41
24
Version:" 1.0
Date: May 26,1998
-------
evaluate performance of the pilot technical operations. Data assessments shall assess
reported data quality. Verification partners perform self-assessments in accordance with
the individual pilot management plans, and EPA performs independent assessments of
verification partners.
9.2 Procedures
Assessments shall be performed according to written and approved procedures, based on careful planning of the scope of the
assessment and the information needed. Assessment results shall be documented and reported to management. Management
shall review the assessments.
Assessments shall be planned according to the scope of the assessment and the
information needed. Suitable written procedures for planning and conducting audits shall
be contained in the operating manuals of EPA quality teams, the operating and quality
manuals of the verification partners, and EPA guidance documents. Assessments are
based on interviews, on the physical examination of objective evidence, and on the
examination of the documentation of past performance. Results are documented in audit
reports, and reviewed by appropriate management.
9.3 Personnel qualifications, responsibility, and authority
Personnel conducting assessments shall have the appropriate technical or management skills to perform the assigned
assessment. Management shall determine and document the level of competence, experience, and training necessary to ensure
the capability of personnel conducting assessments. The responsibilities and authorities of personnel conducting assessments
shall be clearly defined and documented, particularly in regard to authority to suspend or stop work in progress upon
detection and identification of an immediate adverse condition affecting the quality of results or the health and safety of
personnel.
EPA and verification partner management determine and document the level of
competence, experience, and training of their respective audit personnel during hiring and
periodic performance reviews. Qualified audit personnel, as listed in Table 9.1, have
access to the appropriate management personnel and documents required to perform their
audit duties. They are organizationally independent of the program or pilot they are
auditing. They have the responsibility and authority to:
identify and document problems that affect quality of verification results
propose recommendations for resolving problems that affect quality of verification
work processes or results
• independently confirm implementation and effectiveness of solutions
If auditors identify a severe problem affecting verification quality, EPA pilot managers
have the authority to request of the verification partner manager that work be stopped
until the problem is addressed. If auditors identify a problem where the health and safety
of personnel are in danger, they have the responsibility to bring it to the immediate
attention of appropriate EPA management, verification partner management, and onsite
testing personnel.
ETVQMP
Page: 25 of 41
25
Version: 1.0
Date: May 26, 1998
-------
1
i
t
a
2
42
01
I
v>
CO
CU
u e
•si
§ 8
cr
r2
fa
S
i
- I
« i
f ^
• M* V
in ui
en CA
ai T3
« B
s S,
s s
•< OS
Assessors
ent
Ass
Too
I
:S
,w
1
I
cx
dir
ass
agement
ems revi
CT1
t"
o
J
a1
« oo
a
2
'5.
a-
'S u
.•^ o
gement
ms review
•
S
&
^_» ,4_»
O O
1 «3
•*~* es S
5 8 S.
-ft-1
« S «s s
II
IS S
tem
cal
audits
«.1?
OT 3
ti •«
o o
w w
s
&4
I
a o
-
••3.8.1
a. a, ii
-------
9.4 Response
Responses to adverse conclusions from the findings and recommendations of assessments shall be made in a timely manner.
Conditions needing corrective action shall be identified and the appropriate response made promptly. Follow-up action shall
be taken and documented to confirm the implementation and effectiveness of the response action.
When the recommendations and conclusions from the findings of assessments are adverse,
response from the auditee detailing the corrective action shall be expected within 10
working days of receiving the audit report. Auditors shall follow up with appropriate
documentation to confirm the implementation and effectiveness of the response.
10.0 QUALITY IMPROVEMENT
10.1 Annual review for quality improvement
A quality improvement process shall be established and implemented to continuously develop and improve the ETV Quality
System.
The ETV coordinator and EPA directors of quality assurance review the quality and
management plan annually and recommend improvements to the plan.
The EPA directors of quality assurance recommend improvements and negotiate
improvements with the ETV team during the annual meeting and through the ETV
website.
10.2 Detecting and correcting quality system problems
Procedures shall be established and implemented to prevent as -well as detect and correct problems that adversely affect
quality during all phases of technical and management activities.
EPA pilot managers and EPA pilot quality managers report problems in any of the areas to
EPA line management and the ETV directors of quality assurance:
• adequacy of the ETV quality system
• consistency of the quality system
• implementation of the quality system
• correction of quality system procedures
• completeness of documented information
• quality of data
• quality of planning documents
• implementation of the work process
EPA line managers respond promptly to address correction of the quality problem.
ETVQMP
Page: 27 of 41
27
Version: 1.0
Date: May 26,1998
-------
10.3 Cause and effect relationship
When problems are found to be significant, the relationship between cause and effect and the root cause shall be determined.
The following are general procedures. Specific procedures are found in the individual
verification partners written quality systems. When problems are significant, the quality
manager determines and documents the relationship between cause and effect, and when
possible, determines and documents the root cause of the problem. The quality manager
provides this information to the appropriate project managers so corrective action can be
authorized and implemented.
A significant problem is any problem requiring:
• a testing protocol change OR
• a management system change OR
• a quality system change (either internal or external to EPA, but still within the
ETV program)
NOTE: The verification partner quality managers in accordance with their quality systems
are continually reviewing and assessing their projects for conformance with their quality
documents. At the program level, assessment reports from the individual projects are
monitored and evaluated by the ETV directors of quality assurance for trends or recurring
problems that are indicative of significant problems affecting the ETV program as a whole.
Any such situation is immediately communicated to the ETV coordinator. The ETV
coordinator shares the information and any corrective actions with the EPA pilot
managers.
10.4 Root cause
Tile root cause should be determined before permanent preventative measures are planned and implemented.
To guard against implementing ineffective changes, EPA personnel ensure when possible
that root causes are determined before preventative measures are planned and
implemented.
10.5 Quality improvement action
Appropriate actions shall be planned, documented, and implemented in response to findings in a timely manner.
In the event that a significant problem is identified that requires a structural change to the
ETV program, the ETV Coordinator will initiate discussions with EPA line management
appropriate to correct the deficiency.
ETV QMP Version: 1.0
Page: 28 of 41 28 Date: May 26,1998
-------
PARTB
COLLECTION AND EVALUATION OF
ENVIRONMENTAL DATA
Part A of the ETV Quality and Management Plan contains the specifications and guidelines that
are applicable to common or routine quality management functions and activities necessary to
support the ETV program.
Part B of the ETV Quality and Management Plan contains the specifications and guidelines that
apply to test-specific environmental activities involving the generation, collection, analysis,
evaluation, and reporting of test data.
ETVQMP
Page: 29 of 41
29
Version: 1.0
Date: May 26,1998
-------
1.0 PLANNING AND SCOPING
The work of the ETV program at the pilot level is to verify the performance of
commercial-ready technologies. As discussed in part A section 4.0, the planning process
begins with the Statement of Work (SOW) contained in the Request for Applications
(RFA). The successful applicant becomes the verification partner for the pilot.
1.1 Systematic planning of the verification test
All work involving the generation, acquisition, and use of environmental data shall be planned and documented. The type and
quality of environmental data needed for their intended use shall be identified and documented using a systematic planning
process. The test-specific planning must involve the key users and customers of the data. EPA pilot managers should guide
planning activities and ensure that participants are informed of and understand completely the requirements of each test.
The programmatic planning for verification of commercial-ready technologies is discussed
in part A section 7.1.1. This section continues the discussion of systematic planning at the
pilot level.
Verification partners, working with the EPA pilot managers, begin a systematic process to
plan the individual pilot tests. Systematic planning may be accomplished through any
demonstrated technique including the data quality objectives process (EPA QA/G-4) and
the observational method. The planners perform the following actions:
• refine the scope of their respective technology areas
• determine interest in verification from the manufacturers of commercial-ready
technologies within the defined scope of the technology areas
• convene stakeholder groups, containing representatives of verification customer
groups, which advise during the planning process
• mediate and facilitate the selection of focus areas
• prepare generic verification protocols which are developed to promote uniform
testing for a given type of technology
• coordinate the review and revision of the protocols (See the review and approval
scheme in part A section 5.0.) keeping in mind both the customers' and EPA's
objectives for verification as defined in the ETV Strategy
• solicit vendor agreements to participate in verification of their products based on
the generic protocol (some iteration of the two previous points frequently occurs
here as the vendors review and request revision of portions of the generic
protocols)
prepare test/QA plans for the acquisition of data to verify the performance of the
vendors' technologies
The protocols and test/QA plans describe the experimental approach, with clearly stated test
objectives and associated quality objectives for the related measurements.
ETVQMP
Page: 30 of 41
30
Version: 1.0 '
Date: May 26,1998
-------
1.2
1.2.1
1.2.2
1.2.3
1.2.4
Systematic Planning for verification testing
Organizations that participate in the test shall participate in the planning.
The scope and objectives of the verification testing and the desired action or result from the work shall be defined
The data to be collected to achieve verification shall be identified, and the QA and QC requirements to establish the
quality of the data shall be defined.
Verification tests shall undergo a design process.
Verification tests shall be documented.
Equipment, operators, and skill levels required for the verifications shall be identified.
Any constraints (e.g.. time and budget) shall be identified.
Conditions which will suspend work shall be identified.
Assessment tools shall be determined.
Methods and procedures for storing, retrieving, analyzing, and reporting the data shall be identified.
Methods and procedures for minimizing, characterizing, and disposal of hazardous waste generated during the test
shall be identified.
Planning personnel
The verification partner shall coordinate test planning among the participating
organizations including EPA, the stakeholders, the vendors, and any testing organizations
and laboratories participating in the test. The verification partner, with the concurrence
and oversight of the EPA pilot manager, shall identify the planning roles of the various
players, and shall conduct planning activities by shared communication via teleconference,
video conference, and in-person meetings, as appropriate, and within the constraints of the
budget.
Purpose, scope and objectives
The purpose of this testing is to verify the performance of commercial-ready technologies.
Another objective is to develop an efficient method for testing commercial-ready
technologies. Many of the pilot tests accomplish this objective by preparing generic
verification protocols whereby the performance of similar technologies can be verified in
the future using the same protocol. The characteristics of individual technologies and the
specifics of individual tests are covered in the test/QA plan which incorporates the generic
verification protocol by reference.
Data to be collected and design of experiment
During planning of the technology verification test, the process, environmental, laboratory,
response, and QA data to be collected are identified. Also identified are testing
organizations, test personnel, skill levels, methods, procedures, and equipment unique to
each verification test. Planning is integrated into design as discussed in part B section 2.0.
Documentation and reporting
Records generated during the pilot tests are listed in part A section 5.0. Records consist of
both paper and electronic records. Electronic methods for storing, retrieving, analyzing,
and reporting the data are generally commercially-available programs for word processing,
spreadsheet, or database processing, or commercial software developed especially for data
ETVQMP
Page: 31 of 41
31
Version: 1.0
Date: May 26, 1998
-------
collection and processing on a specific instrument or piece of equipment. Pilots may also
develop software/hardware configurations, as appropriate, in their technology verification
tests. The use of computer hardware and software is discussed in part A section 6.0.
Paper records such as field notebooks, bench sheets, field data sheets, custody sheets, and
instrument printouts are part of the raw data test record and kept with the study records.
1.2.5 Assessments
The assessment tools and minimum frequencies of assessments for the verification tests are
identified in part A section 9. The definitions of the assessment tools and suggested
frequencies are given in part B section 4.0.
1.2.6 Constraints, suspension of work, waste minimization and disposal
Verification partners work under the constraints of time and resources communicated to
them by the EPA ETV Coordinator and the EPA pilot manager. When constraints are
determined by the verification partner to affect quality, the resolution of the problem
proceeds as described in part A section 1.5. Circumstances under which work can be
suspended are discussed in part A section 1.7. If waste is generated as part of the
verification testing, the verification partner seeks to minimize the amount, and disposes of
it in accordance with applicable local, state, and federal laws.
2.0 DESIGN OF TECHNOLOGY VERIFICATION TESTS
2.1 Design process
The design shall incorporate those activities pertaining to verification of performance identified during the planning process,
establish test specifications, and identify appropriate controls. The design shall include
selection of field sampling or testing equipment, and its operational parameters, as appropriate
selection of field sampling or testing methods, as appropriate
sample types, numbers, quantities, handling, packaging, shipping, and custody, if applicable
sampling locations, storage, and holding times, if applicable
selection of analytical methods, quality measures of performance, analysis providers, if applicable
requirements for calibration standards, and performance evaluation samples, as appropriate
requirements for field and/or laboratory QA/QC activities
requirements for qualifications of testing, sampling and/or analysis personnel
protection of health and safety of test personnel and the public
readiness reviews prior to data collection
assessments required including technical and performance audits, audits of data quality, and assessments of data use
limitations
data reporting requirements
methods for validating and verifying the data
requirements for data security, archival, and retention
integration of time and schedule constraints
procedures for minimization or disposal of wastes generated during verification activities
ETVQMP
Page: 32 of 41
32
Version: 1.0
Date: May 26,1998
-------
2.1.1 Design technique
In designing technology performance verification operations, designers use verification
testing design techniques including statistical methods, as appropriate. The design takes
into account constraints of time, scheduling, and resources.
2.1.2 Field and laboratory equipment and methods
During the design process, the appropriate field and laboratory equipment which were
identified during planning for the testing of the technology verification performance, are
incorporated. Appropriate test methods and operating parameters are specified.
2.1.3 Sampling and analysis
The design process produces a testing plan based upon the data quality objectives for the
verification of the technology performance.
• The plan specifies the tests to be conducted, the baseline parameters, the number
of replicate tests, and the controls.
• If the testing involves samples, the plan specifies sampling methods, sample types,
numbers, quantities, handling, packaging, shipping, and custody. Also specified
are sample locations, storage conditions, and holding times.
• Analysis methods, quantitative measures of performance, calibration standards,
calibration check standards, and performance evaluation samples, as appropriate,
and as identified in the planning process, are incorporated into the design.
• Methods and procedures are included to ensure the test produces data of known
and acceptable quality.
• The design incorporates any other field or laboratory QA/QC activities identified
by planners.
The design specifies the requirements for qualifications of technical staff
responsible for obtaining, analyzing, and evaluating the data. Protection of the
health and safety of testing personnel and the public is incorporated into the
design.
• Procedures for the minimization and disposal of wastes generated are designed
into the verification activities.
2.1.4 Assessments
Assessments incorporated into the design include self-assessments (internal audits) by the
verification partner and independent assessments by EPA. The assessments identified in
the planning process are incorporated into the design. The type and minimum number of
assessments are identified in part A section 9.0. A suggested schedule of assessments is
given in part B section 4.0.
2.1.5 Validating, reporting, securing, and archiving data
Data are validated as indicated under Audits of Data Quality in part A section 9.0. Data
are reported in ETV verification reports and ETV verification statements. Data records
are stored as discussed in part A, section 5.0 and in Appendix A.
ETVQMP
Page: 33 of 41
33
Version: 1.0
Date: May 26,1998
-------
2.2 Generic verification protocols and test/QA plans: planning documents from the
design process
Planning documents from the design process include generic verification protocols and test/QA plans.
Writing planning documents is generally a lengthy process involving many iterations of
review and revision. Authors should be knowledgeable of the activity and the equipment
described in the planning documents. Two types of planning documents have been
identified as the core documentation needed for operation of an ETV pilot: the generic
verification protocol and the test/QA plan. The generic verification protocol is meant to
promote uniform testing for a single pilot and, therefore, is considered a more general
document. The test/QA plan contains the specific information needed to conduct a
verification test.
2.2.1 Generic verification protocols provide the necessary framework for development of the
more detailed test/QA plan. The specific content and level of detail given in generic
verification protocols will vary greatly between pilots. For some pilots, the generic
verification protocol may be so detailed that the test/QA plan may require very little
additional information. Conversely, other pilots may use the generic verification protocol
to describe the general procedures that guide the pilot. Given the highly variable nature of
the generic verification protocol, no specific format has been proposed.
The issues that may be addressed in the generic verification protocol are the following:
• General description of the pilot
• Responsibilities of all involved organizations
• Experimental design
• Equipment capabilities and description
• Description and use of field test sites
• Description and use of laboratory test sites
QA/QC
• Data handling
• Requirements for other documents
• Health and safety
• References
The QA/QC section of the generic verification protocol typically describes the activities
which verify the quality and consistency of the work. Preparation and use of appropriate
QA procedures such as QC samples, blanks, split and spiked samples, and performance
evaluation (PE) samples to verify performance of the technology being tested can be
described. Criteria for success can be included. Frequency of calibrations and QC checks
and the rationale for them can be described. Procedures for reporting QC data and results
can be given. Who or what organization is responsible for each QA activity, and who has
the responsibility for identifying and taking corrective action can be specified. However, if
ETVQMP Version:'1.0
Page: 34 of41 34 Date: May 26,1998
-------
these items vary between tests within a given pilot, the more appropriate document in •-
which to describe them may be the test/QA plan.
The protocol may cite documents or procedures that explain, extend, and/or enhance the
protocol such as related procedures, the published literature, or methods manuals. The
specific location of any reference not readily available from a full citation in the reference
section should be given (as in a facility-specific standard operating procedure) or attached
to the protocol.
2.2.2 Test/OA plans contain the following required elements. Not all elements listed are
appropriate to every test. The author of the test/QA plan will note and explain those
elements that are not applicable.
• Title and approval sheet
• Table of contents, distribution list
• Test description, test objectives
• Identification of the critical measurements, data quality objectives, data quality
indicator goals, schedule, milestones
• Test (including QA) organization and responsibilities
• Documentation and records
• Experimental design
• Sampling procedures
• Sampling handling and custody
• Analytical procedures
• Test-specific procedures for assessing data quality indicators
• Instrument calibration and frequency
• Data acquisition and data management procedures
• Internal systems audits
• Internal performance audits (where applicable)
• Corrective action procedures (response actions to audit findings)
• Assessment reports to EPA
• Data reduction, data review, data validation, data reporting
• Reporting of data quality indicators for critical measurements
• Limitations of the data
The generic verification protocol is incorporated by reference. One reference document
available for writing test/QA plans is EPA/QA G-5, Guidance for Quality Assurance
Project Plans.
If another level of detail is required for describing test activities, for example operation of
an instrument, a standard operating procedure may be written and attached to the QA/test
plan. The following topics, from EPA QA/G-6, Guidance for Development of Standard
Operating Procedures (SOPs), may be included (or a reference provided) in the standard
ETVQMP
Page: 35 of 41
35
Version: 1.0
Date: May 26, 1998
-------
operating procedure:
• Scope and applicability
• Summary of procedures
• Definitions (acronyms, abbreviations, etc.)
• Personnel qualifications
• Health and safety warnings (Warn of activities which could result in possible
personal injury.)
• Cautions (Warn of activities which could damage equipment, degrade samples, or
invalidate results.)
• Apparatus and materials
• Calibration
• Sample collection, sample labeling, sample tracking
• Handling and preservation of samples
• Interferences
• Sample preparation and analysis
• Data acquisition, calculations and data reduction
• Requirements for computer hardware and software used in data reduction and
reporting
• Data management and records management
3.0 IMPLEMENTATION OF PLANNED OPERATIONS
3.1 Implementation of planning
Environmental data operations shall be implemented according to the approved planning documents. Deviations shall be
documented and reported to and evaluated by management. Approved changes shall be made and distributed to test personnel
to replace previous versions of the documents.
Technology performance verifications are implemented according to the generic
verification protocols and test/QA plans prepared during planning. During
implementation, changes are incorporated, reviewed and approved according to the
scheme discussed in part A section 5.0. Test personnel have access to the approved
planning documents, approved changes to planning documents, and all referenced
documents. The final protocols are posted on the ETV web page for future use for similar
technology verifications.
All implementation activities are documented. Suitable documents are bound notebooks,
field and laboratory data sheets, spreadsheets, computer records, and output from
instruments (both electronic and hardcopy). All documentation is developed as described
in the planning documents. All implementation activities are traceable to the planning
documents and to test personnel.
ETVQMP Version: 1.0
Page: 36 of 41 36 Date: May 26,1998
-------
3.2 Services and items
Only qualified and accepted services and items shall be used in the performance verification operations. Acceptance shall be
identified on the items themselves and/or in documents traceable to the items. Tools, gauges, instruments, and other
sampling, measuring, and testing equipment used for activities affecting quality shall be controlled as required and, at
specified intervals, calibrated to maintain accuracy with specified limits. Documentation of calibration shall be maintained
and shall be traceable to the equipment. Periodic preventative and corrective maintenance of equipment shall be performed,
and it shall be recalibrated prior to use.
The ETV program services are delivered by the verification partners. The verification
partners are accepted via the request for application, proposal, and assistance agreement
process as discussed in part A section 4.0.
Qualified and accepted services and items used in testing are provided for in the
verification partners' quality systems. The pilot quality management plan contains
provisions for acceptance of services and items, and documentation of acceptance.
Control of equipment, calibration to maintain accuracy within specified limits,
maintenance, and documentation are the responsibility of the verification partner. The
verification partner verifies that the tools, gauges, instruments, and any other sampling,
measuring, and testing equipment used for activities affecting quality are controlled as
required by the planning documents, and calibrated at specified intervals to maintain
accuracy within specified limits. Equipment found to be out-of-specification is not used
without documented repair and reassessment of performance. All maintained and repaired
equipment is recalibrated as necessary before it is used for measurement work.
Oversight is the responsibility of EPA during the pilot period, and is conducted through
review and acceptance of the verification partners' quality system documents, the pilot
quality management plan, and through independent audits.
3.3 Field and laboratory samples
Handling, storage, cleaning, packaging, shipping, and preservation of field and laboratory samples shall be performed
according to required specifications, protocols, or procedures to prevent damage, loss, deterioration, artifacts, or
interference. Sample chain of custody shall be tracked and documented.
If samples for analysis are taken in the field, they are to be handled according to
procedures in the verification partners' quality systems and the pilot quality management
plan. The oversight responsibility of EPA during the pilot phase is to determine that the
approved systems and plans contain adequate procedures for handling, storage, cleaning,
packaging, shipping, and preservation of field and laboratory samples to prevent damage,
loss, deterioration, artifacts, or interferences. The verification partner provides adequate
chain of custody procedures, if they are required.
ETVQMP
Page: 37 of 41
37
Version: 1.0
Date: May 26, 1998
-------
3.4 Data and information management
Data or information management, including transmittal, storage, validation, assessment, processing, and retrieval, shall be
performed in accordance v/ith the approved instructions, methods, and procedures.
ETV program records and the procedures for handling them are listed in part A section
5.0.
4.0 ASSESSMENT AND RESPONSE
4.1 Assessment types
Management system review - Audit of a quality system for conformance to a quality
management plan
Technical systems audit - Qualitative onsite audit of the physical setup of the test. The
auditors determine the compliance of testing personnel with the test/QA plan.
Performance evaluation audit - Quantitative audit in which measurement data are
independently obtained and compared with routinely obtained data to evaluate the
accuracy (bias and precision) of a measurement system.
Audit of data quality - Qualitative and quantitative audit in which data and data handling
are reviewed and data quality and data usability are assessed.
4.2 Assessment frequency
Activities performed during technology verification performance operations that affect the quality of the data shall be assessed
regularly, and the findings reported to management to ensure that the requirements stated in the generic verification protocols
and the test/QA plans are being implemented as prescribed.
The types and minimum frequency of assessments for the ETV program are listed in part
A section 9.0. The pilot tests will have at minimum the following types and numbers of
assessments:
• management systems review - one independent assessment by EPA, as provided in
the pilot quality management plan
• technical systems audits - self-assessments for each test as provided for in the
test/QA plan and independent assessments by EPA, twice per pilot
• performance evaluation audits - self-assessments, as applicable, for each test as
provided in the test/QA and independent assessments, as applicable for each pilot
• audits of data quality - self-assessments of at least 10% of all the verification data;
and independent assessment, as applicable for each pilot
ETVQMP
Page: 38 of 41
38
Version: 1.0
Date: May 26,1998
-------
Additional assessments may be provided for in individual test/QA plans. Assessments by
the verification partner will remain steady throughout the pilot period, but independent
assessment by EPA will decrease in keeping with the policy of preparing the pilots to
eventually operate independently.
4.3 Response to assessment
Appropriate corrective actions shall be taken and their adequacy verified and documented in response to the findings of
the assessments. Data found to have been taken from non-conforming equipment shall be evaluated to determine its
impact on the quality of the data. The impact and the action taken shall be documented.
Assessments are conducted according to procedures contained in the verification partners'
quality systems or the quality procedures available to EPA personnel, as discussed in part
A, section 9.0. Findings are provided in audit reports. Responses to adverse findings are
required within 10 working days of receiving the audit report. Follow-up by the auditors
and documentation of response are required.
5.0 ASSESSMENT AND VERIFICATION OF DATA USABILITY
5.1 Data validation
Data obtained during verification tests shall be assessed, verified, and qualified according to their intended use (as
verification performance data). Any limitations on this intended use shall be expressed (quantitatively to the extent
practicable) and shall be documented in the ETV verification report.
Audits of data quality are used to validate data at the frequency cited in Table 9.1 and are
documented in the data audit report. The goal of an audit of data quality is to determine
the useability of test results for reporting technology performance, as defined during the
design process. Validated data are reported in the ETV verification reports and ETV
verification statement along with any limitations on the data and recommendations for
limitations on data usability.
5.2 Existing data
Any data obtained from sources that did not use a quality system equivalent to the E4 Standard shall be assessed according to
approved and documented procedures.
Existing data may be used for planning, subject to the individual rules set up by each pilot.
Data used for verification collected outside the ETV test is subject to rigorous scrutiny
according to the procedure in Appendix C.
ETVQMP
Page: 39 of 41
39
Version: 1.0
Date: May 26,1998
-------
5.3 Reports reviewed
ETVverification reports containing data and reporting the results of technology verification performance shall be reviewed
independently (i.e., by others than those who produced the data or the reports) to confirm that the data or results are presented
correctly. Tltese reports shall be approved by management prior to release, publication, or distribution.
The procedure for ETV verification report and ETV verification statement review and
approval is given in part A section 5.0. ETV verification reports are peer-reviewed, and
during the pilot phase ETV verification statements are signed by the EPA laboratory
directors.
ETVQMP
Page: 40 of 41
40
Version: 1.0
Date: May 26,1998
-------
REFERENCES
Guidance for the Preparation of Standard Operating Procedures (SOPs)for Quality Related
Documents, EPA/600/R-96/027,Washington DC: U.S. Environmental Protection Agency, 1995.
Guidance for Quality Assurance Project Plans, EPA QA/G-5 Washington DC: U.S.
Environmental Protection Agency, 1998.
Simes, G. F., Preparation Aids for the Development of Category II Quality Assurance Project
Plans, EPA/600/8-91/004. Cincinnati OH: U.S. Environmental Protection Agency, 1991.
Guidance for the Data Quality Objectives Process, EPA QA/G-4, EPA/600/R-96/055.
Washington DC: U.S. Environmental Protection Agency, 1994.
Guidance for Data Quality Assessment, EPA QA/G9, EPA/600/R-96/084. Washington DC: U.S.
Environmental Protection Agency, 1996.
USEPA Office of Research and Development. Environmental Technology Verification Program
Verification Strategy. EPA/600/K-93/003. US Government Printing Office; 1997.
American Society for Quality Control, Energy and Environmental Quality Division,
Environmental Issues Group. AMERICAN NATIONAL STANDARD Specifications and
Guidelines for Quality Systems for Environmental Data Collection and Environmental Technology
Programs. ANSI/ASQC E4-1994. American Society for Quality, 1994.
ENTICE White Paper - 1994
Science Advisory Board Review Report- 1995
ETV Team and Duties Memorandum - 1995
Budget Memoranda - Fiscal Year 1995
Budget Memoranda - Fiscal Year 1996
Budget Memoranda - Fiscal Year 1997
ETVQMP
Page: 41 of 41
41
Version: 1.0
Date: May 26,1998
-------
-------
APPENDIX A
EPA SERIES NO. 003A
U.S. EPA RECORDS CONTROL SCHEDULE
SERIES TITLE: Grants and Other Program Support Agreements
PROGRAM: All Programs, except Superfund Site Specific and Wastewater Construction and
State Revolving Fund Grants
EPA SERIES NO: 003A
NARA SCHEDULE NO. Nl-412-94-2/1
(Use this number to retire records to the FRC)
APPLICABILITY: Agency-wide
IDENTIFYING INFORMA TION:
DESCRIPTION: Includes records that document all types of agreements with other Federal,
State, or local government agencies, universities and other institutions to which EPA is a party,
and which support EPA's environmental programs (other than Superfund site specific and
wastewater construction grants). Specific types of agreements include assistance agreements,
grants, cooperative agreements, Interagency Agreements, and other types of program support
agreements administered by Headquarters or EPA regions and which provide for research,
demonstration projects, training, fellowships, investigation, surveys, studies, or other type's of
program support activities.
Includes:
Supporting documentation - Specific types of records include documentation of significant actions
and decisions, justifications, cost estimates, scopes of work, correspondence, applications,
pre-award reviews, funding decisions, award documentation, commitment notices, transmittal
correspondence, agreements, agreement oversight activities, non-compliance/dispute
documentation, audit records, closeout documentation for completed agreements, and reports and
evaluations resulting from agreements.
Excludes: Final products and deliverables, Superfund site specific grants, and agreements and
wastewater construction grants which are scheduled separately.
ETVQMP
A-l
Version: 1.0
Date: May 26,1998
-------
ARRANGEMENT: Arranged by agreement.
TYPE OF RECORDS:
Case file
MEDIUM:
Microfilm, paper, forms,
electronic
SPECIFIC RESTRICTIONS:
None
VITAL RECORD:
No
FUNCTIONS SUPPORTED:
Program operations
SPECIFIC LEGAL REQUIREMENTS:
Varies according to program
40 CFR 30, 31, 35, Subparts A, H, P, 40, 45-47
DISPOSITION INFORM A TION:
FINAL DISPOSITION:
Disposable
TRANSFER TO FRC PERMITTED:
Yes
FILE BREAK INSTRUCTIONS: Break files immediately after closeout of the agreement.
DISPOSITION INSTRUCTIONS: Keep inactive materials in office at least 1 year after file
break, then retire to FRC. Destroy 7 years after file break.
If record copy is in microform, break file upon completion of microform quality assurance check.
Retire one silver and one diazo copy to the FRC along with finding aids and indexes. Destroy 7
years after file break. Retain one or more sets for office use. Destroy any Agency microform
copies when superseded or no longer needed.
APPLICA TION GUIDANCE:
REASONS FOR DISPOSITION: The retention period for supporting documentation has been
extended because the records are needed in the event of a claim against the Agency. The statute
of limitation on such claims is 7 years. Final products and deliverables are covered in EPA 258A.
AGENCY-WIDE GUIDANCE: Final products and deliverables are permanent records and are
scheduled as EPA 258A.
Agreement closeout is when the Agency determines all administrative actions and required work
is completed (submission of the final expenditure report, SF 269 - Financial Status Report, by the
recipient) or when the agreement is terminated or annulled and any disputes settled. Final
closeout documentation may consist only of an internal Agency memo.
ETVQMP
A-2
Version: 1.0
Date: May 26,1998
-------
The Grants Administrator (also called the Grants Management Officer), Grant Project Officer,
and Financial Management Officer are responsible for the record copies of grant agreement
records and implementing the disposition. Records can include unique program files maintained
by the grant project officer or client or technical representative. All other copies may be
destroyed when no longer needed.
The following offices and managers are responsible for maintaining a complete record set and
dispositioning documents as designated below:
Grants Management Officer (Grants Specialist) - Record copy of applications; reviews and
amendments related to the application; administrative review checklist; certifications; agreements
and any amendments; award documentation; requests for deviations; stop work orders;
documentation relating to termination actions, disputes and appeals, annulments and audits; legal
opinions; financial status reports; and increases and decreases; correspondence and other related
documents.
Program Office (Project Officer) - Record copy of documents used for day-to-day technical
direction of the grant or interagency agreement such as draft and final products and deliverables;
work plans and progress reports; draft documents and comments provided or other records of
technical direction. Copies of applications, awards, amendments and other administrative and
financial documents.
Financial Management Officer - Record copy of reimbursement requests, payment vouchers,
payment files, federal cash transaction reports; copies of financial status report and other related
documents.
See EPA 274A for Unsuccessful Grant Application Files. This item does not include Superfund
site specific grants which are scheduled as EPA 001A or Waste Water Construction and State
Revolving Fund Grants which are covered in EPA 232A. Contracts are covered under EPA
020A, EPA 055A, EPA 202A, and EPA 258A. The Grants Information and Control System is
scheduled as EPA 575A.
PROGRAM OFFICE GUIDANCE/ DESCRIPTIVE INFORMATION: Previous schedule
items combined into this schedule were for the following programs: Federal Activities, Water,
Solid Waste, Emergency and Remedial Response, Toxic Substances, Mobile Source, Air and
Hazardous Waste, Regional Administrator, Research and Development, Pesticides, Radiation, and
Information and Resources Management. Specific item numbers are cited below.
ETVQMP
A-3
Version: LO
Date: May 26,1998
-------
CUSTODIAL INFORMATION:
CONTROLLING UNIT: Multiple units CONTACT POINT:
Name: Name:
Location: Mail Code:
Inclusive Dates: Telephone:
Volume on Hand (Feet): Office:
Annual Accumulation: Room:
(feet or inches)
CONTROL INFORMA TION:
RELATED ITEMS: EPA 001 A, EPA 020A, EPA 055A, EPA 202A, EPA 232A, EPA 258A,
EPA274A.EPA575A
PREVIOUSLY APPROVED BY
NARA SCHEDULE NOS: NC1-412-75-6/1, NCI-412-76-1/III/14 and 20,
NC1-412-76-9/25, NCI-412-77-1/8 and 9, NCI-412-77-4/1, NCI-412-77-5/11,
NCl-412-78-10/6b, NC1-412-82-12/11, NCI-412-85-6/8 and 15,
NC1-412-85-7/8, NC1-412-85-12/6, NC1-412-85-14/7, NCI-412-85-17/2,
NC1-412-85-18/2, NCI-412-85-19/4, NCl-412-85-23/4a, NCl-412-85-25/5a and b, NC1-412-
85-26/1/4, Nl-412-86-1/8, Nl-412-86-3/7
Approval Approval Entry Last
Date EPA Date NARA Date Modified
11/27/96 3/20/91 12/27/96
ETVQMP . . Version: 1.0
A-4 Date: May 26,1998
-------
APPENDIX B
The following measures of success is excerpted from the November 1997 ETV meeting:
What Constitutes Success for ETV?
Timing
• No more than one year for partner selection.
• No more than one year after partner selection for organizational phase (i.e.,
stakeholder selection, technology prioritization, initial protocol development,
stakeholder approval of protocols).
• For each technology test event, no more than one year between vendor meeting
and draft final report.
• No more than two months for EPA approval and one month for publication.
Cost of Operation, Testing, Participation
Customer Satisfaction
• Significant number of States accept ETV data for permitting.
• Significant number of consulting engineers use ETV data for making technology
recommendations.
Effects
Vendor sales data; technology use data.
ETVQMP
B-l
Version: 1.0
Date: May 26,1998
-------
-------
APPENDIX C
ENVIRONMENTAL TECHNOLOGY VERIFICATION PROGRAM
EXISTING DATA: POLICY AND PROCESS
Background
The Environmental Technology Verification program was established by the U.S. EPA for the
purpose of verifying the performance of commercial-ready technologies for their ability to
monitor, prevent, control, or clean-up pollution. Verification is accomplished by the evaluation of
objectively-collected, quality-assured data which are provided to potential purchasers and
permitters as an independent and credible assessment of the performance of a technology. Data
are collected and evaluated in partnership with independent third party verification partners
chosen from the public sector (such as states), the private sector (such as non-profit research
institutions), federal laboratories, and others. During the pilot phase (1994-2000), EPA provides
oversight of the verification partner to assure the credibility of the process and data, and keeps the
authority for the verification process and decision (except in the case of an independent pilot).
After the pilot phase, responsibility and authority revert to the verification partner.
The ETV program seeks to identify optimal methods to verify environmental technologies without
compromising quality. Stakeholder groups, consisting of representatives of major verification
customer groups, advise and assist EPA and the verification partners in this effort. One consistent
and urgent request has been that existing data, i.e., data collected prior to the ETV program, be
used for ETV verification. This suggestion is reinforced by the programs of individual states, as
well as those of other countries, that routinely consider previously-collected data in the
verification of vendor claims for a technology. The purpose of this document is to establish a
guideline whereby the ETV program may use these "historical," "existing," or "secondary" data
to increase and enhance the scope of individual pilot projects.
POLICY
Currently, under the U.S. ETV program, the verification partner and the technology developers
typically plan and execute tests which provide the objective and quality-assured data by which the
environmental technologies are evaluated. Existing data are used to support test plan
development. Measurements and data are collected in a demonstration of the technology by the
developer, under the direction of the verification partner, and overseen by EPA. Reports are
peer-reviewed and verification statements are issued. In this closely-monitored scenario, the
origin and quality of the data upon which the verification statement rests are generally known and
documented, and therefore the possibility for verification decision error is minimized. The
consequences of a serious verification decision error can include verification of fraudulent claims,
litigation, and loss of credibility for the ETV program, the verification partners, and EPA.
ETVQMP
C-l
Version: 1.0
Date: May 26,1998
-------
Compelling arguments exist for considering using certain qualified existing data to replace some
or all of the verification testing for a given technology. Some technologies are time-consuming
and expensive to evaluate. Due to resource constraints, demonstrations can, at best, show the
performance of the technology under only limited conditions. A test may provide only one small
performance snapshot in time as opposed to providing data from several years of performance
collected by the developer or his customers under a full range of conditions. Limited resources
may require that testing focus on only one component of a technology rather than its full range of
capability. Before coming to the commercially viable stage of development, these technologies
may have been tested numerous times with acceptably reproducible results.
Judicial precedent provides argument for the defensible use of existing data. In Daubert v. Merrill
Dow Pharmaceuticals. Inc., the Supreme Court in 1993 adopted a new standard for the
admissibility of scientific evidence. The Court there held that Federal Rule of Evidence 702
requires that, when presented with proposed scientific testimony, the district court must make a
preliminary assessment of whether the reasoning or methodology underlying the testimony is
scientifically valid, and therefore reliable. The Court declined to adopt a definitive checklist or
test, but noted several factors a court should consider. Those factors include: (1) does the theory
or technique involve testable hypotheses; (2) has the theory or technique been subject to peer
review and publication; (3) are there known or potential error rates and are there standards
controlling the technique's operation; and (4) is the method or technique generally accepted in the
scientific community? The court must also consider the relevance or fit of the proposed testimony
by determining if the reasoning and methodology can properly be applied to the facts at issue.
The Clean Air Act Credible Evidence Revisions (see Federal Register, Vol. 62, No. 36, February
24,1997) provide precedent within the Agency for defensible consideration of existing data for
verification use. These revisions clarify that data from methods which are not EPA Standard
Reference Methods can be used in enforcement actions and for compliance certification.
Conversely, emission sources will be able to use any credible evidence (ACE) for contesting
allegations of noncompliance in enforcement actions. As the rule states, it "exemplifies EPA's
'common sense' approach to environmental protection, which encourages smarter, cheaper and
more flexible means of achieving environmental goals without compromising the fundamental
health and environmental protections provided by federal environmental laws." It follows that if
EPA can use ACE for enforcement actions, it can be considered for verification.
Other precedent within the Agency exists at the Office of Air Quality Planning and Standards
(OAQPS). OAQPS uses secondary data, defined as data that are utilized for a purpose other than
that for which they were initially collected, in its regulatory efforts. In order to effectively focus its
quality assurance (QA) efforts within the constraints of available resources, OAQPS concentrates
its consideration of secondary data according to category of project. The QA activities associated
with evaluating secondary data are conducted to assure that the data will be adequate and
sufficient for their planned secondary use.
ETVQMP _ _ Version: 1.0
C-2 Date: May 26,1998
-------
Recognizing therefore that it is neither prudent nor cost-effective to ignore existing data, the ETV
program establishes by this document a consistent process to evaluate these data for the extent of
their credibility and usability in the verification decision. Data to be considered for use to replace
verification testing undergo a rigorous process of evaluation using stringent criteria. The
following guidelines are used to qualify existing data for verification purposes (detailed
procedures follow in the "process" section of this document):
1. Data are evaluated using qualified reviewers following the data evaluation process
established in the "process" section of this document.
2. The documentation of the candidate data is sufficient to allow the reviewers to assess the
quality of the data and its usability for verification.
3. The data are evaluated to determine that they meet the same minimum quality acceptance
criteria as that collected in a comparable ETV pilot demonstration.
4. All of the data used for a verification must have been objectively collected, independently
of the vendor.
5. Only data collected under a well-defined, documented quality system3 will be considered.
Such data sets should contain all the elements required to withstand peer review, and thus
be useful for verification.
Recognizing that useful data exist which will not qualify for verification under these guidelines,
and responding to customer needs, individual pilots may establish individual evaluation criteria by
which existing data may be considered. These data may not be used directly for verification, but
may be used, for example, to support planning or to augment verification testing. No ETV
program-wide guidelines are necessary for the use of existing data for purposes other than for
verification.
PROCESS
Identifying and Qualifying the Data
The vendor proposes the data to be evaluated. EPA and the verification partner shall (with input
from the stakeholder group, as applicable) identify for the vendor the procedures and acceptance
criteria used in the pilot demonstrations to evaluate technology performance. These procedures
and criteria are the same as that used for other technologies evaluated by the verification partner.
The data requirements are developed by EPA, the verification partner, and interested stakeholders
for the pilot, and are not specific to the existing data. The vendor and verification partner
perform the initial evaluation.
1 It is suggested that testing entities having a quality system which is modeled after the American National
Standard Institute/American Society for Quality Control (ANSI/ASQC) Standard E-4-1994, Specifications and
Guidelines for Quality Systems for Environmental Data Collection and Environmental Technology Programs, or
the International Organization for Standardization (ISO) Standard 9000, Quality Management and Quality
Assurance Standards: Guidelines for Selection and Use, may have appropriate quality systems. Other similar
quality systems may be accepted at the discretion of the reviewers.
ETVQMP
C-3
Version: 1.0
Date: May 26,1998
-------
The vendor shall provide the verification partner with the detailed protocols and test plans used to
develop the existing data. The vendor shall identify those data that he believes will meet the
acceptance criteria, qualify those data, and submit the data along with detailed evidence that the
data meet the requirements of the pilot project. The evidence shall be submitted to the EPA and
verification partner in a detailed report. The report shall show how the data verify the
performance of the technology, identify data that were excluded, give an explanation of how and
why they were excluded, and address other requirements specific to the pilot project. The vendor
shall be prepared to provide all of the raw data.
The verification partner shall review the planning documents to determine whether they meet the
requirements of those being used by the verification partner for evaluation tests of other
technologies. At a minimum the existing data protocols and test plans shall require the same level
of QA/QC, replicate tests, data treatment, and reporting as that required by the verification
partner in its technology demonstrations. The verification partner shall conduct a detailed review
of the vendor's data report to determine whether the data adequately evaluate the performance of
the technology. The verification partner has access to the raw data and works through a
reasonable random sample (suggest 10% of the data). A recommended method for evaluation of
data is tracing a random selection of data points from the raw data set to the final report.
Minimum General Acceptance Criteria
• The technology is based on sound scientific and engineering principles.
• The conditions under which the data were collected are clearly defined and were
appropriate for the demonstration of the capabilities of the technology.
• The data are quality assured. For example, where appropriate, the documentation
provides a measure of the bias and precision of the measurements. Where needed,
minimum detection limits have been determined and reported. Where applicable,
the measurement range of the technology is given. A narrative statement will
include a discussion of how well the data represent the capabilities of the
technology in its intended environmental application
• Sufficient data are supplied to allow the technology to be verified. Sufficiency of
the data will be determined by the reviewers.
• Vendor-generated data may be reviewed as part of the evaluation process because
it is a rich source of knowledge about the technology. Only data collected
objectively and independently of the vendor, however, may be used to replace
verification testing.
ETVQMP
C-4
Version: 1.0
Date: May 26,1998
-------
Specific Acceptance Criteria
In addition to the general acceptance criteria, the specific pilot project stakeholders may impose
specific acceptance criteria which must be as stringent as the acceptance criteria for the data
collected during verification testing.
Convening the Data Evaluation Panel
If the verification partner determines that the report does not adequately evaluate the performance
of the technology, the vendor is notified and no further action is required. If the verification
partner determines that the vendor's report does adequately evaluate the performance of the
technology, then a data evaluation panel (DEP) is appointed. The verification partner enlists the
services of 3 qualified reviewers to serve on the DEP. During the pilot phase of the ETV
program, the DEP will generally consist of one person from EPA, one person from the
verification partner, and one person who is an outside expert in the technology being evaluated.
The DEP must contain members who are credible, experienced, knowledgeable, and qualified in
the technical areas critical to the technology being evaluated. The members of the DEP must be
objective and have no real or perceived conflict of interest with the commercial developer of the
technology they are evaluating. DEP members must be independent; they cannot have been
involved in the collection of the data being evaluated.
Evaluation of the Data by the DEP
The DEP reviews and agrees on the acceptance criteria and determines their applicability to the
data to be evaluated. The evaluation shall follow the procedures and criteria developed by the
verification partner and EPA for other technology verifications conducted in the pilot project.
The verification partner provides a written summary of its review to the DEP. When the
verification partner submits the data to the DEP, it ceases to be proprietary. The DEP reviews
and evaluates the data using the applicable acceptance criteria.. The DEP determines that the data
were gathered following appropriate test protocols similar to the protocol used for verification
testing. It ensures that the data were gathered following written test plans developed using a
similar protocol. Planning must have included specific test objectives, experimental design,
criteria for data quality, QA/QC procedures followed and reported, number of samples or
frequency of sampling, and sampling and analytical procedures. The DEP must determine that the
data quality meets or exceeds the minimum data quality requirements of the verification testing
conducted during the pilot.
The quality and usability of the existing data shall be evaluated against clearly defined data quality
requirements based on the data quality requirements of the ETV pilot project. The data shall be
sufficient to evaluate the performance of the technology.
ETVQMP
C-5
Version: 1.0
Date: May 26, 1998
-------
Recommendations for Acceptance of Data for Verification Role
The DEP shall prepare a report on its findings. At a minimum the report must address the
following:
• Were the data collected by following the protocol and test plan provided by the
vendor?
• Do the data meet the minimum QA/QC requirements of the ETV pilot project
demonstrations?
• Do the data adequately evaluate the performance of the technology? Are there
enough data, and are the data of sufficient quality for the verification partner, the
ETV program, and EPA to place their reputations on the line?
The DEP provides a written statement of the performance of the technology as provided by the
data, a statement of how well the data meet the acceptance criteria, and a data acceptance
recommendation.
Revi&v and Acceptance of Recommendation by Verification Partner and EPA
The EPA reviews the report, determines whether to accept the data acceptance recommendation,
and signs the verification statement.
ETVQMP
C-6
Version: 1.0
Date: May 26,1998
•&U.S. GOVERNMENT PRINTING OFFICE: 1998 -*50-0fll/80199
------- |