U.S. Environmental Protection Agency
Environmental Technology Verification Program
Advanced Monitoring Systems Center
and
The Canadian Environmental Technology
Verification
Program
Joint Test/QA Plan for Verification
of Airborne Leak Detection Systems
-------
JOINT TEST/QA PLAN
for
Verification of
Airborne Leak Detection Systems
Version 1.0
September 30, 2010
Prepared by
Battelle
505 King Avenue
Columbus, OH 43201-2693
and
ETV Canada
2070 Hadwen Road
Mississauga, ON L5K 2C9
-------
ACKNOWLEDGEMENTS
This joint verification test/quality assurance plan was developed through a collaboration between
the Environmental Technology Verification (ETV) programs of the United States and Canada
and is intended for use by both Canada and the United States either individually or together in
jointly verifying technologies which will be recognized by both countries. This joint plan can
also be used by a vendor who would like to receive verification from either or both countries.
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 1 of 45
Version 1.0
September 30, 2010
VENDOR APPROVAL PAGE
U.S. EPA ETV Advanced Monitoring Systems Center
and
ETV Canada
Test/QA Plan for Verification of
Airborne Leak Detection Systems
Version 1.0
September 30, 2010
APPROVAL:
Name
Company
Date
Disclaimer
The U.S. Environmental Protection Agency, through its Office of Research and Development,
partially funded, managed, and collaborated in the research described herein. It has been
subjected to the Agency's peer and administrative review. Any opinions expressed in this report
are those of the author (s) and do not necessarily reflect the views of the Agency, therefore, no
official endorsement should be inferred. Any mention of trade names or commercial products
does not constitute endorsement or recommendation for use.
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 2 of 45
Version 1.0
September 30, 2010
TABLE OF CONTENTS
Section Page
Vendor Approval Page 1
Table of Contents 2
Distribution List 5
Abbreviations/Acronyms 6
PROJECT MANAGEMENT 7
Al Verification Test Organization 7
Al.l Battelle 8
A1.2 ETV Canada 10
A1.3 Vendors 11
A1.4 EPA 11
A1.5 Environment Canada 13
A1.6 RMOTC 13
A1.7 Technical Panel 13
A2 Background 14
A2.1 Technology Need 14
A2.2 Technology Category Description 14
A3 Verification Test Description and Schedule 15
A3.1 Verification Test Description 16
A3.2 Proposed Testing Schedule 17
A3.3 Test Facilities 17
A4 Quality Objectives and Criteria for Measurement Data 18
A5 Special Training/Certification 19
A6 Documentation and Records 21
MEASUREMENT AND DATA ACQUISITION 22
Bl Experimental Design 22
Bl.l Field Testing Procedures 22
B 1.1.1 Leak location mapping capabilities 25
Bl.l.2 Leak rate determination 25
Bl.l.3 Pipeline inspection rate and capabilities 26
Bl.l.4 Speciation (ethane/methane discrimination) 26
Bl.l.5 Data processing time between runs 26
Bl.l.6 Data Completeness 27
Bl.l.7 Operational Factors 27
B1.2 Reporting 27
B2 Sampling Method Requirements 28
B3 Sample Handling and Custody Requirements 28
B4 Analytical Method Requirements 28
B4.1 FID Hydrocarbon Measurements 28
B5 Quality Control Requirements 29
B6 Instrument/Equipment Testing, Inspection, and Maintenance 29
B7 Instrument Calibration and Frequency 30
B7.1 FID Analyzer 30
B7.2 Meteorological Sensors 30
B7.3 Flow Meters 31
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 3 of 45
Version 1.0
September 30, 2010
B7.4 Aircraft Sensors 31
B8 Inspection/Acceptance of Supplies and Consumables 32
B9 Non-Direct Measurements 32
BIO Data Management 32
ASSESSMENT AND OVERSIGHT 35
Cl Assessments and Response Actions 35
Cl.l Performance Evaluation Audit 35
C1.2 Technical Systems Audits 37
C1.3 Data Quality Audits 37
C1.4 QA/QC Reporting 38
C2 Reports to Management 38
DATA VERIFICATION AND USABILITY 40
Dl Data Review and Verification Requirements 40
D2 Verification Methods 40
D3 Reconciliation with User Requirements 41
REFERENCES 43
APPENDIX A 44
-------
Section
Airborne Leak Detection Systems
Test/QA Plan
Page 4 of 45
Version 1.0
September 30, 2010
Page
FIGURES
Figure 1. Organizational Chart 8
Figure 2. Diagram of Sampling Near Intentional Leak 24
TABLES
Table 1. Planned Verification Test Schedule 17
Table 2. Average and Record Seasonal Weather Conditions in Casper, Wyoming 18
Table 3. DQIs and Criteria for Critical Measurements for Reference Methods 20
Table 4. Quality Control Activities 29
Table 5. Resolution and Calibration Acceptance Criteria for Meteorological Sensors 31
Table 6. Summary of Data Recording Process 34
Table 7. Methods and Acceptance Criteria for PEA Measurements 36
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 5 of 45
Version 1.0
September 30, 2010
DISTRIBUTION LIST
U.S. EPA
John McKernan, ETV Advanced Monitoring Systems (AMS) Center Project Officer
Michelle Henderson, EPA ETV Quality Assurance Manager
Environment Canada
Raymond Klicius, Head, Technology Programs, Science and Technology Branch
Benoit Desforges, Project Engineer, Science and Technology Branch
Battelle
Amy Dindal, AMS Center Manager
Kenneth Cowen, Verification Test Coordinator
Thomas Kelly, Verification Testing Leader
Rosanna Buhl, AMS Center Quality Manager
Zachary Willenberg, Battelle Quality Assurance Officer
ETV Canada
Mona El Hallak, Director, Technology Assessment and Quality Assurance Services
John Neate, Senior Associate
Technical Panel
Eben Thoma, U.S. EPA, Office of Research and Development
Phil Galvin, New York State Department of Conservation
Donald Stedman, University of Denver
Shannon Phillips, TransCanada PipeLines Limited
Mark Kuhl, Environment Canada, Oil, Gas and Alternative Energy Division
Roy Me Arthur, Environment Canada, Inventory Improvements, Projections, & Sectoral
Engagement
Collaborators/Subcontractors
Vicki Stamp, Rocky Mountain Oilfield Testing Center
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 6 of 45
Version 1.0
September 30, 2010
ABBREVIATIONS/ACRONYMS
ADQ audit of data quality
AMS Advanced Monitoring Center
DQI data quality indicator
DQO data quality objective
EPA U.S. Environmental Protection Agency
ETV Environmental Technology Verification
FID flame ionization detection
GFCR Gas-Filter Correlation Radiometry
GPS global positioning system
GVP General Verification Protocol
LEL lower explosive limit
LRB laboratory record book
m meter
NIST National Institute of Standards and Technology
nm nanometer
PEA performance evaluation audit
QA quality assurance
QAM Quality Assurance Manager
QAO Quality Assurance Officer
QC quality control
QMP Quality Management Plan
RMOTC Rocky Mountain Oilfield Testing Center
scfh standard cubic feet per hour
Tg terragrams
TQAP test/quality assurance plan
TSA Technical systems audit
VTC Verification Test Coordinator
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 7 of 45
Version 1.0
September 30, 2010
SECTION A
PROJECT MANAGEMENT
Al VERIFICATION TEST ORGANIZATION
This test/quality assurance plan (TQAP) describes specific procedures for the
performance of a joint verification test of airborne leak detection technologies for the inspection
of natural gas pipelines. The verification test described in this document will be conducted under
the United States Environmental Protection Agency (EPA) Environmental Technology
Verification (ETV) Program and the Canadian ETV Program. The verification test will be
coordinated and directed jointly by Battelle and ETV Canada. The intent is for the results of this
test to be recognized by both the U.S. and Canadian ETV programs. This TQAP reflects the
requirements of a U.S. EPA Quality category III project.
Testing will be performed at the Rocky Mountain Oilfield Testing Center (RMOTC) and
involve a field testing campaign under potentially challenging measurement conditions. As
described in this TQAP, testing will involve manned flights over a predefined virtual pipeline
that includes a number of intentional natural gas leaks that will serve as targets for the leak
detection technologies. Ground-based measurements will confirm the rates and geospatial
coordinates of the leaks as well as provide estimates of airborne concentrations of natural gas
(ethane/methane) in the vicinity of the leaks. Meteorological measurements will be recorded to
establish conditions during testing.
The ETV Advanced Monitoring Systems (AMS) Center will facilitate the necessary
reviews and audits, coordinate stakeholders and quality assurance (QA) personnel, develop the
TQAP, lead the testing activities, and prepare the verification reports and statements. This role
will provide the necessary oversight for the U.S. ETV program to ensure a quality process of
evaluation, data collection, and reporting for this project. ETV Canada will facilitate the
necessary activities by Environment Canada program management, stakeholders, and QA
personnel including reviews of the TQAP and the verification reports and statements, and audit
of the testing activities. This will provide the necessary oversight for the Canadian ETV
program to ensure a quality process of evaluation, data collection, and reporting. QA oversight
will be provided jointly by the Battelle AMS Center Quality Manager and the ETV Canada
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 8 of 45
Version 1.0
September 30, 2010
Quality Assurance Manager (QAM). Additional QA oversight will be provided by the EPA
AMS Center Quality Manager at her discretion.
The organization chart in Figure 1 identifies the responsibilities of the organizations
associated with the verification test. Roles and responsibilities are defined further below.
United States Environmental
Protection Agency, Environmental
Technology Verification Program
(U.S. EPAETV)
Battelle and U.S. EPA ETV
Advanced Monitoring
Systems (AMS) Center
Technology Vendor
U.S. ETV AMS Center
Air Stakeholder
Committee
Rocky Mountain Oilfield
Testing Center Testing Staff
(RMOTC)
Figure 1. Organizational Chart
Environment Canada
Ontario Centre for
Environmental Technology
Advancement (OCETA),
ETV CANADA
Al.l Battelle
Dr. Kenneth Cowen is the AMS Center Verification Test Coordinator (VTC) for this test.
In this role, Dr. Cowen will have overall responsibility for ensuring that the technical, schedule,
and cost goals established for the verification test are met. Specifically, he will:
> Serve as the primary point of contact for vendor representatives and collaborators;
> Hold a kick-off meeting approximately one week prior to the start of the verification test
to review the critical logistical, technical, and administrative aspects of the verification
test;
> Be responsible, or assign responsibility, for each aspect of the verification test;
> Direct the team (Battelle testing staff and vendors) in performing the verification test in
accordance with this TQAP;
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 9 of 45
Version 1.0
September 30, 2010
> Ensure that all quality procedures specified in the TQAP and in the AMS Center Quality
Management Plan1 (QMP) are followed;
> Maintain real-time communication with the Battelle AMS Center Manager and EPA
AMS Center Project Officer and QAM on any potential or actual deviations from the
TQAP;
> Coordinate with testing staff to ensure that the planned testing will not interfere with the
test site operations;
> Provide test data, including data from the first day of testing, to the Battelle AMS Center
Manager and EPA AMS Center Project Officer and QAM;
> Prepare the draft and final TQAP, verification report(s), and verification statement(s);
> Conduct a technical review of all test data. Designate an appropriate Battelle technical
staff member to review data generated by the VTC;
> Revise the draft TQAP, verification report(s), and verification statement(s) in response to
reviewers' comments;
> Respond to any issues raised in assessment reports and audits, including instituting
corrective action as necessary;
> Serve as the primary point of contact for vendor representatives and collaborators;
> Coordinate distribution of the final TQAP, verification report(s), and statement(s); and
> Establish a budget for the verification test and manage staff to ensure the budget is not
exceeded.
Ms. Amy Dindal is Battelle's manager for the AMS Center. Ms. Dindal will:
> Review the draft and final TQAP;
> Review the draft and final verification report(s) and verification statement(s);
> Ensure that necessary Battelle resources, including staff and facilities, are committed to
the verification test;
> Ensure that confidentiality of sensitive vendor information is maintained;
> Maintain communication with EPA's AMS Center Project Officer and Quality Manager;
and
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 10 of 45
Version 1.0
September 30, 2010
> Facilitate a stop-work order if Battelle or EPA QA staff discover adverse findings that
will compromise data quality or test results.
Ms. Rosanna Buhl is Battelle's Quality Manager for the AMS Center. Ms. Buhl will:
> Review the draft and final TQAP;
> Assign a Quality Assurance Officer (QAO) for this verification test;
> Delegate to other Battelle quality staff any QAO responsibilities assigned below as
needed to meet project schedules;
> Review any audit checklists prepared by the QAO for completeness and detail;
> Review draft and final audit reports prior to release to the VTC and/or EPA for clarity
and appropriate assessment of findings;
> Review audit responses for appropriateness;
> Review and approve TQAPs and deviations;
> Review draft and final verification report(s) and verification statement(s);
> Maintain real-time communication with the QAO on QA activities, audit results, and
concerns;
> Work with the QAO, VTC, and Battelle's AMS Center Manager to resolve data quality
concerns and disputes; and
> Recommend a stop-work order if audits indicate that data quality or safety is being
compromised.
A1.2 ETV Canada
During this joint verification test, John Neate of ETV Canada will:
> Provide a person from ETV Canada to be the point of contact for Battelle and the
vendors;
> Provide input, review, and comment on the TQAP, Verification Report, Verification
Statement, and other documents pertaining to verification of the vendor technologies;
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 11 of 45
Version 1.0
September 30, 2010
> Coordinate review of the TQAP, Verification Report, Verification Statement, and other
documents pertaining to verification of the vendor technologies with Environment
Canada and Canadian expert reviewers as needed;
> Participate in a (virtual) joint kick-off meeting, led by Battelle prior to test initiation;
> Inform Environment Canada of the final TQAP, Verification Report, including the
Verification Statement; and
> Comply with all quality procedures and program requirements specified in the TQAP,
and ETV Canada General Verification Protocol (GVP)2, as follows:
o Review the TQAP;
o Review test data;
o Provide summary of review; and
o Conduct on-site field audit.
A1.3 Vendors
The responsibilities of the airborne leak detection system vendors are as follows:
> Review and provide comments on the draft TQAP;
> Approve the final TQAP prior to test initiation;
> Provide a complete leak detection system for evaluation during the verification test;
> Provide all other equipment/supplies/reagents/consumables needed to operate their
leak detection system for the duration of the verification test;
> Supply a representative to operate and maintain their technology, including the
airborne platform, during the verification test;
> Provide the data from the vendor's leak detection system to the Battelle field testing
staff at the end of each day of testing; and
> Review and provide comments on the draft Verification Report and Statement for
their leak detection system.
A1.4 EPA
EPA's responsibilities in the AMS Center are based on the requirements stated in the U.S. ETV
Program QMP.3 The roles of specific EPA staff are as follows:
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 12 of 45
Version 1.0
September 30, 2010
Michelle Henderson is the EPA AMS Center QAM. For the verification test, Ms. Henderson
will:
> Review the draft and approve the final TQAP;
> Attend the verification kickoff meeting, as available;
> Review checklists, reports, report responses, and closure statements of the technical
systems audit (ISA), performance evaluation audit (PEA), and audits of data quality
(ADQs) conducted by Battelle and/or ETV Canada;
> Perform at her option an external ISA of testing activities, and/or an ADQ during the
verification test;
> Notify the EPA AMS Center Project Officer of the need for a stop-work order if
evidence indicates that data quality is being compromised;
> Prepare and distribute an assessment report summarizing results of the external audit
performed;
> Review the first day of data from the verification test and provide immediate
comments if concerns are identified; and
> Review the draft and approve the final Verification Reports and Verification
Statements.
John McKernan is EPA's Project Officer for the AMS Center. Dr. McKernan will:
> Review the draft TQAP;
> Approve the final TQAP;
> Attend the verification kickoff meeting, as available;
> Be available during the verification test to authorize any TQAP deviations by phone
and provide the name of a delegate to the Battelle AMS Center Manager should he
not be available during the testing period. Review the first day of data from the
verification test and provide immediate comments if concerns are identified;
> Review the draft verification reports and verification statements;
> Oversee the EPA review process for the TQAP, verification reports, and verification
statements;
> Coordinate the submission of Verification Reports and Verification Statements for
final EPA approval; and
> Post the TQAP, Verification Reports, and Verification Statements on the ETV Web
site.
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 13 of 45
Version 1.0
September 30, 2010
A1.5 Environment Canada
The Canadian ETV program is delivered by ETV Canada under a license agreement from
Environment Canada. ETV Canada will consult with Environment Canada as required.
A1.6 RMOTC
RMOTC will provide field testing sites for this verification and will coordinate access to
these sites by testing personnel, vendor representatives, and subcontractors. Specifically,
RMOTC staff will:
> Ensure the Battelle testing staff, the vendors, and aircraft operators have appropriate
access to the test site prior to and during the testing period;
> Coordinate the installation of the necessary equipment to generate intentional leaks at
the host facility;
> Communicate needs for safety and other training of staff working at the test site;
> Provide a secure location for safe-keeping of equipment when not being used for
testing;
> Provide information regarding test site features for the Verification Reports;
> Review the draft TQAP; and
> Review the draft Verification Report and Statement.
A1.7 Technical Panel
The following experts provided input to this TQAP through participation in technical panel
discussions:
• Eben Thoma, EPA, Office of Research and Development;
• Phil Galvin, New York State Department of Conservation;
• Donald Stedman, University of Denver;
• Shannon Phillips, TransCanada PipeLines Limited;
• Mark Kuhl, Environment Canada; and
• Roy McArthur, Environment Canada.
Mr. Galvin and Ms. Phillips also provided peer review of this TQAP.
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 14 of 45
Version 1.0
September 30, 2010
A2 BACKGROUND
The purpose of verification programs such as the EPA and Canadian ETV programs is to
provide objective and quality-assured performance data on environmental technologies, so that
users, developers, regulators, and consultants can make informed decisions about purchasing and
applying these technologies. Stakeholder committees of buyers and users of such technologies
recommend technology categories, and technologies within those categories, as priorities for
testing. Among the technology categories recommended for testing are airborne leak detection
systems for the identification of natural gas leaks in gas processing plants, distribution lines, and
transmission lines.
A2.1 Technology Need
Fugitive emissions from natural gas systems have been estimated to account for
approximately 18% of all methane emissions in the U.S. in 2007, and have a global warming
potential equivalent to approximately 105 terragrams (Tg = 106metric tons) of carbon dioxide
(Tg carbon dioxide equivalent, see http://www.methanetomarkets.org/documents/oil-
gas_fs_eng.pdf). These emissions are the result of leaking equipment, system upsets, and
deliberate flaring and venting at production fields, processing facilities, transmission lines,
storage facilities, and gas distribution lines. Identifying and mitigating the sources of these
fugitive emissions will benefit not only the companies who own and operate the natural gas
systems but will also lessen the environmental burden introduced by greenhouse gases in the
atmosphere. However, while some sources of fugitive emissions are readily detected, those
along gas pipelines are generally more difficult to detect because of the extended lengths of the
pipelines, their frequent remote location, and the variable terrain and vegetative cover where the
pipelines are located.
A.2.2 Technology Category Description
A variety of technologies are included in the category of airborne leak detection systems.
At present, this TQAP has been developed for the verification of systems that employ a remote
sensing technique known as Gas-Filter Correlation Radiometry (GFCR). However, this TQAP
can be used for the verification of any airborne leak detection technology whether GFCR or
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 15 of 45
Version 1.0
September 30, 2010
another technique is employed. GFCR is a remote sensing technique used to measure the
amount of a gas of interest in the section of an atmosphere located within the field of view of an
instrument that employs this technique. As the name suggests, a sample of the target gas fills a
container, referred to as a correlation cell, located within the path of the incoming
electromagnetic radiation in the form of light. Light wavelengths are in the range of 10
nanometers (nm) for ultraviolet light to 300,000 nm for infrared light. This range includes
visible light, which is between 400 and 700 nm. Light incident upon the cell first passes through
a narrow band-pass filter that allows only a narrow range of infrared light frequencies through.
The range allowed through the filter is centered on the absorption lines of the target gas in the
correlation cell. The filtered light is then split into two beams, one of which is passed through
the correlation cell, and the other is passed through a "null" cell containing a radiatively inert gas
(i.e., transparent to the incoming filtered light). As the light passes through the two cells, the gas
in the correlation cell absorbs some of the light, thereby reducing the amount of light exiting the
cell. On the other hand, the "null" cell has no effect on the light passing through it. The
radiation that exits both cells is then measured by separate detectors.
By modulating and demodulating the intensity of the radiation being absorbed, the signal
associated with the absorption of target gas can be isolated from the background radiation which
is not modulated. This modulation can be done by manipulating the density of the gas sample in
the correlation cell, thus changing the width and strength of the absorption line and, implicitly,
the transmittance of the cell. The difference between the detector signals from the correlation
and null cells allows determination of light intensity modulation, allowing the technology to
determine the intensity of the light being absorbed by the correlation cell. The presence and
concentration of different gasses in the atmosphere can be determined by this modulation
technique, simply by filling the correlation cell with the gas of interest.
A3 VERIFICATION TEST DESCRIPTION AND SCHEDULE
The purpose of this verification test is to generate performance data on airborne leak
detection technologies. The data generated from this verification test are intended to provide
organizations and users interested in these technologies with information on their potential utility
and operational performance.
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 16 of 45
Version 1.0
September 30, 2010
A3.1 Verification Test Description
The verification test will include one field testing campaign conducted under potentially
challenging conditions for the technologies being verified. Testing will focus on the evaluation
of the following performance parameters:
> Leak location mapping capabilities;
> Leak rate determination;
> Pipeline inspection rate;
> Speciation of ethane/methane; and
> Data processing time between runs.
For this test, multiple intentional gas leaks will be established along a predetermined virtual
pipeline at the RMOTC testing site. The gas leaks will include both aboveground releases as
well as buried releases, and will involve a range of pre-determined leak rates. The leaks will be
established in variable terrain conditions to challenge the technologies over a variety of test
conditions. Testing will involve the performance of multiple flight passes per day over the
pipeline sections during a one week testing period. Testing will be conducted during different
times of the day to assess performance under different illumination and meteorological
conditions. Furthermore, the locations and the leak rates for the intentional leaks will be varied
from run to run to provide differing challenge conditions. Ground-based measurements will be
made to confirm the rate and location of each leak, as well as airborne natural gas concentrations
in the vicinity of the leaks and meteorological conditions.
Subsequent to the verification test, a joint Verification Report including a joint
Verification Statement will be drafted for each technology. This report will be reviewed by the
vendor and by peer reviewers, after which they will be revised, and submitted to EPA and
Environment Canada for final approval. In performing the verification test, Battelle will follow
the technical and QA procedures specified in this TQAP and will comply with the data quality
requirements in the AMS Center QMP1 and the Canadian ETV Program GVP.
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 17 of 45
Version 1.0
September 30, 2010
A3.2 Proposed Testing Schedule
Table 1 shows the planned schedule of testing and data analysis/reporting activities to be
conducted for this verification. The field campaign will be conducted over a period of one week.
Exact dates for testing will be established based on availability of testing staff and the vendor
representatives, and the on-site conditions that meet the criteria of this TQAP.
Table 1. Planned Verification Test Schedule
Time line (In Series)
1 week
5 weeks
3 weeks
4 weeks
Activity
Field testing campaign
Prepare draft report
Conduct peer review
Finalize report
Data Analysis and Reporting
Prepare report template
Review and summarize field testing staff
observations
Compile data from leak detection systems
Compile reference method results
Begin draft report
Perform data analysis
Conduct Technical Systems Audit
Conduct audit of Data Quality
Complete draft report
Complete peer review of draft report
Revise draft report
Submit final report
A3.3 Test Facilities
The RMOTC field site (www.rmotc.doe.gov) is a U.S. Department of Energy field test
site located 35 miles north of Casper, Wyoming, within the Naval Petroleum Reserve No. 3 in
the Teapot Dome Oilfield. The RMOTC field site provides a test site for energy-related
technologies and techniques for the federal government, private sector producers, service
companies, equipment manufacturers, and research organizations. The RMOTC field site is a
10,000-acre operating oil and gas field with approximately 1,200 well bores and approximately
600 producing wells. Produced natural gas is currently processed, compressed, and reinjected
via the RMOTC gas plant. The produced natural gas will be used as the gas source for many of
the leak sites. Gas cylinders placed on the ground but hidden from view will be used for the
leaks of single component gases (e.g., ethane or methane).
The climate and terrain at the RMOTC field site can be characterized as a high desert
plain with an elevation of approximately 5,200 feet above sea level. Table 2 summarizes the
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 18 of 45
Version 1.0
September 30, 2010
average high and low temperatures, record temperatures, and average precipitation for Casper,
Wyoming in the spring, summer, and fall.
Table 2. Average and Record Seasonal Weather Conditions in Casper, Wyoming
Seasonal Data
Average High Temperature
Average Low Temperature
Record High Temperature
Record Low Temperature
Average Precipitation
Spring
56 °F
29 °F
84 °F
-6°F
1.5 inches
Summer
87 °F
53 °F
104 °F
30 °F
1.3 inches
Fall
73 °F
42 °F
97 °F
16 °F
0.98 inches
A4 QUALITY OBJECTIVES AND CRITERIA FOR MEASUREMENT DATA
The objective of this verification test is to evaluate the performance of the airborne leak
detection technologies under potentially challenging operating conditions. This evaluation will
involve a single field testing campaign where technologies being tested will be flown over
predefined virtual pipelines that include a series of intentional natural gas releases to simulate
leaks in the pipelines. The performance of the technologies being tested will be evaluated on
their capabilities with respect to the performance parameters listed in Section A3.1.
Additionally, the verification test will rely upon operator or Battelle testing staff observations (if
Battelle staff are allowed to accompany the operator during test flights) to assess other
performance characteristics of the airborne leak detection technologies, including data
completeness, ease of use, and maintenance requirements.
To ensure that this verification test provides suitable data for a robust evaluation of the
performance of the airborne leak detection technologies being verified, data quality objectives
(DQOs) have been established. The DQOs indicate the minimum quality of data required to
meet the objectives of the verification test and were established to assess the performance of the
airborne leak detection technologies relative to the stated performance parameters. In order to
provide a suitable benchmark for comparison, the intentional leaks introduced to the virtual
pipelines must be well characterized. The DQOs for this verification test include specific
objectives for the accuracy of the leak rates, positioning of the leaks, meteorological
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 19 of 45
Version 1.0
September 30, 2010
measurements, and "ground-truth" concentration measurements. The DQOs are quantitatively
defined in Table 3 in terms of specific data quality indicators (DQIs), along with their acceptance
criteria.
The quality of the data used for evaluation of the airborne leak detection system will be
assured by adherence to these DQI criteria and the QA/quality control (QC) activities, which are
discussed in detail in Sections B2-B7 of this TQAP. Calibration equipment and compressed gas
standards used during this verification test will meet National Institute of Standards and
Technology (NIST) traceability, when available.
The verification test relies in part on observations of the Battelle and ETV Canada field
testing staff for assessment of the performance of the systems being tested. The requirements for
these observations are described in Section B1.1.7. The discussion of documentation
requirements and data review is found in Section BIO, and Section D contains a discussion of
and data verification requirements for this verification test.
Battelle's Quality Manager or designee will perform a TSA once during this verification
test to review these QA/QC requirements. The ETV Quality Managers of the respective
programs (U.S. and Canada) also may conduct an independent TSA at their discretion.
A5 SPECIAL TRAINING/CERTIFICATION
Documentation of training related to technology testing, field testing, data analysis, and
reporting should be maintained for all technical staff participating in verification testing. The
Battelle Quality Manager may verify the presence of appropriate training records prior to the
start of testing. Battelle technical staff and QA staff supporting this verification test will have a
minimum of a bachelor's degree in science/engineering.
The verification test described in this TQAP will be performed at RMOTC. All
participants in this verification test will adhere to the health and safety requirements of the
RMOTC facility, and Battelle's standard environmental health and safety practices.
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 20 of 45
Version 1.0
September 30, 2010
Table 3. DQIs and Criteria for Critical Measurements for Reference Methods
DQI
Gas composition
Leak rate
accuracy
Leak rate
consistency
Leak position
accuracy
Flame lonization
Detector (FID)
or LEL
measurement
accuracy
Wind
speed/direction
accuracy
Performance
Parameter(s)
Leak rate
determination;
Speciation of
ethane/methane
Leak location mapping
capabilities;
Leak rate determination
Leak location mapping
capabilities;
Leak rate determination
Leak location mapping
capabilities;
Pipeline inspection rate
Leak location mapping
capabilities;
Leak rate determination
Leak location mapping
capabilities;
Leak rate determination
Method of
Assessment
Measurement of
collected gas sample
by GC analysis per
ASTMD1945-
03(2010)
Comparison to
independent flow
transfer standard
Measurement before
and after each test run
Comparison to
independent GPS
coordinates
Calibration with
compressed gas
standards
Comparison to
independent sensor(s)
Responsible
Party
Commercial
analytical
laboratory
Battelle
Battelle
Battelle
Battelle
Battelle
Frequency
Once within 1
week prior to
testing
campaign
At least once
for each leak
location
Each leak
When hardware
for leak
generation is
installed
At start and end
of each test day
At least once
during periods
1-3 of each day
of testing
campaign
Acceptance
Criteria
NA
±10% of
target flow
rate
±10% of
target flow
rate
±2m
agreement
±10% of
calculated
concentration
± 10%
Corrective Action
NA
Adjust leak rate; replace
needle valve/orifice as
needed.
Investigate discrepancy.
Replace leak source for
future tests. Flag data.
Investigate discrepancy if
possible. Repeat
measurements. Replace
measurement device(s) if
necessary.
Investigate discrepancy if
possible. Repeat
measurement and
recalibrate FID/LEL if
necessary.
Investigate discrepancy if
possible. Repeat
measurement and
recalibrate or replace sensor
if necessary.
NA - Not Applicable
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 21 of 44
Version 1.0
September 30, 2010
During the verification test, vendor staff will operate their own technologies. The vendors will
be responsible for securing the services of a qualified pilot to conduct the test flights during
verification of their technologies.
A6 DOCUMENTATION AND RECORDS
The records and documents generated for this verification test include the TQAP, chain-
of-custody forms, laboratory record books (LRB), data collection forms, electronic files (both
raw data and spreadsheets), video and photographic records, final verification reports, and
verification statements. The documentation and results of the leak rate measurements, leak
positions, meteorological data, and "ground-truth" concentration measurements will be compiled
and provided to the Battelle VTC. Copies of all of these records will be maintained by the
Battelle VTC during the test, shared with EPA and ETV Canada at the end of each testing
period, and then transferred to permanent storage within 2 months of the final verification reports
being issued. All Battelle LRBs are stored indefinitely by Battelle's Record Management Office.
EPA will be notified before disposal of any files. Section BIO further details the data recording
practices and responsibilities.
All data generated during the conduct of this project will be recorded directly, promptly,
and legibly in ink. All data entries will be dated on the date of entry and signed or initialed by
the person entering the data. Any changes in entries will be made so as not to obscure the
original entry, will be dated and signed or initialed at the time of the change and shall indicate
the reason for the change. Project-specific data forms will be developed prior to testing to ensure
that all critical information is documented in real time. The draft forms will be provided to the
Battelle QAM for review.
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 22 of 45
Version 1.0
September 30, 2010
SECTION B
MEASUREMENT AND DATA ACQUISITION
Bl EXPERIMENTAL DESIGN
The verification test described in this joint TQAP will address verification of airborne
leak detection technologies. The verification test has been designed to include a single field
testing campaign under potentially challenging conditions. Testing will focus on the evaluation
of the following performance parameters:
> Leak location mapping capabilities;
> Leak rate determination;
> Pipeline inspection rate;
> Speciation (ethane/methane); and
> Data processing time between runs.
To the extent possible, the testing procedures performed during the field campaign will be
comparable. Detailed descriptions of the testing procedures are given below.
Bl.l Field Testing Procedures
During the field testing campaign, a series of intentional leaks of compressed natural gas
will be generated by RMOTC staff to simulate leaks along a "virtual" pipeline. The virtual
pipeline will be laid out by RMOTC staff to simulate an actual natural gas pipeline and the
vendors participating in the verification test will be provided with global positioning system
(GPS) coordinates for the pipeline, but not of the intentional leaks. Infrastructure will be
installed to allow for a number of potential leak sources along the pipeline, and the potential
leaks will include both aboveground and buried leaks. Flow restrictions (e.g., critical orifice or
pressure gauge) will be used to allow the leak sources to be set to a predetermined flow rate. The
potential leaks along the pipeline will vary in leak rate intensity over a range of flow rates from
50 to 2,000 standard cubic feet per hour (scfh). During each testing day, a selected number of
the potential leak sources will be used, and the flow rates from the individual potential sources
may be varied from day to day. The leaks will be initiated by RMOTC at the direction of
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 23 of 45
Version 1.0
September 30, 2010
Battelle technical staff and on each day, the position and leak rate of one leak during each test
run will be disclosed to the vendors and serve as a calibration point for the systems being
verified each day. Other than the calibration leak, the positions and leak rates of the other
intentional leaks will not be disclosed to vendor representatives.
During each day of testing, the flow rates at each of the intentional leaks will be
measured by Battelle staff before and after the testing period using one or more dry gas meters or
rotameters with an accuracy of 10% of target flow rate to ensure consistent leak rates. If the
flow rates for more than one of the leaks vary by more than 10%, an additional test flight will be
performed if possible, otherwise the drift in flow rate will be noted, and the lower of the two
flow rates will be used for the technology evaluation. Additionally, after sufficiently low gas
concentrations are confirmed by RMOTC safety personnel, concentration measurements will be
made by Battelle staff using a flame ionization detector (FID) hydrocarbon monitor or lower
explosive limit (LEL) sensor. These measurements will only be made when airborne
concentrations are below 10% of the LEL. Because of the flammable nature of the gases used in
this verification test, appropriate safety procedures will be implemented to avoid potentially
hazardous situations. All testing staff, vendor representatives, and test observers will adhere to
Battelle environmental health and safety, as well as all RMOTC site specific safety procedures.
A LEL sensor will be used to ensure that testing staff are performing activities within a safe
environment. The concentration measurements will be made at a minimum of four equally
spaced positions around the leak on concentric circles at two distances of approximately 1 and 5
meters from the leak (see Figure 2), with at least one position nominally downwind of the
perceived predominant wind direction. The actual measurement distances will be determined
based on readings from an LEL sensor. The concentration measurements will be made at three
heights corresponding to approximately knee, waist, and chest height and are intended to be
indications of the plume position and relative concentration for illustrative purposes representing
an "instantaneous" indication of the plume position rather than a long-term monitoring of plume
position during testing.
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 24 of 45
Version 1.0
September 30, 2010
•^Sampling locations
Figure 2. Diagram of Sampling Near Intentional Leak
A minimum of one portable meteorological station will be installed by Battelle staff and
operated in the vicinity of the virtual pipeline during each day of testing to record ambient
temperature, barometric pressure, relative humidity, and wind speed/direction. If available,
additional portable meteorological stations will be deployed along the virtual pipeline.
Meteorological measurements will also be recorded using sensors installed at the RMOTC
central meteorological station. Spot checks of ambient temperature, barometric pressure, relative
humidity, and wind speed will also be recorded daily using calibrated handheld sensors.
Testing will be performed over three days during the field campaign. On each day, a
total of three to six test flights will be performed to characterize leaks in the virtual pipeline.
Between four and eight leaks will be established for each test flight. The number and position of
the leaks will not be revealed to the vendors until after completion of the test flights and receipt
of the data report from the vendors. Between flights, the technology operator will be allowed to
review collected data and make adjustments to the technology or data collection parameters as
deemed necessary. Any adjustments will be observed and documented by Battelle technical
staff. During each day of testing at least one flight will be performed within each of the
following time periods to evaluate performance under differing daylight conditions:
Period 1: 8:00 - 10:00 a.m.,
Period 2: 11:00- 1:00 p.m.,
Periods: 3:00 - 5:00 p.m.
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 25 of 45
Version 1.0
September 30, 2010
Additional flights may be performed within these periods or between periods. Between flights,
the intentional leaks may be changed by RMOTC, Battelle, or ETV Canada staff to vary the
conditions of testing.
As soon as possible after the completion of each test day, the operators will provide the
Battelle and ETV Canada technical staff with preliminary results in the form of a completed
table indicating the position and estimated leak rate (if available) for each of the detected leaks
(see example table in Appendix A). Preliminary results should be provided within 24-hours after
each testing day, with a complete, detailed report provided within two weeks of the completion
of the testing campaign.
Bl.1.1 Leak location mapping capabilities
During the field campaign, a summary table will be prepared by Battelle technical staff
showing the GPS coordinates of each leak along with the measured leak rates in scfh and
ground-truth concentration measurements, (i.e., measurements from FID or LEL sensors). The
GPS coordinates of the actual leaks and the leaks detected by the airborne leak detection
technologies will be tabulated and reported to assess leak location mapping capabilities. False
positive and false negative indications (i.e., detect/non-detect) by the airborne leak detection
technologies will also be indentified and reported.
Bl.1.2 Leak rate determination
Quantitative estimates of leak rates determined from the airborne leak detection
technologies will be compared to the actual measured leak rates and will be reported, along with
the relative percent difference between the measured and actual rates. Since the technologies
being tested may measure only one component of natural gas, the composition of the natural gas
released from the intentional leaks will be measured and provided to the technology vendors to
allow for the calculation of the gas mixture leak rate based on the measurement of only one
component (e.g., natural gas leak rate based on measurement of ethane only).
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 26 of 45
Version 1.0
September 30, 2010
Bl.1.3 Pipeline inspection rate and capabilities
The pipeline inspection capabilities will be determined from both the fraction of pipeline
that the systems are able to fly over and the time required to inspect the virtual pipeline used for
the verification test. The fraction of pipeline scanned per flight pass will be calculated and
reported as:
Miles of pipeline inspected
Fraction =
Total miles of pipeline in test.
The pipeline inspection rate will be calculated and reported in terms of miles of pipeline
inspected per hour of flight as:
Miles of pipeline inspected
Rate =
Duration of inspection.
For each technology, the minimum leak rate that was detected will also be reported to indicate
the level of sensitivity that was achieved at the reported inspection rate to allow potential users of
these technologies to assess appropriateness for their needs.
Bl.1.4 Speciation (ethane/methane discrimination)
The majority of the intentional leaks will involve natural gas, however, the ability of the
technologies to speciate pure methane and ethane will be assessed by including intentional leaks
of pure ethane or methane at different locations along the pipeline section being inspected.
Based on the measurement capabilities of the technologies being verified, at least one of the
leaks for each test run will be either methane or ethane. Species that cannot be measured by the
technology being tested will not be included as a pure gas.
Bl.1.5 Data processing time between runs
Data processing time for the individual technologies being tested may vary dramatically
depending on the amount of post-test processing that is necessary. For this verification test, the
vendors will submit preliminary data within 24 hours of completion of each day of testing, with a
final comprehensive report submitted within two weeks of completion of the testing campaign.
Data processing time will be reported in terms of the number of days for which preliminary data
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 2 7 of 45
Version 1.0
September 30, 2010
are submitted within 24 hours of testing completion. Preliminary results may simply include the
identification of approximate GPS coordinates for individual leak sites. The time from
completion of the testing campaign that is required to submit the final reports will also be
presented.
Bl.1.6 Data Completeness
Data completeness will be calculated as the percentage of valid data collected during
each test run from the initiation to completion of the flight path over the virtual pipeline. The
data completeness for each test run will be reported along with the fraction of test runs from
which at least 75% valid data are collected.
El. 1.7 Operational Factors
Operational factors such as maintenance needs, data output, consumables used, ease of
use, repair requirements, etc., will be evaluated based on observations recorded by Battelle or
ETV Canada staff, and explained by the vendors as needed. Battelle or ETV Canada technical
staff will be present during testing, including during each test run, and will record all activities
performed on the monitoring systems. A LRB will be maintained by Battelle and will be used to
enter daily observations on these factors. Examples of information to be recorded in the record
books include observations about meteorological conditions; use or replacement of any
consumables for the airborne leak detection systems; the effort or cost associated with
maintenance or repair; vendor effort for repair or maintenance; the duration and causes of any
down-time or data acquisition failure; and observations about ease of use of the airborne leak
detection systems. These observations will be summarized to aid in describing operational
performance in the joint verification report.
B1.2 Reporting
Reporting on each of the variables described above will be based on the results of the
verification testing, and information on the operational performance will be compiled and
reported. A joint Verification Report will be prepared that presents a summary of the test
procedures and test data, as well as the results of the evaluation of those data.
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 28 of 45
Version 1.0
September 30, 2010
Operational aspects of the leak detection systems will be recorded by Battelle testing staff
at the time of observation during the field tests, and summarized in the joint Verification Report.
For example, descriptions of the data acquisition procedures, use of vendor-supplied proprietary
software, consumables used, repairs and maintenance needed, and the nature of any problems
will be presented in the report. The verification report will briefly describe the U.S. and
Canadian ETV programs, and the procedures used in verification testing. The results of the
verification test regarding performance of the leak detection systems will be stated
quantitatively. The draft joint verification report will be subjected to review by the vendor, EPA,
Environment Canada, and peer reviewers. The resulting review comments will be addressed in a
subsequent revision of the report, and the peer review comments and responses will be tabulated
to document the peer review process and submitted to EPA and Environment Canada. The
reporting and review process will be conducted according to the requirements of the ETV/AMS
Center QMP and the Canadian ETV Program GVP.
B2 SAMPLING METHOD REQUIREMENTS
No samples will be collected during this verification test.
B3 SAMPLE HANDLING AND CUSTODY REQUIREMENTS
No samples will be collected during this verification test.
B4 ANALYTICAL METHOD REQUIREMENTS
B4.1 FID Hydrocarbon Measurements
FID measurements will be performed in the vicinity of each of the leaks prior to and at
the end of each testing period. Field measurements will be made by Battelle staff using a
Foxboro TVA 1000 FID analyzer, or equivalent. Field testing staff will make measurements at
eight locations surrounding each leak, with four measurements approximately 90 degrees apart
on a circle with a radius of one meter centered on the leak, and four similar measurements five
meters from the leak. The concentration measurements will be made at three heights
corresponding to approximately knee, waist, and chest height and are intended to be indications
of the plume position and relative concentration for illustrative purposes rather than definitive
mapping of the plume. The actual distances from the leak will be determined using an LEL
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 29 of 45
Version 1.0
September 30, 2010
sensor to ensure safety for the measurement personnel. FID measurements will only be made
when airborne concentrations are below 10% of the LEL.
B5 QUALITY CONTROL REQUIREMENTS
Table 4 summarizes the QC requirements for the verification test, including frequency of
the QC activities, the acceptance criteria, and corrective actions.
Table 4. Quality Control Activities
QC Activity
Leak rate
measurement
FID calibration
FID calibration
check
Duplicate
measurement of GPS
coordinates
Frequency
Each leak
At the beginning and
end of the testing
day
At the beginning of
the testing day
Each leak
Minimum Acceptance
Criterion
± 10% of nominal
± 10% of actual
concentration for span
value;
±1% of range setting for
zero air
± 10% of actual
concentration
±2m
Corrective Action
Adjust flow rate. Replace
metering device if necessary.
Recalibrate FID. Remeasure.
Replace or repair FID if
necessary.
Recalibrate FID. Remeasure.
Replace or repair FID if
necessary.
Remeasurement of
coordinates. Replacement of
GPS units if necessary.
B6 INSTRUMENT/EQUIPMENT TESTING, INSPECTION, AND MAINTENANCE
Prior to testing the gas leak system components, including casings, compressed gas
cylinders, regulators, and leak metering devices, will be visually inspected by RMOTC staff to
ensure proper structural integrity and installation. Damaged or improperly operating components
will be removed from the gas system and replaced. Once inspected, care will be observed to
properly maintain the gas system and avoid damage to any components during the field testing
campaign.
Instrumentation and equipment used for measurement of gas flow rates and airborne gas
concentrations will be tested to ensure proper operation as summarized in Table 4. This testing
will include instrument calibrations and calibration checks to be performed prior to and at the
completion of testing. Based on the relatively short duration of the field testing campaign, no
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 30 of 45
Version 1.0
September 30, 2010
preventive maintenance activities outside of routine operating procedures and calibration
procedures are anticipated.
B7 INSTRUMENT CALIBRATION AND FREQUENCY
B7.1 FID Analyzer
The FID analyzers will be calibrated daily during the field campaign. The analyzers will
undergo a minimum of a two-point calibration using a compressed methane gas standard at a
concentration at the upper end of the range of concentrations expected to be measured in the
field (e.g., < 10% LEL), and using zero air. The calibration will be verified by measuring the
calibration standard gases after completion of the calibration, and comparing the measured
concentrations to the expected concentrations. The measured concentration at the upper
calibration point should agree with the actual concentration to within ± 10%. The measured
value at zero should be equal to zero within 1% of the range setting. A separate NIST traceable
standard, whose concentration is approximately midrange of the anticipated testing range, will be
analyzed following the calibration each day as an independent check standard. Agreement
between the measured and actual concentrations should be within ± 10%. Testing will not begin
before this daily calibration standard meets acceptable QC criteria.
B7.2 Meteorological Sensors
The meteorological sensors used for this verification will include those installed on the
central weather station at the RMOTC as well as a portable meteorological station provided by
Battelle. The sensors on the portable meteorological station will be calibrated by the
manufacturer within 12 months prior to the beginning of the field campaign. These calibrated
sensors will be collocated with the RMOTC sensors prior to the beginning of the field campaign
to verify the accuracy of the RMOTC sensors. Handheld meteorological sensor units (Kestrel
Model 4000) will be used to conduct spot check measurements of ambient temperature,
barometric pressure, relative humidity, and wind speed. These sensors will be calibrated within
12 months prior to the beginning of the field campaign. Table 5 shows the nominal sensor
resolution and acceptance criteria for the calibration of the meteorological sensors.
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 31 of 45
Version 1.0
September 30, 2010
Table 5. Resolution and Calibration Acceptance Criteria for Meteorological Sensors
Parameter
Wind speed
Wind direction
Temperature
Barometric pressure
Relative humidity
Sensor Resolution
0.1 m/s
1°
0.1 °C
0.05 mmHg
2%RH
Acceptance Criterion
0.2 m/s + 5%
<5°
0.5 °C
< 1 mmHg
<5%
B7.3 Flow Meters
Approximate flow rates will be established based on preliminary readings from factory
pre-calibrated rotameters (e.g., Dwyer or similar). Actual flow rates of the intentional leaks will
be measured by Battelle staff using one or more calibrated dry gas meters or rotameters. If
needed, multiple flow meters will be used in order to span the range of anticipated flow rates.
The flow meters used to measure the intentional leak rates will undergo NIST-traceable
calibration within two months prior to the beginning of the field campaign, and will have a
calibrated accuracy of < 1.5%. The calibration of these meters will be verified within 2 weeks
of completion of the field campaign to assess the degree of drift that may have been introduced
to the meters. If the measured drift exceeds 1.5%, the leak rates measured during the field
campaign will be flagged and may be corrected based on extrapolation between the pre- and
post-campaign calibration results.
B7.4 Aircraft Sensors
On-board instrumentation for measurement of elevation and air speed will be calibrated
per the standard procedures of the aviation company contracted to perform the flights and will
comply with appropriate Federal Aviation Administration regulations. The accuracy of these
sensors will not be verified in this verification test.
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 32 of 45
Version 1.0
September 30, 2010
B8 INSPECTION/ACCEPTANCE OF SUPPLIES AND CONSUMABLES
Upon receipt of any supplies or consumables used for the testing Battelle or RMOTC
facility staff will visually inspect and ensure that the materials received are those that were
ordered and that there are no visual signs of damage that could compromise the suitability of the
materials. If damaged or inappropriate goods are received, they will be returned or disposed of
and arrangements will be made to receive replacement materials. Certificates of analysis or
other documentation of analytical purity will be checked for all reagents, gases, and standards
used to ensure suitability for this verification test. Unsuitable materials will be returned or
disposed of and arrangements for the receipt of replacement materials will be made.
B9 NON-DIRECT MEASUREMENTS
No non-direct measurements will be used during this verification test.
B10 DATA MANAGEMENT
Various types of data will be acquired and recorded electronically or manually by
Battelle, ETV Canada, RMOTC, and vendor staff during this verification test. Table 6
summarizes the types of data to be recorded. All maintenance activities, repairs, calibrations,
and operator observations relevant to the operation of the monitoring systems being tested will
be documented by Battelle, ETV Canada, or vendor staff in the LRB or data forms developed for
the test. Measurements taken during the collection of the reference method samples will be
compiled by subcontractor staff in electronic format, and submitted to Battelle in the form of an
analytical report at the conclusion of reference sampling periods. A dedicated shared folder
within the ETV AMS Center SharePoint site will be established for all project records.
Battelle will provide technology test data and associated reference data (including
records, data sheets, and notebook records) from the first day of testing within one day of receipt
to EPA and ETV Canada for simultaneous review. The goal of this data delivery schedule is
prompt identification and resolution of any data collection or recording issues. The final data
report generated by each technology vendor for the test campaign will be provided by Battelle to
EPA and ETV Canada within one week of completion of field testing. This will be provided to
EPA and ETV Canada for informational purposes as the Battelle audit of data quality will not
have been completed.
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 33 of 45
Version 1.0
September 30, 2010
Records received by or generated by any of the verification staff during the verification test
will be reviewed by a Battelle staff member within two weeks of receipt or generation,
respectively, before the records are used to calculate, evaluate, or report verification results. The
review will be documented as the dated initials of the reviewer. Some of the checks that will be
performed include the following.
> Check that QC samples and calibration standards were analyzed according to the
TQAP, and the acceptance criteria were met. Ensure corrective action for
exceedances was taken.
> Ensure that 100% hand-entered and/or manually calculated data were checked for
accuracy.
> Check that the calculations performed by software were verified at a frequency
sufficient to ensure that the formulas are correct, appropriate, and consistent.
> Ensure that for each cut and paste function, the first and last data values were correct.
> Ensure data are reported in the units specified in the TQAP.
> Check that the results of QC samples are reported.
If a Battelle staff member generated the record, this review will be performed by a
Battelle technical staff member involved in the verification test, but not the staff member that
originally received or generated the record. The review will be documented by the person
performing the review by adding his/her initials and date to the hard copy of the record being
reviewed. In addition, data calculations performed by verification staff will be spot-checked by
Battelle technical staff to ensure that calculations are performed correctly. Calculations to be
checked include any statistical calculations described in this TQAP.
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 34 of 45
Version 1.0
September 30, 2010
Table 6. Summary of Data Recording Process
Data to Be Recorded
Dates, times, and
details of test events,
leak rates, leak
locations, pipeline
coordinates,
meteorological data
Flight times, flight
paths, and flight
durations for each test
run
Leak detection system
readings
FID method
procedures, results,
calibrations, QC, etc.
Where
Recorded
ETV LRBs, field
sampling records
ETV LRBs, or
electronically
Recorded
electronically by
each system and
then downloaded
to computer daily
ETV LRBs, or
data recording
forms
How Often
Recorded
Start/end of test
event
When
performed
Recorded
continuously
by each
monitoring
system
Throughout
sampling and
analysis
processes
By Whom
Testing staff
Vendor and/or
testing staff
Vendor for
transfer to
testing staff
Testing staff
Disposition of
Data
Used to
organize/check test
results; manually
incorporated in data
spreadsheets as
necessary
Incorporated in
verification report
as necessary
Converted to
spreadsheet for
statistical analysis
and comparisons
Retained as
documentation of
reference method
performance
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 35 of 45
Version 1.0
September 30, 2010
SECTION C
ASSESSMENT AND OVERSIGHT
Cl ASSESSMENTS AND RESPONSE ACTIONS
Every effort will be made in this verification test to anticipate and resolve potential
problems before the quality of performance is compromised. One of the major objectives of this
TQAP is to establish mechanisms necessary to ensure this. Internal QC measures described in
this TQAP, which is peer reviewed by a panel of outside experts, implemented by the technical
staff and monitored by the Battelle and ETV Canada VTCs, will give information on data
quality. The responsibility for interpreting the results of these checks and resolving any potential
problems resides with the Battelle VTC, who will contact the Battelle AMS Center Manager,
Battelle AMS Center Quality Manager, EPA AMS Center Project Officer, and EPA AMS Center
Quality Manager if any deviations from the TQAP are observed. The Battelle VTC will describe
the deviation in a teleconference or by e-mail, and once a path forward is determined and agreed
upon with EPA, the deviation form will be completed.
Technical staff has the responsibility to identify problems that could affect data quality or
the ability to use the data. Any problems that are identified will be reported to the Battelle and
ETV Canada VTCs, who will work with the Battelle and ETV Canada Quality Managers to
resolve any issues. Action will be taken to identify and appropriately address the issue, and
minimize losses and correct data, where possible. Independent of any EPA QA activities,
Battelle and ETV Canada will be responsible for ensuring that the following audits are conducted
as part of this verification test.
Cl.l Performance Evaluation Audit
A PEA will be conducted immediately prior to or within the first day of testing during the
field campaign to assess the quality of the critical measurements associated with the leak rate
determination and the coordinates of the individual leaks. Table 7 shows the critical
measurements to be audited, with the audit procedures and acceptance criteria for the audit
comparisons. If the PEA results do not meet the acceptance criteria shown, they will be
repeated. If the outlying results persist, a change in instrumentation and a repeat of the PEA may
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 36 of 45
Version 1.0
September 30, 2010
be considered, and data will be flagged until the PEA results are acceptable. These audits will be
performed once during the field campaign of the verification test, and will be the responsibility
of the Battelle VTC. Results from the PEAs will be provided to EPA within 10 days of receipt
of the results.
Table 7. Methods and Acceptance Criteria for PEA Measurements
Critical Measurement
Leak flow rate
GPS coordinates
FID measurements
PEA Method
Compare to independent flow
transfer standard
Compare to independent handheld
GPS unit
Measurement of independent
calibration gas
Acceptance Criteria
± 10% actual flow rate
± 2 meters
± 10% actual concentration
The PEA of the leak flow rate will be conducted by Battelle staff using an independent
NIST-traceable flow transfer standard. The target criterion for this audit is agreement between
the measured and nominal flow rate within ± 10%. If this criterion is not met, the cause of the
problem will be investigated and corrected if possible. Components of the leak generation
system will be replaced as necessary until the flow rate criterion is met.
The PEA of the GPS device will be conducted by Battelle staff to document the
coordinates of the individual leaks will involve comparison to an independent GPS unit. The
target agreement between the coordinate readings is ± 2 m. If this criterion is not met, the
readings will be performed again. If necessary, alternate GPS units will be obtained and used for
the coordinate measurements.
The PEA of the FID measurements will be performed by supplying the FID with a
standard compressed natural gas from an independent NIST-traceable gas cylinder at three
concentrations in the range from 0 to 1.0%. These audit sample(s) will be analyzed and
compared to the known sample concentrations. The acceptance criterion for this audit is for
agreement between the measured and actual concentrations within ± 10%. If this criterion is not
met, the cause of the problem will be investigated and corrected. The FID will be recalibrated or
replaced to meet the criterion.
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 3 7 of 45
Version 1.0
September 30, 2010
C1.2 Technical Systems Audits
The Battelle and ETV Canada Quality Managers will each perform a TSA during this
verification test. The purpose of this audit is to ensure that the verification test is being
performed in accordance with the AMS Center QMP,1 the ETV Canada GVP,3 this TQAP, and
any Standard Operating Procedures used by the test facility. The Battelle Quality Manager will
prepare a project-specific checklist based on the TQAP requirements to guide the TSA, which
will compare actual test procedures to those specified or referenced in this plan, and review data
acquisition and handling procedures. In the TSA, the Battelle and ETV Canada Quality
Managers will tour the test site; observe testing procedures; and review/inspect documentation
including data forms and laboratory record books. They may also check gas standard
certifications and data acquisition procedures, and may confer with the vendors and testing
facility staff. Separate TSA reports will be prepared by the two Quality Managers, including
separate statements of findings and the actions taken to address any adverse findings. The
Battelle Quality Manager will prepare an initial TSA report and will submit the report to the EPA
Quality Manager (with no corrective actions documented) and VTC within 10 business days
after completion of the audit. A copy of each final TSA report (with corrective actions
documented) will be provided to the EPA AMS Center Project Officer and Quality Manager
within 20 business days after completion of the audit. At EPA's discretion, EPA QA staff may
also conduct an independent on-site TSA during the verification test. The TSA findings will be
communicated to technical staff at the time of the audit and documented in a TSA report.
C1.3 Data Quality Audits
The Battelle and ETV Canada Quality Managers, or their designees, will audit 100% of
the calibration and QC data and at least 10% of the verification data shown in Table 7. They will
trace the data from initial acquisition (as received from the vendors and testing staff), through
reduction and statistical comparisons, to final reporting. All calculations performed on the data
undergoing the audit will be checked. The Battelle Quality Manager will review the data being
generated in the field during the TSA. Data must undergo a 100% verification by technical staff
(i.e., VTC, or designee) before it will be assessed as part of the data quality audit. Since the
testing is occurring over three days in the field, a single ADQ that assesses overall data quality,
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 38 of 45
Version 1.0
September 30, 2010
including accuracy and completeness of the technical report, will be prepared as a narrative and
distributed to the VTC and EPA within 10 business days of completion of the audit.
C1.4 QA/QC Reporting
Each assessment and audit will be documented in accordance with the respective
verification organization's QMP.1'2 The results of audits will be submitted as a draft that may
not contain finding resolution to the EPA Project Officer within 10 business days of conduct of
the audit. Assessment reports will include the following:
• Identification of project technology, location of audit, primary contact, and dates of
audit;
• Identification of applicable reference documents, standards and calculations used, as
appropriate;
• Identification of any adverse findings or potential problems;
• Recommendations for resolving problems;
• Citation of any noteworthy practices that may be of use to others;
• Copy of the audit checklist used, if appropriate;
• List of persons interviewed during the audit;
• Response to adverse findings or potential problems; and
• Confirmation that solutions have been implemented and are effective.
C2 REPORTS TO MANAGEMENT
The Quality Managers of the verification organizations, during the course of any
assessment or audit, will identify to the technical staff performing experimental activities any
immediate corrective action that should be taken. If serious quality problems exist, the Quality
Manager of either verification organization will notify the respective verification organization
Program Manager to request that a stop-work order be issued. The Battelle Quality Manager will
be responsible for preparation of TSA and ADQ audit reports for Battelle. The QA
representatives for ETV Canada and EPA, respectively, will be responsible for preparing reports
for audits they perform. The Battelle Quality Manager will route reports through the VTC and
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 39 of 45
Version 1.0
September 30, 2010
AMS Center Manager. A draft copy of the ISA and ADQ reports will be provided to the EPA
QAM within 10 business days of completion of the audits for review, although corrective actions
may not have been addressed prior to submission. Once the assessment report has been
prepared, the VTC will ensure that a response is provided for each adverse finding or potential
problem and will implement any necessary follow-up corrective action. The Quality Managers
of the verification organizations will verify that follow-up corrective action has been taken.
VTC responses to the audit comments are due to the QA representatives within 10 business days
of receipt and the final TSA and ADQ reports with responses will be provided to the EPA
Project Officer, QAM, and the ETV Canada Quality Manager within 10 business days.
The TQAP and final reports are reviewed by QA staff and Program Management staff of
the respective verification organizations, EPA, vendors, and peer reviewers and are approved by
Battelle, EPA, and ETV Canada. Upon final review and approval, both documents may be
posted on the verification organization's and verification program's Web site, if applicable.
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 40 of 45
Version 1.0
September 30, 2010
SECTION D
DATA VERIFICATION AND USABILITY
Dl DATA REVIEW AND VERIFICATION REQUIREMENTS
The key data review and data verification specifications for this test are stated in Section
B of this TQAP. In general, the data review requirements specify that the first day's data
generated during this test will be made available to the VTC, EPA AMS Center QAM, and
EPA's Project Officer for the AMS Center within one day for a preliminary review. Also, the
full data set from the field test campaign will be made available to the VTC, EPA AMS Center
QAM, and EPA's Project Officer for the AMS Center within one week of completion. This
process will ensure that the airborne leak detection systems data and supporting leak location,
flow rate, and "ground-truth" concentration data were collected under appropriate testing
conditions and that the data meet the specifications of this TQAP.
The data verification requirements for this test involve an assessment of the quality of the
data relative to the DQIs, stated QC requirements, and audit acceptance criteria specified for this
test in Tables 2, 4, and 6, respectively. The results of the QC activities and the PEA results will
be compared with the appropriate acceptance criteria in Tables 4 and 6, respectively, to ensure
that the measurements were properly performed. Furthermore, the results of the leak rate and
GPS coordinate measurements will be compared with the appropriate DQIs listed in Table 2 to
verify the quality of the data. Any deficiencies in these data will be flagged and may be
excluded from any statistical comparisons to the leak detection systems being tested, unless these
deviations are accompanied by descriptions of their potential impacts on the data quality.
D2 VERIFICATION METHODS
Data verification is conducted by the VTC, or his designee, as part of the data review as
described in Section BIO of this TQAP. A visual inspection of handwritten data will be
conducted to ensure that all entries were properly recorded or transcribed, and that any erroneous
entries were properly noted (i.e., single line through the entry, with an error code and the initials
of the recorder and date of entry). Electronic data from the leak detection systems and
meteorological equipment used during the test will be inspected to ensure proper transfer from
the data logging system. All calculations used to transform the data will be reviewed to ensure
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 41 of 45
Version 1.0
September 30, 2010
the accuracy and the appropriateness of the calculations. Calculations performed manually will
be reviewed and repeated using a handheld calculator or commercial software (e.g., Excel).
Calculations performed using standard commercial office software (e.g., Excel) will be reviewed
by inspection of the equations used for the calculations and verification of selected calculations
by handheld calculator. Calculations performed using specialized commercial software (i.e., for
analytical instrumentation) will be reviewed by inspection and, when feasible, verified by
handheld calculator, or standard commercial office software.
To ensure that the data generated from this test meet the goals of the test, a number of
data verification procedures will be performed. Sections B and C of this TQAP provide a
description of the safeguards employed for this verification test. Data verification efforts include
the completion of QC activities, and the performance of TSAs and PEAs as described in Section
C. The data from this test will be evaluated relative to the measurement DQIs described in
Section A7, and the PEA acceptance criteria given in Section Cl. 1 of this TQAP. Data failing to
meet these criteria will be flagged in the dataset and not used for evaluation of the monitoring
systems, unless these deviations are accompanied by descriptions of their potential impacts on
the data quality.
An ADQ will be conducted by the Quality Managers of the two verification organizations
to ensure that data review and verification procedures were completed, and to ensure the overall
quality of the data.
D3 RECONCILIATION WITH USER REQUIREMENTS
This purpose of this verification test is to evaluate the performance of airborne leak
detection systems. In part, this evaluation will include comparisons of results from the leak
detection systems to the known positions and leak rates of intentional leaks generated in a
"virtual pipeline". To meet the requirements of the user community, the data collected during
this verification test will include thorough documentation of the performance of the leak
detection systems and the intentional leaks generated during the verification test. The data
review and verification procedures described above will assure that data meeting these
requirements are accurately presented in the verification reports generated from this test, and will
assure that data not meeting these requirements will be appropriately flagged and discussed in
the verification reports.
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 42 of 45
Version 1.0
September 30, 2010
This joint TQAP and the resulting verification reports will be subjected to review by the
vendor, verification organization staff, test collaborators, the verification programs, and expert
peer reviewers. The reviews of this TQAP will help to improve the design of the verification test
and the resulting report(s) such that they better meet the needs of potential users of these airborne
leak detection systems.
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 43 of 45
Version 1.0
September 30, 2010
SECTION E
REFERENCES
1. Battelle, Quality Management Plan for the ETV Advanced Monitoring Systems Center,
Version 7.0, U.S. EPA Environmental Technology Verification Program, prepared by
Battelle, Columbus, Ohio, November 2008.
2. ETV CANADA General Verification Protocol (GVP). February 2007.
3. U.S. EPA, Environmental Technology Verification Program Quality Management Plan, EPA
Report No: 600/R-08/009 EPA/600/R-03/021, U.S. Environmental Protection Agency,
Cincinnati, Ohio, January 2008.
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 44 of 45
Version 1.0
September 30, 2010
APPENDIX A
Example Data Sheet
-------
Airborne Leak Detection Systems
Test/QA Plan
Page 45 of 45
Version 1.0
September 30, 2010
Example Field Data Sheet
for Verification Testing of Airborne Leak Detection Systems
Vendor:
Test Date:
Test Run:
Technology:
Flight start time:
Flight end time:
Leak#
GPS Coordinates
Leak
Detection
Time
Pass
number
Estimated
Leak Rate
Comments
Signed:
Received by:
Date:
Date/Time:
------- |