Batteiie
Tne Business of Innovation
Environmental Technology
Verification Program
Advanced Monitoring Systems Center
Test/QA Plan for Verification of
Alternative Technologies for Sealed
Source Radiography Cameras
-------
Test/QA Plan for Verification of
Alternative Technologies for Sealed Source Radiography Cameras
May 28, 2010
Prepared by
Battelle
505 King Avenue
Columbus, OH 43201-2693
Under
Contract No. GS-23F-0011L
Task Order 1145
National Homeland Security Research Center (NHSRC)
US EPA
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 3 of 39
Version 1.0
Date: 5/28/2010
TABLE OF CONTENTS
Section Page
SECTION A PROJECT MANAGEMENT 8
Al Verification Test Organization 8
A2 Background 15
A3 Verification Test Description and Schedule 16
A4 Quality Objectives 18
A5 Special Training/Certification 19
A6 Documentation and Records 19
SECTION B MEASUREMENT AND DATA ACQUISITION 21
Bl Experimental Design 21
B2 Sampling Requirements 29
B3 Sample Handling and Custody Requirements 29
B4 Reference Method 29
B5 Quality Control Criteria for Reference Method Measurement Data 30
B6 Instrument/Equipment Testing, Inspection, and Maintenance 30
B7 Instrument Calibration and Frequency 31
B8 Inspection/Acceptance of Supplies and Consumables 31
B9 Non-Direct Measurements 31
BIO Data Management 31
SECTION C ASSESSMENT AND OVERSIGHT 33
Cl Assessments and Response Actions 33
C2 Reports to Management 35
SECTION D DATA VALIDATION AND USABILITY 36
Dl Data Review, Validation, and Verification Requirements 36
D2 Validation and Verification Methods 36
D3 Reconciliation with User Requirements 37
SECTION E REFERENCES 39
El References 39
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 4 of 39
Version 1.0
Date: 5/28/2010
List of Figures Page
Figure 1. Organization Chart for the Verification Test 9
Figure 2. Example of zones for assessment of Pipe Sample 1 weld 24
List of Tables
Table 1. Planned Verification Test Schedule 17
Table 2. Summary of Data Recording Process 20
Table 3. Weld Defect Assessment Table 25
Table 4. Corrosion Image Assessment Table 25
Table 5. Simulated Corrosion Defect Assessment 26
Table 6. Drill Hole Defect Assessment 27
Table 7. DQIs and Criteria of Critical Measurements for Reference Method 30
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 5 of 39
Version 1.0
Date: 5/28/2010
ETV Advanced Monitoring Systems Center
Test/QA Plan for Verification of
Alternative Technologies for Sealed Source Radiography Cameras
May 28, 2010
APPROVAL:
Name
Company
Date
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 6 of 39
Version 1.0
Date: 5/28/2010
ACRONYMS AND ABBREVIATIONS
ADQ Audit of Data Quality
AMS Advanced Monitoring Systems
ASNT American Society for Nondestructive Testing
ATI Alternative Technologies Initiative
DQI Data quality indicator
EPA U.S. Environmental Protection Agency
ETV Environmental Technology Verification
IAEA International Atomic Energy Agency
IQI Image quality indicator
IRRSP Industrial Radiography Radiation Safety Personnel
LRB Laboratory record book
NDT Non-destructive testing
NHSRC National Homeland Security Research Center
OAR Office of Air and Radiation
pdf Adobe portable document format
PE Performance evaluation
QA Quality assurance
QC Quality control
QMP Quality management plan
SOP Standard Operating Procedure
TSA Technical systems audit
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 7 of 39
Version 1.0
Date: 5/28/2010
DISTRIBUTION LIST
Vendors
VJ Inspections Systems
80 Commerce Street
East Haven, CT 06512
EPA Task Order Project Officer
Madeleine Nawar
USEP A/OAR
Mail Code 6608J
1200 Pennsylvania Ave, NW
Washington, DC 20460
EPA QA Manager
Michelle Henderson
U.S. Environmental Protection Agency
26 West Martin Luther King Dr.
MS 207
Cincinnati, OH 45268
EPA Project Officer
John McKernan
U.S. Environmental Protection Agency
26 West Martin Luther King Dr.
MS 208
Cincinnati, OH 45268
Peer Reviewers
Terry Webb
BP Refining and Marketing
Office: BP Toledo Refinery
Toledo, OH
Mike Eagle
USEP A/OAR
Mail Code 6608J
1200 Pennsylvania Ave, NW
Washington, DC 20460
Temeka Taplin
U.S. Department of Energy
NA-211/Forrestal Building
1000 Independence Ave., S.W.
Washington, D.C. 20585
Battelle
Amy Dindal
Stephanie Buehler
Karen Riggs
Zachary Willenberg
Battelle
505 King Ave.
Columbus, OH 43201
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 8 of 39
Version 1.0
Date: 5/28/2010
SECTION A
PROJECT MANAGEMENT
Al VERIFICATION TEST ORGANIZATION
The verification test will be conducted under the U.S. Environmental Protection Agency
(EPA) Environmental Technology Verification (ETV) Program. It will be performed by Battelle,
which is managing the ETV Advanced Monitoring Systems (AMS) Center through a cooperative
agreement with EPA. The scope of the AMS Center covers verification of monitoring
technologies for contaminants and natural species in air, water, and soil.
The day to day operations of this verification test will be coordinated and supervised by
Battelle, with the participation of the vendors who will be having the performance of their
technologies which offer an alternative to sealed radioactive sourced radiography cameras
verified. Testing will be conducted at Battelle facilities in West Jefferson, Ohio. Each vendor
will provide and operate their respective technology.
The organization chart in Figure 1 identifies the responsibilities of the organizations and
individuals associated with the verification test. Roles and responsibilities are defined further
below. Quality Assurance (QA) oversight will be provided by the Battelle QA Manager and also
by the EPA AMS Center Quality Manager, at her discretion. Because this verification will be
referenced by the Office of Air and Radiation's Alternative Technology Initiative, it was decided
to establish the testing as a Quality Category II, requiring a QA review of 25% of the test data
and additional peer-reviewers (see Section Cl).
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 9 of 39
Version 1.0
Date: 5/28/2010
Battelle
Management
Amy Dindal
Battelle AMS
Center
Manager/Verification
Testing Leader
Dr. John McKernan
EPA AMS Center
Project Officer
Zachary Willenberg
Battelle
Quality Manager
Michelle Henderson
EPA AMS Center
Quality Manager
Dr. Stephanie Buehler
Battelle
Verification
Test Coordinator
Battelle Technical Staff
And
Reference Analysis
Subcontractor (Mistras)
Vendor
Representatives
Figure 1. Organization Chart for the Verification Test
(dotted lines indicate lines of communication)
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 10 of 39
Version 1.0
Date: 5/28/2010
Al.l Battelle
Dr. Stephanie Buehler is the AMS Center's Verification Test Coordinator for this test. In
this role, Dr. Buehler will have overall responsibility for ensuring that the technical, schedule,
and cost goals established for the verification test are met. Specifically, Dr. Buehler will:
• Prepare the test/QA plan, verification reports, and verification statements;
• Revise the test/QA plan, verification reports, and verification statements in response
to reviewers' comments;
• Establish a budget for the verification test and manage staff to ensure the budget is
not exceeded;
• Assemble a team of qualified technical staff to conduct the verification test;
• Direct the team in performing the verification test in accordance with this test/QA
plan;
• Hold a kick-off meeting approximately one week prior to the start of the verification
test to review the critical logistical, technical, and administrative aspects of the
verification test (responsibility for each aspect of the verification test will be
confirmed);
• Ensure that all quality procedures specified in this test/QA plan and in the AMS
Center Quality Management Plan1 (QMP) are followed;
• Serve as the primary point of contact for vendor representatives;
• Ensure that confidentiality of sensitive vendor information is maintained;
• Assist vendors as needed during verification testing;
• Become familiar with the operation and maintenance of the technologies through
instruction by the vendors, if needed;
• Respond to any issues raised in assessment reports, audits, or from test staff
observations, and institute corrective action as necessary; and
• Coordinate distribution of the final test/QA plan, verification reports, and verification
statements.
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 11 of 39
Version 1.0
Date: 5/28/2010
Ms. Amy Dindal will serve as Verification Testing Leader and is also Battelle's Manager
for the AMS Center. Ms. Dindal will:
• Support Dr. Buehler in preparing the test/QA plan and organizing the testing;
• Review the final test/QA plan;
• Attend the verification test kick-off meeting;
• Review the draft and final verification reports and verification statements;
• Ensure that necessary Battelle resources, including staff and facilities, are committed
to the verification test;
• Ensure that confidentiality of sensitive vendor information is maintained;
• Support Dr. Buehler in responding to any issues raised in assessment reports and
audits;
• Maintain communication with EPA's technical and quality managers; and
• Issue a stop work order if Battelle or EPA QA staff discovers adverse findings that
will compromise test results.
Battelle Technical Staff will support Dr. Buehler in planning and conducting the
verification test. The responsibilities of the technical staff will be to:
• Assist in planning for the test and making arrangements for the receipt of the
technologies;
• Attend the verification test kick-off meeting;
• Assist vendor staff as needed during verification testing;
• Conduct and observe verification testing, as appropriate;
• Coordinate and observe reference testing, as appropriate;
• Perform statistical calculations specified in this test/QA plan on the technology data
as needed;
• Provide results of statistical calculations and associated discussion for the verification
reports as needed; and
• Support Dr. Buehler in responding to any issues raised in assessment reports and
audits related to statistics and data reduction as needed.
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 12 of 39
Version 1.0
Date: 5/28/2010
Mr. Zachary Willenberg is Battelle's QA Manager for this test. Mr. Willenberg will:
• Review the draft and final test/QA plan;
• Attend the verification test kick-off meeting;
• Conduct a technical systems audit at least once during the verification test, or
designate other QA staff to conduct the audit;
• Audit at least 25% of the verification data or designate other QA staff to conduct the
data audit;
• Prepare and distribute an assessment report for each audit;
• Verify implementation of any necessary corrective action;
• Request that Battelle's AMS Center Manager issue a stop work order if audits
indicate that data quality is being compromised;
• Provide a summary of the QA/QC activities and results for the verification reports;
and
• Review the draft and final verification reports and verification statements.
A1.2 Technology Vendors
The responsibilities of the technology vendors are as follows:
• Review and provide comments on the draft test/QA plan;
• Accept (by signature of a company representative) the final test/QA plan prior to test
initiation;
• Provide one unit of their technology for evaluation during the verification test;
• Provide all equipment/supplies/reagents/consumables needed to operate their
technology for the duration of the verification test;
• Provide an appropriately trained and licensed (e.g., American Society for
Nondestructive Testing (ASNT) Industrial Radiography Radiation Safety Personnel
(IRRSP) Certification) person to operate their technology for the duration of the
verification test;
• Provide a copy of radiation safety operation license (e.g., ASNT IRRSP Certification,
Ohio Radioactive Materials License) or equivalent document for inclusion in the
verification test file;
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 13 of 39
Version 1.0
Date: 5/28/2010
• Provide maintenance and repair support for their technology, on-site if necessary,
throughout the duration of the verification test;
• Review and provide comments on the draft verification report and statement for their
respective technology; and
• Provide any applicable documentation related to testing of the vendor's technology,
such as calibration, testing results and observations, final images, and supply or
reagent certificate of authenticity (COA).
A1.3 EPA
EPA's responsibilities in the AMS Center are based on the requirements stated in the
"Environmental Technology Verification Program Quality Management Plan" (EPA ETV
QMP).2 The roles of specific EPA staff are indicated below.
Ms. Michelle Henderson is EPA's AMS Center QA Manager. For the verification test,
Ms. Henderson will:
• Review the draft and approve the final test/QA plan;
• Attend the verification kick-off meeting, as available;
• Review checklists, reports, report responses, and closure statements of TSA,
performance evaluation (PE) audits, and audits of data quality systems (ADQs)
conducted by Battelle;
• Perform an external TSA of field and/or laboratory activities, PE audits, and/or an
audit of data quality during the verification test, as available;
• Notify the EPA AMS Center Project Officer of the need for a stop work order if
evidence indicates that data quality is being compromised;
• Prepare and distribute an assessment report summarizing results of any external audit
performed;
• Review the first day of data from the verification test and provide immediate
comments if concerns are identified; and
• Review the draft and approve the final verification reports and verification
statements.
Dr. John McKernan is EPA's Project Officer for the AMS Center. Dr. McKernan will:
• Review the draft test/QA plan;
• Approve the final test/QA plan;
• Attend the verification kick-off meeting, as available;
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 14 of 39
Version 1.0
Date: 5/28/2010
• Be available during the verification test to review and authorize test/QA plan
deviations by phone and provide the name of a delegate to the Battelle AMS Center
Manager should he not be available during the testing period;
• Review the first day of data from the verification test and provide immediate
comments if concerns are identified;
• Review the draft verification reports and verification statements;
• Oversee the EPA review process for the test/QA plan, verification reports, and
verification statements;
• Coordinate the submission of verification reports and verification statements for final
EPA approval; and
• Post the test QA plan, verification reports, and verification statements on the ETV
website.
A1.4 Radiography Camera Subcontractor
This test will require the use of a radiography camera as the reference instrument. A
subcontractor with an appropriate safety license (e.g., ASNT ISSRP) to operate a radiography
camera will be selected to conduct all reference measurements. The responsibilities of the
subcontractor include the following:
• Conduct all reference measurements using a radiography camera;
• Supply all necessary equipment to obtain images using the radiography camera;
• Supply a licensed (e.g., ASNT IRRSP) technician to operate the radiography camera
and ensure that all appropriate radiation safety protocols are being properly followed,
including controlling access to areas near the radiological source during operation of
the radiography camera;
• Provide a copy of radiation safety operation license (e.g., ASNT IRRSP) to Battelle
for inclusion in the verification test file;
• Provide digital images of the selected defects to Battelle, including an accompanying
description or analysis of the findings (e.g., defect size, depth, etc.) as appropriate;
• Follow all Battelle radiation safety procedures, as appropriate; and
• Perform testing activities and data acquisition as specified in this test/QA plan.
A1.5 Verification Test Stakeholders
This test/QA plan and the verification report(s) and verification statement(s) based on
testing described in this document will be reviewed by experts in the fields related to NDT
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 15 of 39
Version 1.0
Date: 5/28/2010
pipeline inspections and alternatives to sealed-source technologies. Three stakeholders have
been providing input to this test/QA plan and have agreed to provide a peer review.
• Terry Webb, BP, Refining NOT Specialist
• Temeka Taplin, National Nuclear Security Administration, Department of Energy
• Madeleine Nawar, U.S. EPA
The responsibilities of verification test stakeholders include:
• Review and provide input to the test/QA plan; and
• Review and provide input to the verification report(s)/verification statement(s).
A2 BACKGROUND
Radioactive chemicals, such as sealed sources of Cobalt-60 and Cesium-137 oxides, can
be found in industrial, commercial and medical devices such as those used for measuring the
thickness of materials. Many such devices widely used in industrial and commercial applications
are often small in size and thus can be easily lost, stolen, abandoned, or improperly disposed.
In some instances sealed radioactive sources can be replaced by a non-radioactive source
of energy to accomplish the same function. For some uses, there exist alternative technologies
which can replace devices that use sealed sources.
Currently, radiography cameras are being used to monitor the structural integrity of
pipes and tanks in manufacturing and chemical plants, and refineries. Chemical plants,
refineries, and other manufacturing facilities are considered visible targets for terrorist attacks.
Monitoring the structural integrity of these facilities can help identify intentional damage or
potential compromises, and ensure security. However the radioactive sources in these cameras
themselves can present a safety risk. Minimizing the number of such radioactive sources in the
public domain will decrease the opportunity for terrorists to obtain these sources when they
might be inappropriately disposed. In an effort to do so, the EPA's Office of Radiation and
Indoor Air in the Office of Air and Radiation (OAR), established the EPA's Alternative
Technologies Initiative (ATI) (http://www.epa.gov/radiation/source-reduction-management/alt-
technologies.html). Part of the EPA-ATI is fostering the acceptance and voluntary market
adoption of non-radioactive technologies; i.e., alternative technologies to those that currently
use sealed sources. The EPA-ATI is focusing primarily on alternative technologies for devices
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 16 of 39
Version 1.0
Date: 5/28/2010
with Category 3 and 4 radioactive sources as classified by the International Atomic Energy
Agency (IAEA). Commercial-ready or available alternatives to radiography cameras (such as
pulsed x-ray, ultrasound, and other technologies) are being considered. As with any new
technology, the likelihood of acceptance can be significantly increased by independent
evaluation and verification of a technology's capabilities.
The purpose of this test/QA plan is to specify procedures for a verification test applicable
to commercially available alternatives to radiography cameras which can replace technologies
that use sealed radioactive sources. The purpose of the verification test is to evaluate the
performance of participating technologies in a simulated field environment. In performing the
verification test, Battelle will follow the technical and QA/QC procedures specified in this
test/QA plan and will comply with the data quality requirements in the AMS Center QMP.1
A3 VERIFICATION TEST DESCRIPTION AND SCHEDULE
A3.1 Summary of Technology Category
Radiography cameras are a major component of non-destructive testing (NDT). They are
used to inspect materials for hidden flaws using gamma-rays to penetrate the material and
provide an image of the flaws. Radiography cameras employ film cassettes to record an image
of the pipe or vessel being inspected. Iridium-192 and Cobalt-60 are the most common gamma
radiation sources used. Sealed source radiation has significant safety concerns if mishandled or
improperly disposed. Specific licensing and regulation requirements must also be met to use
these cameras, and large areas must be controlled to restrict access while the camera is in use.
The use of non-radioactive sourced alternative technologies, where applicable, could help to
eliminate these health and safety concerns. These technologies include x-ray (pulsed or high
voltage), ultrasound, and eddy current sources. This verification will be testing x-ray
technologies and their ability to conduct defect testing on pipes similar to those that might be
found in an oil and gas industry refinery.
Non-radioactive source x-ray devices can be operated more safely than sealed-source
radiography cameras and do not have the same waste concerns. However, their ability to
perform comparably to sealed-source radiography cameras in all situations is not well
characterized. Although non-radioactive source x-ray technologies have been used for decades,
isotope based radiography is still commonly used in refineries because the sources are generally
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 17 of 39
Version 1.0
Date: 5/28/2010
easier to transport and position. One particular area of interest in the capabilities of non-
radioactive source x-ray devices is their ability to detect pipeline defects through insulation.
This verification test will evaluate the ability of non-radioactive source x-ray technologies to
determine defects in a pipeline, particularly to adequately identify defects through insulation.
A3.2 Verification Test Schedule
Table 1 shows the planned schedule of testing and data analysis/reporting activities to be
conducted in this verification test. As shown in Table 1, preparation to test the technologies will
begin in April 2010. Battelle will be performing the testing. Following testing, a separate ETV
verification report will be drafted for each participating technology. The reports will be
reviewed by the technology vendor, peer reviewers selected from oil and gas trade groups or
industry, and the EPA. The final verification statement(s) will be submitted to EPA for
signature, and these documents will be made publicly available on both the EPA/ETV and the
Battelle AMS Center websites.
Table 1. Planned Verification Test Schedule
Month Year
April to May
2010
June 20 10
June to July
2010
July to
August 20 10
August 2010
September
2010
Testing Activities
• Coordinate with vendor representative
• Coordinate schedules for technologies
and reference testing
• Perform testing
Data Analysis and Reporting
• Begin preparation of ETV report template
• Compile data from all technologies
• Compile testing environment conditions
• Collect and analyze data from reference samples
• Analyze and finalize all data
• Complete common sections of reports
• Prepare draft reports
• Internal review of draft reports
• Vendor review of draft reports
• Revise draft reports
• Peer review of draft reports
• Revise draft reports
• Submit final reports for EPA approval
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 18 of 39
Version 1.0
Date: 5/28/2010
A3.3 Test Sites
Testing will be conducted at Battelle's Pipeline Facility in West Jefferson, Ohio. In
performing this verification test, Battelle will follow the procedures specified in the test/QA plan
and will comply with quality requirements in the AMS Center QMP.1
A3.4 Health and Safety
Battelle will conduct all verification testing and reference measurements following the
safety and health guidelines in place for Battelle's Pipeline Facility. This includes maintaining a
safe work environment and a current awareness of radiation exposure potential. Testing
involving the release of radiation will be performed by appropriately trained personnel under the
guidance of Battelle's radiation safety officer in the Environment, Safety, Health & Quality
(ESH&Q) Services group.
A4 QUALITY OBJECTIVES
In performing the verification test, Battelle will follow the technical and QA procedures
specified in this test/QA plan and will comply with the data quality requirements in the AMS
Center QMP.1 This verification test is designed to evaluate the performance of non-radioactive
source x-ray technologies for detecting defects in pipeline commonly used in oil and gas industry
refineries. Calibrations of x-ray technologies will follow manufacturer specified procedures and
acceptance criteria. If possible, this calibration will be performed on-site prior to testing so that
observations on the calibration process can be noted by testing staff. The verification of
technology performance will include a comparison of the technology results to results obtained
using a radiography camera. In addition, environmental factors and testing conditions, such as
weather conditions and temperature, will be documented. The Battelle QA Manager or designee
will carry out QA/QC oversight and auditing. This will include a Technical Systems Audit
(TSA) and a data quality audit. The planned audit procedures are described in Section Cl. The
EPA QA Manager also may conduct an independent TSA, at her discretion.
Data quality objectives indicate the minimum data quality required to meet the x-ray
technology verification objectives. Data quality objectives for this verification test include those
related to the performance of the reference method and x-ray technology, as well as those related
to documenting verification testing staff observations. Data quality objectives for the reference
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 19 of 39
Version 1.0
Date: 5/28/2010
method (see Section B4) are presented in terms of data quality indicator (DQI) criteria for the
critical measurements associated with the reference method and are listed in Table 5 (see Section
B5). In the field, the reference method data quality relies, in part, on proper operation of the
reference technology. The radiography camera subcontractor will follow the manufacturer's
instructions and/or any applicable standard operating procedures generated by the subcontractor
for the safe operation of the radiography camera.
Battelle will rely on the vendor's data quality objectives for each x-ray technology in
order to insure that the technology is performing properly during testing. The technology data
quality relies on proper operation and maintenance of the x-ray technologies. The results from
these technologies are expected to be qualitative and quantitative, and will be reported as either
detecting or not detecting the defect for the test conditions in the field, as well as providing
information on the characteristics of the defect (for example number of pits, and depth of pits).
A5 SPECIAL TRAINING/CERTIFICATION
Operation of each technology (radioactive and non-radioactive source) will be carried out
by a trained vendor or subcontractor representative during testing. In this scenario, the vendor
will verify that the operator is sufficiently trained to safely operate the technology.
Documentation of appropriate radiation training, licensing and safe operation approval (e.g.,
ASNT IRRSP) will be provided by both the vendor and radiography camera subcontractor to
Battelle. The Battelle verification test coordinator or assigned Battelle staff will verify the
presence of the appropriate licenses (and ensure that they are current) prior to the start of testing.
A6 DOCUMENTATION AND RECORDS
The records for this verification test will be contained in the data collection forms and
electronic files (both raw data and spreadsheets). The documents for this verification test will
consist of the test/QA plan, the final verification reports and statements, and the audit reports.
All of these documents and records will be maintained in the Verification Test Coordinator's
office or at the testing site during the test and will be transferred to permanent storage at
Battelle's Records Management Office within two months of the finalization of the verification
reports, except for audit reports, which are permanently stored with the Battelle QA Manager.
All Battelle LRBs are stored indefinitely, either by the Verification Test Coordinator or
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 20 of 39
Version 1.0
Date: 5/28/2010
Battelle's Records Management Office. EPA will be notified before disposal of any files. Table
2 has further details regarding the data recording practices and responsibilities.
All written records must be in ink. Any corrections to notebook entries, or changes in
recorded data, must be made with a single line through the original entry. The correction is then
to be entered, initialed, and dated by the person making the correction. In all cases, strict
confidentiality of data from each vendor's technology will be maintained. Separate files
(including manual records, printouts, and/or electronic data files) will be kept for each vendor's
technology.
Table 2. Summary of Data Recording Process
Data to Be
Recorded
Dates, times, and
details of test
events, technology
maintenance, down
time, ease of use,
etc.
Technology
calibration
information
Technology
readings
Reference method
analysis
procedures,
calibrations, QA,
etc.
Reference method
results
Where Recorded
ETV LRBs or data
recording forms
ETV LRBs, data
recording forms, or
electronically
Recorded
electronically by
the technology and
downloaded to an
independent
computer, or hard
copy data printed
by the technology
and taped into the
ETV LRB, or hand
entered into ETV
LRBs or data
recording forms
LRBs, or other
data recording
forms
Electronically or
manually into ETV
LRBs or data
recording forms
How Often
Recorded
Start/end of test
procedure, and at
each change of a
test parameter or
change of
technology status
At technology
calibration or
recalibration
Recorded
continuously for
electronic data,
printed after each
measurement for
hard copy print-
outs, or recorded
manually with each
reading
Throughout
reference analysis
imaging
Every image taken
By Whom
Battelle or
technology
operator
Technology
operator
Technology
operator
Subcontractors,
Battelle, or others
assisting in
reference analysis
Subcontractors,
Battelle, or other
reference analysis
technician
Disposition of Data
Used to organize
and check test
results; manually
incorporated in
data spreadsheets
as necessary
Incorporated in
verification report
as necessary
Converted to or
manually entered
into spreadsheet
for statistical
analysis and
comparisons
Retained as
documentation of
sample collection
or reference
method
performance
Transferred to
spreadsheets for
calculation of
results, and
statistical analysis
and comparisons
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 21 of 39
Version 1.0
Date: 5/28/2010
SECTION B
MEASUREMENT AND DATA ACQUISITION
Bl EXPERIMENTAL DESIGN
X-ray technologies will be tested at Battelle's Pipeline Facility. This will allow for
performance evaluation under simulated "real world" conditions. Overall, the performance of
the x-ray technologies will be verified based on the following factors:
• Detection of defects known to be on the pipe;
• Detection of appropriate number and size of components of individual defects; and
• Operational factors (ease of use, sampling time, sampling costs).
The responses to these performance factors will be collected as analog images on
phosphor imaging plates. After exposure, the imaging plate is placed into a computed
radiography scanner where the image is retrieved using laser light scanning and stored as a
digital file. The images will be assessed and evaluated by the vendor or vendor representative to
determine specific characteristics of the defect (e.g., pit depth) that will be used for analysis of
the technology. Battelle technical staff that specialize in nondestructive testing measurements
will also review the images from both the vendor and the reference instrument to confirm the
results, to the extent possible. Pipes selected for testing have specific, known defects either
manually placed or naturally occurring on the pipe. The geometry of each defect will be known
based on previous mapping of the pipe.
Radiographic testing is regulated in the state of Ohio and must be performed by licensed
operators following approved safety procedures. The evaluations will be performed according to
the vendor's approved procedures as described in the user's instructions or manual and will be
carried out by a trained and licensed operator provided by the vendor. Similarly, calibration and
maintenance of the technologies will be performed by an operator provided by the vendor. If
possible, calibration of the vendor and reference technology will be conducted on-site so that
Battelle technical staff can observe the procedure. The technologies will be evaluated based on
their ability to characterize features on simulated oil/gas pipe segments. Results from the
technologies being verified will be recorded manually by the operator on appropriate data sheets
or captured in an electronic data system, and then transferred manually or electronically for
further data analysis. The results from each technology will be reported individually. No direct
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 22 of 39
Version 1.0
Date: 5/28/2010
comparison will be made between tested technologies, but each technology will undergo the
same testing so it is convenient for end users to evaluate the ETV results.
Bl.l Test Procedures
The following describes the test procedures that will be used to evaluate non-radioactive
source x-ray technologies at the Battelle Pipeline Facility.
Two pipe samples will be examined by both the x-ray device and the radiography
camera. These pipe samples are meant to be as similar as possible to pipes that would be
encountered in a refinery. Though the test pipes are not identical to those used in refineries (i.e.,
the verification test pipes likely have thinner walls), they do have a similar radius. Refineries
typically use 4 to 12 inch diameter pipes. The pipe samples used in this test will be 6 to 8 inches
in diameter. One pipe will be low cost carbon steel, and the other a higher cost alloy steel. The
pipes are part of Battelle's Pipeline Facility where inspection technology is tested on a regular
basis. The test will be conducted outside in Battelle's pipe specimen storage yard. The pipe will
be placed on stands or timbers so that pipes are about 3 feet off the ground. To the extent
possible, the same type of x-ray detection plates will be used by both the x-ray device and the
radiography camera. The images will be qualitatively and quantitatively compared.
Pipe Sample 1 will be a seam-welded carbon steel pipe measuring approximately 35 feet
in length. The wall thickness is 0.188 inches. This sample consists of three pipe sections welded
together (two circumferential welds) and contains simulated corrosion defects set along two test
lines 180° apart. The simulated corrosion was created using electrochemical etching techniques.
A five foot section in the middle of Pipe Sample 1 also contains natural corrosion from a pipe
pulled from service. The pipe sections with the simulated corrosions were manufactured to API
specification X-52. The API grade of the pipe section with natural corrosion is not known.
While Pipe Sample 1 has over a dozen corrosion areas and three welds, a subset of the
welds and corrosion will be used to assess the x-ray technologies. This assessment will include
collecting images of a variety of features.
• Three simulated corrosion defects - two images of each defect will be taken with the
x-ray beam oriented 90 degrees to the centerline of the pipe. One image will assess
the length and width of the corrosion, and the other will assess the length and depth.
• One weld - two images will be taken 90 degrees to the centerline.
• One natural corrosion area - two images will be taken 90 degrees to the centerline.
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 23 of 39
Version 1.0
Date: 5/28/2010
The natural corrosion is close to the weld and both areas can possibly be assessed from
the same image. We plan to use three simulated corrosion defects (Pl-18, Pl-7 and Pl-23), one
natural corrosion defect (PI-9), and the weld next to this natural corrosion defect.
Pipe Sample 2 is stainless steel alloy of unknown composition measuring approximately
52 inches in length. The wall thickness is about 0.5 inch. The surface is nominally in the original
manufactured condition. Since there are no corrosion anomalies, three holes of varying diameter
and depth will be drilled into the pipe using handheld tools. The actual diameter and depth of
these defects will be determined after they are made using a micrometer and calipers. The depth
of the defects will be measured using a Starrett 449 or equivalent depth micrometer. The
diameter of the defects will be measured using a Starrett 120 or equivalent slide caliper. The
accuracy of these measurements should be within +/- 10% of the wall thickness of the Pipe
Sample 2 (0.018 inches). Accuracy will be measured to +/- 0.002 inches. Two images will be
taken of Pipe Sample 2 by the x-ray technology and the radiography camera; one to assess the
diameter of the drilled hole, and the other to assess the depth.
To simulate the refinery environment, the pipes will be insulated with calcium silicate
material. The insulation will be jacketed with either aluminum or stainless steel sheet metal.
The jackets will be held on by steel banding material.
Bl.1.3 Testing Parameters
A total of 10 images will be taken by each the x-ray technology and the reference
instrument on Pipe Samples 1 and 2. Qualitative and quantitative assessments will be made.
The images acquired using the radiography camera will be used as the reference. It is not
expected that the images from the x-ray device and the radiography camera will be identical
since small positioning differences between the source, detector and pipe as well as exposure
time will cause differences in image intensity at anomalies. The following sections describe in
detail the evaluation of the testing parameters.
Bl. 1.3.1 Detection of Defects - Qualitative Results
In general, the detection of a single defect will be determined by viewing the resulting
image(s) of the defect and assessing that the technology did indeed discover a defect in the
appointed area. The defects location, size, and shape will be known from a previous mapping of
the pipe. Whether or not the defect was under insulation will be noted. The ability of the x-ray
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 24 of 39
Version 1.0
Date: 5/28/2010
technology to detect the defect will be compared to the radiography camera's findings and the
results will be discussed.
The weld images for Pipe Sample 1 will also be compared qualitatively. The weld region
will be divided into 10 zones, isolating weld anomalies such as lack of penetration in the root
pass, undercut in the crown, slag inclusion, and porosity as well as regions of acceptable welds.
These welds are not high quality, rather they were fabricated to hold the pipe together; therefore
weld defects are expected with the potential for the entire weld to be defective. The weld would
be divided into 10 areas or zones as partially illustrated in Figure 2. Zone numbers continue
from 6 to 10 in the lower part of the arc as well. The presence or absence of defects in each zone
will be noted by the technology operator and then reported in a tabular format (see Table 3).
Both weld images will be assessed. The number and type of weld defects found by the x-ray
technology will be compared qualitatively to the number and type of weld defects found by the
radiography camera, and the results will be discussed.
Figure 2. Example of zones for assessment of Pipe Sample 1 weld.
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 25 of 39
Version 1.0
Date: 5/28/2010
Table 3. Weld Defect Assessment Table
Zone
1
2
3
4
5
6
7
8
9
10
Root
Crown
Slag
Porosity
No Defect
For evaluating the x-ray technologies' performance in detecting the natural corrosion on
Pipe Sample 1, a similar process will be used. The natural corrosion region will be divided into
10 zones. The level of corrosion will be noted by the technology operator with the qualitative
terms of light, moderate, severe, or absence of defects in each zone in a tabular format (see
Table 4). Both corrosion images will be assessed. These will be compared to actual corrosion
depth measurements that have already been made in a previous mapping of the pipe with:
• None < 10% wall loss;
• Light 10% < depth < 25 % wall loss;
• Moderate 25% < depth < 50 % wall loss; and
• Heavy > 50 % wall loss.
Table 4. Corrosion Image Assessment Table
Zone
1
2
3
4
5
6
7
8
9
10
Corrosion Level
Actual Corrosion Depth
Measurement
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 26 of 39
Version 1.0
Date: 5/28/2010
The corrosion levels found by the x-ray technology will be compared qualitatively to the
levels found by the radiography camera and the actual corrosion depth measurements. The
results will be discussed.
B1.1.3.2 Detection of Defects - Quantitative Results
Quantitative measures will be used to assess the performance of the x-ray technologies in
measuring simulated corrosion anomalies on Pipe Sample 1. The image will be used to assess
the following parameters:
1. Axial extent (length) in inches;
2. Circumferential extent (width) inches;
3. Number of pits (typically 2 or 3);
4. Axial extent of deepest pit (Pit length) in inches;
5. Circumferential extent of deepest pit (Pit width) in inches; and
6. Depth of deepest pit in inches.
The results will be tabulated for each defect (see Table 5). Actual measurements for the
parameters listed in Table 5 have already been made in a previous mapping of the pipe.
Table 5. Simulated Corrosion Defect Assessment
Patch Length (inches)
Patch Width (inches)
Number of Pits
Pit Length (inches)
Pit Width (inches)
Pit Depth (inches)
Radiography
X-Ray
Actual
For the drilled holes in the stainless steel Pipe Sample 2, the depth and diameter of each
defect will be assessed (see Table 6). Actual measurements for the parameters listed in Table 6
will be taken using a micrometer and calipers, as discussed in Section Bl.l.
A percentage error will be calculated for each measure in Tables 5 and 6. An error of+/-
10% of the wall thickness is generally accepted in the industry corrosion assessment. A percent
difference will also be calculated between the results from the radiography camera and the x-ray
technology for the specific corrosion and defect measurements listed in Tables 5 and 6 to
compare the performance of the x-ray technology to that of the reference instrument results.
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 27 of 39
Version 1.0
Date: 5/28/2010
Table 6. Drill Hole Defect Assessment
Pit 1 Diameter (inches)
Pit 1 Depth (inches)
Pit 2 Diameter (inches)
Pit 2 Depth (inches)
Pit 3 Diameter (inches)
Pit 3 Depth (inches)
Radiography
X-Ray
Actual
B1.1.3.3 Operational Factors
Operational factors such as maintenance needs, power needs, calibration frequency, data
output, consumables used, ease of use, repair requirements, training and certification
requirements, safety requirements, and image throughput will be evaluated based on testing
observations and input provided from the vendor. Input will either be provided by the vendor
on-site during the verification test and be recorded by Battelle staff or will be provided in
documentation to the Verification Test Coordinator after completion of the verification test. To
the extent possible, Battelle technical staff will also observe and record their own observations of
these operational factors. Examples of information to be recorded include the daily status of
diagnostic indicators for the technology, use or replacement of any consumables, use and nature
of power supply needed to operate the technology, the effort or cost associated with maintenance
or repair, vendor effort (e.g., time on site) for repair or maintenance, the duration and causes of
any technology down time or data acquisition failure, observations about technology startup,
ease of use, clarity of the user's instruction manual, user-friendliness of any needed software,
overall convenience of the technologies and accessories/consumables, the safety hazard
associated with the use of the technology, or the number of images that could be taken and
processed per hour or per day. These observations will be summarized to aid in describing the
technology performance in the verification report on each technology.
B1.2 Statistical Analysis
The statistical methods and calculations used for evaluating quantitative performance
parameters are described in the following sections.
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 28 of 39
Version 1.0
Date: 5/28/2010
B1.2.1 Percent Error
The quantitative results will be assessed by calculating the percent error between the
actual and measured defect characteristics. The pipeline industry typically normalizes the error
to the wall thickness of the pipe rather than the actual reading5. This method is useful since
small defects are not as important, but small errors on small defects can lead to large and
misleading errors percentages when actual depths are used as the normalizing factor. Percent
error will be calculated using the following equation:
Estimate - Actual ,..,
%Error = x 100 (1)
Wall Thickness
Bl.2.2 Percent Difference
The quantitative results will also be assessed by calculating the percent difference
between the measurements made by the x-ray technologies and the radiography camera. This
evaluation, in conjunction with the qualitative parameters, will help in assessing the performance
of the x-ray technology in relation to that of the reference instrument results. Percent difference
will be calculated using the following:
„.„-,. (X - Ray Technology Result - Radiography Camera Result] . .
^Difference = ± — '- x 100 (2)
Radiography Camera Result
B1.3 Reporting
The data obtained in the verification test will be compiled separately for each vendor's
technology, and the data evaluations will be applied to each technology's data set without
reference to any other. At no time will data from different vendor's technology be compared or
ranked. Following completion of the data evaluations, a draft verification report and verification
statement will be prepared for each vendor's technology, stating the verification test procedures
and documenting the performance observed. For example, descriptions of the data acquisition
procedures, use of vendor supplied proprietary software, consumables used, repairs and
maintenance needed, and the nature of any problems will be presented in the draft report. Each
report will briefly describe the ETV Program, the AMS Center, and the procedures used in
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 29 of 39
Version 1.0
Date: 5/28/2010
verification testing. The results of the verification test will be stated quantitatively, without
comparison to any other technology tested, or comment on the acceptability of the technology's
performance. Each draft verification report will be submitted for review by the respective
technology vendor and by EPA and other peer reviewers. Comments on the draft report will be
addressed in revisions of the report. The peer review comments and responses will be tabulated
to document the peer review process. The reporting and review process will be conducted
according to the requirements of the AMS Center QMP.1
B2 SAMPLING REQUIREMENTS
B2.1 Sample Collection, Storage and Shipment
Samples in the form of pipe defects in a field environment will be detected by the x-ray
technologies in real time. The reference method will collect information on the same defects.
All pipe defects will be analyzed on-site. No samples will be collected, stored, or shipped as part
of this verification test.
B3 SAMPLE HANDLING AND CUSTODY REQUIREMENTS
No samples will be collected, stored, or shipped as part of this verification test.
B4 REFERENCE METHOD
A radiography camera will be used as the reference method for this test. The radiography
camera will be used to collect images of the same pipeline defects that the x-ray technology will
evaluate. This radiography camera will be part of a computed radiography system that will
provide digital images to Battelle. The radiography camera will be operated by a licensed and
properly certified radiographer that will meet all appropriate state safety requirements for
operation of a sealed-source technology. Battelle will hire a licensed subcontractor to conduct
the reference analyses with the radiography camera. It is expected that film will be used to
collect the image. Efforts will be made to use plates similar to those being used by the x-ray
technology.
The radiography camera operator will follow the specified operation of the particular
radiography camera used per the instruction manual for that camera. Interpretations of the
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 30 of 39
Version 1.0
Date: 5/28/2010
images from the reference analyses will be provided to Battelle in either digital or hardcopy
form. The images taken of the defects using the reference method will also be provided to
Battelle by the subcontractor. The QA/QC requirements for the documentation and performance
of the analytical method are described as data quality indicators (DQI) in Section B5.
B5 QUALITY CONTROL CRITERIA FOR REFERENCE METHOD
MEASUREMENT DATA
Table 7 presents the DQIs and criteria for the reference method measurements. The
reference method measurement quality will be assured by adherence to these DQI criteria.
Prior to starting the reference sampling, the radiography camera will be calibrated
according to the manufacturer's specified procedure, if applicable. On each day of testing the
device will be calibrated, if applicable.
Table 7. DQIs and Criteria of Critical Measurements for Reference Method
DQI
Confirmation of
Detected Defects
Image Quality
Method of
Assessment
Identification of
defect in appropriate
location and of
appropriate size in
accordance with
known mapping of
defect
Image Quality
Indicator (IQI) or
comparator, as
applicable
Frequency
All field test
samples
Each image
Minimum
Acceptance
Criteria
N/A
Sensitivity of 2% or 2-
2T is met, the correct
penetrameter hole/wire
is discernable, or
meets applicable and
accepted IQI
standards, such as
ASTME-943 or those
specified in the
radiography camera
subcontractors SOP
Corrective Action
Data considered suspect
and reanalyzed
Re-image feature
B6 INSTRUMENT/EQUIPMENT TESTING, INSPECTION, AND MAINTENANCE
The reference equipment, micrometer, and calipers used in this test will be tested,
inspected, and maintained as per the manufacturer's recommendations so as to meet the
performance requirements established in this document.
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 31 of 39
Version 1.0
Date: 5/28/2010
B7 INSTRUMENT CALIBRATION AND FREQUENCY
Prior to start of the reference sampling, the radiography camera, micrometer, and calipers
will be calibrated according to the manufacturer's specified procedure, as applicable. The
micrometer and calipers are generally calibrated at least annually. Prior calibration within a year
before testing will satisfy the calibration requirement for these instruments. The participating x-
ray technology will be calibrated by the operator or vendor according to the technology's
specified procedures. If possible, this calibration will be performed on-site prior to testing.
B8 INSPECTION/ACCEPTANCE OF SUPPLIES AND CONSUMABLES
All materials, supplies, and consumables will be ordered by the Verification Test
Coordinator or designee. Where possible, Battelle will rely on sources of materials and
consumables that have been used previously as part of ETV verification testing without
problems. Battelle will also rely on previous experience or recommendations from EPA
advisors, or the vendor.
B9 NON-DIRECT MEASUREMENTS
No non-direct measurements will be used during this verification test.
BIO DATA MANAGEMENT
Various types of data will be acquired and recorded electronically or manually by
Battelle during the verification test. Table 2 summarizes the types of data to be recorded. All
maintenance activities, repairs, calibrations, and operator observations relevant to the technology
operation will be documented by technical staff in LRBs or on data sheets. Results from the
reference method, including raw data, analyses, and final results, will be compiled by Battelle.
Records received by or generated by any technical staff during the verification test will
be reviewed by a Battelle staff member within two weeks of generation or receipt, before the
records are used to calculate, evaluate, or report verification results. If a Battelle staff member
generated the record, this review will be performed by a Battelle technical staff member involved
in the verification test, but not the staff member who originally generated the record. The review
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 32 of 39
Version 1.0
Date: 5/28/2010
will be documented by the designated person by adding his/her initials and date to the hard copy
of the record being reviewed. A technical review of 100% of the test and reference data produced
will be conducted. In addition, any calculations performed by technical staff will be checked by
Battelle QA and/or technical staff to ensure that calculations are performed correctly.
Calculations to be checked include any statistical calculations described in this test/QA plan. The
data obtained from this verification test will be compiled and reported independently for each
technology. Results for technologies from different vendors will not be compared with each
other.
Among the QA activities conducted by Battelle QA staff will be an audit of data quality.
This audit will consist of a review by the Battelle QA Manager (or his designee) of at least 25%
of the test data. During the course of any such audit, the Battelle QA Manager will inform the
technical staff of any findings and any need for immediate corrective action. If serious data
quality problems exist, the Battelle QA Manager will request that Battelle's AMS Center
Manager issue a stop-work order. Once the assessment report has been prepared, the Verification
Test Coordinator will ensure that a response is provided for each adverse finding or potential
problem, and will implement any necessary follow-up corrective action. The Battelle QA
Manager will ensure that a follow-up corrective action has been taken.
Data obtained during the verification test will be maintained confidentially at Battelle,
and used only for purposes of the technology verification. Data reporting in the final report will
consist of tabular results of the calculations in Section B.
It is anticipated that testing of an x-ray technology will take place in one day. As such,
Battelle will provide technology test data and associated reference data (including records; data
sheets; notebook records) within 2 weeks of generation to EPA. The goal of this data delivery
schedule is prompt identification and resolution of any data collection or recording issues.
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 33 of 39
Version 1.0
Date: 5/28/2010
SECTION C
ASSESSMENT AND OVERSIGHT
Cl ASSESSMENTS AND RESPONSE ACTIONS
Every effort will be made in this verification test to anticipate and resolve potential
problems before the quality of performance is compromised. One of the major objectives of this
test/QA plan is to establish mechanisms necessary to ensure this. The procedures described in
this test/QA plan, which is peer reviewed by a panel of outside experts, implemented by the
technical staff and monitored by the Verification Test Coordinator, will give information on data
quality on a day-to-day basis. The responsibility for interpreting the results of these checks and
resolving any potential problems resides with the Verification Test Coordinator. Technical staff
have the responsibility to identify problems that could affect data quality or the ability to use the
data. Any problems that are identified will be reported to the Verification Test Coordinator, who
will work with the Battelle QA Manager to resolve any issues. Action will be taken to control
the problem, identify a solution to the problem, and minimize losses and correct data, where
possible. Independent of any EPA QA activities, Battelle will be responsible for ensuring that
the following audits are conducted as part of this verification test.
Any changes to the approved test/QA plan must be reported within 24 hours and
documented in a formal deviation submitted to the Battelle AMS Center Manager, EPA AMS
Center Project Officer, and EPA AMS Center Quality Manager. If approval by EPA or its
designee is not received within 24 hours of notification, testing will be halted until a suitable
resolution has been achieved.
Cl.l Performance Evaluation Audits
Because of the nature of the samples to be evaluated in this verification test (i.e., defects
on a pipe), a Performance Evaluation (PE) audit will not be conducted as PE audit samples are
not available.
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 34 of 39
Version 1.0
Date: 5/28/2010
C1.2 Technical Systems Audits
The Battelle QA Manager will perform a technical systems audit (TSA) at least once
during this verification test. The purpose of this audit is to ensure that the verification test is
being performed in accordance with the AMS Center QMP,1 this test/QA plan, and any Standard
Operating Procedures (SOPs) used by Battelle. In the TSA, the Battelle QA Manager or a
designee may review the reference method used, compare actual test procedures to those
specified or referenced in this plan, and review data acquisition and handling procedures. The
Battelle QA Manager will tour the test site, observe and review the test procedures, and review
record books. He will also check calibration certifications for test measurement devices. A draft
TSA will be prepared within 2 weeks of performance of the TSA and sent to EPA. The final
TSA report will be prepared, include a statement of findings and the actions taken to address any
adverse findings. At EPA's discretion, EPA QA staff may also conduct an independent on-site
TSA during the verification test. The TSA findings will be communicated to technical staff at
the time of the audit and documented in a TSA report.
C1.3 Data Quality Audits
The Battelle QA Manager will audit at least 25% of the verification data acquired in the
verification test. The Battelle QA Manager will trace the data from initial acquisition, through
reduction and statistical comparisons, to final reporting. All calculations performed on the data
undergoing the audit will be checked.
C1.4 QA/QC Reporting
Each audit will be documented in accordance with Sections 3.3.4 and 3.3.5 of the AMS
Center QMP.1 The results of the audits (both TSA and ADQ) will be submitted to EPA. Audit
reports may include the following:
• Identification of any adverse findings or potential problems;
• Response to adverse findings or potential problems;
• Recommendations for resolving problems; and
• Citation of any noteworthy practices that may be of use to others.
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 35 of 39
Version 1.0
Date: 5/28/2010
C2 REPORTS TO MANAGEMENT
The Battelle QA Manager, during the course of any audit, will identify to the technical
staff performing experimental activities any immediate corrective action that should be taken. If
serious quality problems exist, the Battelle QA Manager is authorized to request that Battelle's
AMS Center Manager issue a stop work order. Once the audit report has been prepared, the
Verification Test Coordinator will ensure that a response is provided for each adverse finding or
potential problem and will implement any necessary follow-up corrective action. The Battelle
QA Manager will ensure that follow-up corrective action has been taken. The test/QA plan and
final report are reviewed by EPA AMS Center QA staff and EPA AMS Center program
management staff. Upon final review and approval, both documents will then be posted on the
ETV website (www.epa.gov/etv).
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 36 of 39
Version 1.0
Date: 5/28/2010
SECTION D
DATA VALIDATION AND USABILITY
Dl DATA REVIEW, VALIDATION, AND VERIFICATION REQUIREMENTS
The key data review requirements for the verification test are stated in Section BIO of
this test/QA plan. In general, the data review requirements specify that the data generated during
this test will be reviewed by a Battelle technical staff member within two weeks of data
generation. The reviewer will be familiar with the technical aspects of the verification test, but
will not be the person who generated the data. This process will serve both as the data review
and the data verification, and will ensure that data have been recorded, transmitted, and
processed properly.
The data validation requirements for this test involve a data quality audit relative to the
DQIs and audit acceptance criteria specified for this test. The DQIs listed in Section B5 will be
used to validate the data quality. The QA audits described within Section C of this document,
including the performance evaluation audit and data quality audit, are designed to validate the
data quality.
D2 VALIDATION AND VERIFICATION METHODS
As part of the normal data and report review process the US EPA will have the
opportunity to review the draft final report and provide comments. Data verification is
conducted as part of the data review, as described in Section BIO for this test/QA plan. A visual
inspection of handwritten data will be conducted to ensure that all entries were properly recorded
or transcribed and that any erroneous entries were properly noted (i.e., single line through the
entry with an error code and the initials of the recorder and date of entry). Electronic data from
the technologies and other instruments used during the test will be inspected to ensure proper
transfer from the data logging system. Data manually incorporated into spreadsheets for use in
calculations will be checked against handwritten data to ensure that transcription errors have not
occurred. All calculations used to transform the data will be reviewed to ensure the accuracy and
the appropriateness of the calculations. Calculations performed manually will be reviewed and
repeated using a handheld calculator or commercial software (e.g., Excel). Calculations
performed using standard commercial office software (e.g., Excel) will be reviewed by
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 37 of 39
Version 1.0
Date: 5/28/2010
inspecting the equations used in calculations and verifying selected calculations by handheld
calculator. Calculations performed using specialized commercial software (i.e., for analytical
instrumentation) will be reviewed by inspection and, when feasible, verified by handheld
calculator, or standard commercial office software.
To ensure that the data generated from this test meet the goals of the test, a number of
data validation procedures will be performed. Section C of this test/QA plan provides a
description of the validation safeguards employed for this verification test. Data validation and
verification efforts include the completion of QC activities and the performance of ISA as
described in Section C. The data from this test will be evaluated relative to the measurement
DQIs described in Section A4 and B5 of this test/QA plan. Data failing to meet these criteria will
be flagged in the data set and not used for evaluation of the technologies, unless these deviations
are accompanied by descriptions of their potential impacts on the data quality.
A data quality audit will be conducted by the Battelle QA Manager to ensure that data
review, verification, and validation procedures were completed, and to assure the overall data
quality.
D3 RECONCILIATION WITH USER REQUIREMENTS
The purpose of a verification test performed following this test/QA plan is to evaluate the
performance of commercial technologies which detect defects in pipelines used in the oil and gas
industry in a simulated field environment. This test evaluates the non-radioactive source x-ray
technology capabilities. This evaluation will include comparisons of the results from the
technologies to results from the standard reference technique, radiography cameras. To meet the
requirements of the user community, the data obtained in such a verification test will include
thorough documentation of the technology's performance during the verification test. The data
review, verification, and validation procedures described above will assure that verification test
data meet these requirements, are accurately presented in the verification reports generated from
the test, and that data not meeting these requirements are appropriately flagged and discussed in
the verification reports. Additionally, all data generated using the reference method, which are
used to evaluate technology results during the verification test, should meet the QA requirements
of any applicable standard operating procedures or instrumentation instruction manuals.
This test/QA plan and any resulting ETV verification report(s) generated following
procedures described in this test/QA plan will be subjected to review by participating technology
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 38 of 39
Version 1.0
Date: 5/28/2010
vendors, ETV AMS Center staff, test collaborators, EPA, and external expert peer reviewers.
These reviews will assure that this test/QA plan, verification test(s) of participating technologies,
and the resulting report(s) meet the needs of potential users and regulators. The final report(s)
will be submitted to EPA in 508 compliant Adobe Portable Document Format (pdf) and
subsequently posted on the ETV website.
-------
Verification of Alternative Technologies for Sealed Source Radiography Cameras Test/QA Plan
Page 39 of 39
Version 1.0
Date: 5/28/2010
SECTION E
REFERENCES
El REFERENCES
1. Quality Management Plan for the ETV Advanced Monitoring Systems Center, Version
7.0, U.S. EPA Environmental Technology Verification Program, Battelle, Columbus,
Ohio, November 2008.
2. Environmental Technology Verification Program Quality Management Plan, EPA/600/R-
08/009, U.S. Environmental Protection Agency, Cincinnati, Ohio, January 2008.
3. ASTM Standard E-94, "Standard Practice for Radiographic Testing", ASTM
International, West Conshohocken, PA, 2009.
4. American Petroleum Institute Standard 1163, "In-line Inspection Systems Qualification
Standard, Edition 5", API Publications, Englewood, CO, August 2005.
------- |