Batreiie
The Business of Innovation
Environmental Technology
Verification Program
Advanced Monitoring Systems Center
Quality Assurance Project Plan
for Verification of
Picometrix, LLC T-Ray 4000® Time-
Domain Terahertz System
-------
Quality Assurance Project Plan
For
Verification of
Picometrix, LLC T-Ray 4000R Time-Domain Terahertz System
February 23, 2011
Prepared by
Battelle
505 King Avenue
Columbus, OH 43201-2693
-------
Verification of Picometrix THz System
QAPP
Page 3 of 52
Version 1.0
Date: 2/23/11
SECTION A
PROJECT MANAGEMENT
Al VENDOR APPROVAL PAGE
ETV Advanced Monitoring Systems Center
Quality Assurance Project Plan for Verification of
Picometrix, LLC T-Ray 4000® Time-Domain Terahertz System
February 23, 2011
APPROVAL:
Name
Company
Date
-------
Verification of Picometrix THz System
QAPP
Page 4 of 52
Version 1.0
Date: 2/23/11
A2 TABLE OF CONTENTS
Section
SECTION A PROJECT MANAGEMENT 3
Al Vendor Approval Page 3
A2 Table of Contents 4
A3 Acronyms and Abbreviations 6
A4 Distribution List 7
A5 Verification Test Organization 8
A6 Background 15
A7 Verification Test Description and Schedule 20
A8 Quality Objectives and Criteria For Measurement Data 22
A9 Special Training/Certification 24
A10 Documentation and Records 25
SECTION B MEASUREMENT AND DATA ACQUISITION 27
Bl Experimental Design 27
B2 Sampling Requirements 41
B3 Sample Handling and Custody Requirements 41
B4 Reference Method 42
B5 Quality Control Requirements 42
B6 Instrument/Equipment Testing, Inspection, and Maintenance 42
B7 Instrument Calibration and Frequency 43
B8 Inspection/Acceptance of Supplies and Consumables 43
B9 Non-Direct Measurements 43
BIO Data management 44
SECTION C ASSESSMENT AND OVERSIGHT 45
Cl Assessments and Response Actions 45
C2 Reports to Management 47
SECTION D DATA VALIDATION AND USABILITY 49
Dl Data Review, Verification, and Validation Requirements 49
D2 Verification and Validation Methods 49
D3 Reconciliation with User Requirements 50
SECTION E REFERENCES 52
El References 52
-------
Verification of Picometrix THz System
QAPP
Page 5 of 52
Version 1.0
Date: 2/23/11
List of Figures
Figure 1. Organization Chart for the Verification Test (dotted lines indicate lines of
communication) 9
Figure 2A. Collinear THz Reflection Sensor 19
Figure 2B. Illustration of Pulse's Origins 19
Figure 3. Relative Position of Sample Sheet, Nuclear (Beta) Gauge and THz Sensor 28
Figure 4. Illustration of Laboratory Sample Sectioning to Achieve Spatial Sampling of Test
Sheets 39
List of Tables
Table 1. Planned Verification Test Schedule 21
Table 2. DQI and Criteria for Critical Supporting Measurements 23
Table3. Summary of Data Recording Process 26
Table 4. Summary of Assessment Reports 48
-------
Verification of Picometrix THz System
QAPP
Page 6 of 52
Version 1.0
Date: 2/23/11
A3 ACRONYMS AND ABBREVIATIONS
ADQ
AMS
ATI
DQI
EM
EPA
ERS
ETV
IAEA
gsm
LRB
mm
OAR
pdf
ps
QA
QAPP
QC
QMP
RI
SOP
TD-THz
THz
ToF
TSA
VTC
Audit of Data Quality
Advanced Monitoring Systems
Alternative Technologies Initiative
data quality indicator
electromagnetic
U.S. Environmental Protection Agency
External Reference Structure
Environmental Technology Verification
International Atomic Energy Agency
grams per square meter
laboratory record book (the test logbook)
millimeter = 10~3 meter
Office of Air and Radiation
Adobe portable document format
1 9
picosecond = 10" second
quality assurance
quality assurance project plan
quality control
quality management plan
Refractive Index
Standard Operating Procedure
Time-Domain Terahertz
terahertz
Time-of-Flight
Technical Systems Audit
microwatt = 10~6 watt
Verification Test Coordinator
-------
Verification of Picometrix THz System
QAPP
Page 7 of 52
Version 1.0
Date: 2/23/11
A4 DISTRIBUTION LIST
Vendor
Picometrix
John J. Riccardi
1355 Lynnfield Road #245
Memphis, TN 38119
EPA QA Manager
Michelle Henderson (or qualified alternate)
U.S. Environmental Protection Agency
26 West Martin Luther King Dr.
MS 208
Cincinnati, OH 45268
EPA Project Officer
John McKernan
U.S. Environmental Protection Agency
26 West Martin Luther King Dr.
MS 212
Cincinnati, OH 45268
Peer Reviewers
Madeleine Nawar
U.S. EPA/OAR
Mail Code 6608J
1200 Pennsylvania Ave, NW
Washington, DC 20460
Michael Barlament
Kimberly-Clark Corp.
1400 Holcomb Bridge Road
Bldg.300
Roswell, GA 30076
Paul Thomas
3M Company
PI&CS Engineering
Bldg. 275-6E-21
3M Center
St. Paul, MN 55144
Temeka Taplin
U.S. Department of Energy
NA-211/Forrestal Building
1000 Independence Ave., S.W.
Washington, D.C. 20585
Collaborator
Mike Friese
Appleton Paper
1000 North Lawe Street
Appleton, WI 54912-0359
Battelle
DawnDeojay
Amy Dindal
Stephanie Buehler
Rosanna Buhl
Battelle
505 King Ave.
Columbus, OH 43201
-------
Verification of Picometrix THz System
QAPP
Page 8 of 52
Version 1.0
Date: 2/23/11
A5 VERIFICATION TEST ORGANIZATION
The verification test will be conducted under the U.S. Environmental Protection Agency
(EPA) Environmental Technology Verification (ETV) Program. It will be performed by Battelle,
which is managing the ETV Advanced Monitoring Systems (AMS) Center through a cooperative
agreement with EPA. The scope of the AMS Center covers verification of monitoring
technologies for contaminants and natural species in air, water, and soil.
The day to day operations of this verification test will be coordinated and supervised by
Battelle, with the participation of the vendor (Picometrix) who will be having the performance of
their technology (Picometrix, LLC T-Ray 4000® Time-Domain Terahertz System) verified. The
Picometrix technology offers an alternative to using sealed radioactive source nuclear gauges.
Testing will be conducted at Appleton Paper in Appleton, Wisconsin. The vendor will provide
and operate their technology.
The organization chart in Figure 1 identifies the roles and communication structure of the
organizations and individuals associated with the verification test. Roles and responsibilities are
defined further below. Quality Assurance (QA) oversight will be provided by the Battelle
Quality Manager and also by the EPA AMS Center Quality Manager, at her discretion. This
verification test is a Quality Category II test which requires a QA review of 25% of the test data
(see SectionCl).
-------
Verification of Picometrix THz System
QAPP
Page 9 of 52
Version 1.0
Date: 2/23/11
Battelle
Management
Amy Dindal
Battelle AMS
Center
Manager/Verification
Testing Leader
Dr. John McKernan
EPA AMS Center
Project Officer
Rosanna Buhl
Battelle
Quality Manager
EPA AMS Center
Quality Manager
Dr. Stephanie Buehler
Battelle
Verification
Test Coordinator
Picometrix
Representatives
Test Execution
(Battelle Technical Staff
and
Appleton Paper Staff)
Figure 1. Organization Chart for the Verification Test (dotted lines indicate lines of
communication)
-------
Verification of Picometrix THz System
QAPP
Page 10 of 52
Version 1.0
Date: 2/23/11
A5.1 Battelle
Dr. Stephanie Buehler is the AMS Center's Verification Test Coordinator (VTC) for this
test. In this role, Dr. Buehler will have overall responsibility for ensuring that the technical,
schedule, and cost goals established for the verification test are met. Specifically, Dr. Buehler
will:
• Prepare the quality assurance project plan (QAPP), verification report, and
verification statement;
• Revise the QAPP, verification report, and verification statement in response to
reviewers' comments;
• Establish a budget for the verification test and manage staff to ensure the budget is
not exceeded;
• Assemble a team of qualified technical staff to conduct the verification test;
• Direct the team in performing the verification test in accordance with this QAPP;
• Hold a kick-off meeting approximately one week prior to the start of the verification
test to review the critical logistical, technical, and administrative aspects of the
verification test. Responsibility for each aspect of the verification test will be
confirmed;
• Ensure that all quality procedures specified in this QAPP and in the AMS Center
Quality Management Plan1 (QMP) are followed;
• Serve as the primary point of contact for vendor representatives;
• Ensure that confidentiality of sensitive vendor information is maintained.
• Assist vendors as needed during verification testing;
• Become familiar with the operation and maintenance of the technologies through
instruction by the vendors, if needed;
• Respond to QAPP deviations and any issues raised in assessment reports, audits, or
from test staff observations, and institute corrective action as necessary; and
• Coordinate distribution of the final QAPP, verification report, and verification
statement.
-------
Verification of Picometrix THz System
QAPP
Page 11 of 52
Version 1.0
Date: 2/23/11
Ms. Amy Dindal will serve as Verification Testing Leader and is also Battelle's Manager
for the AMS Center. Ms. Dindal will:
• Support Dr. Buehler in preparing the QAPP and organizing the testing;
• Review the final QAPP;
• Attend the verification test kick-off meeting;
• Review the draft and final verification report and verification statement;
• Ensure that necessary Battelle resources, including staff and facilities, are committed
to the verification test;
• Ensure that confidentiality of sensitive vendor information is maintained;
• Support Dr. Buehler in responding to any issues raised in assessment reports and
audits.
• Maintain communication with EPA's technical and quality managers; and
• Issue a stop work order if Battelle or EPA QA staff discover adverse findings that
will compromise test results.
Battelle Technical Staff will support Dr. Buehler in planning and conducting the
verification test. The responsibilities of the technical staff will be to:
• Assist in planning for the test;
• Attend the verification test kick-off meeting;
• Assist vendor staff as needed during verification testing1;
• Conduct and observe verification testing on-site, as appropriate. Immediately report
deviations to this QAPP to the VTC1;
• Coordinate and observe reference testing on-site, as appropriate. Immediately report
deviations to this QAPP to the VTC1;
• Perform statistical calculations specified in this QAPP on the technology data as
needed;
• Provide results of statistical calculations and associated discussion for the verification
report as needed; and
• Support Dr. Buehler in responding to any issues raised in assessment reports and
audits related to statistics and data reduction as needed.
Battelle's Quality Manager will perform these activities.
-------
Verification of Picometrix THz System
QAPP
Page 12 of 52
Version 1.0
Date: 2/23/11
Ms. Rosanna Buhl is Battelle's Quality Manager. Ms. Buhl will:
• Review the draft and final QAPP;
• Attend the verification test kick-off meeting;
• Be present throughout testing to observe tests procedures;
• Conduct a technical systems audit during the verification test, or designate other QA
staff to conduct the audit;
• Audit at least 25% of the reported verification data or designate other QA staff to
conduct the data audit;
• Prepare and distribute an assessment report for each audit;
• Verify implementation of any necessary corrective action;
• Request that Battelle' s AMS Center Manager issue a stop work order if audits
indicate that data quality is being compromised;
• Provide a summary of the QA/QC activities and results for the verification reports;
and
• Review the draft and final verification report and verification statement.
A5.2 Picometrix
The responsibilities of the technology vendor are as follows:
• Provide significant input into the draft QAPP;
• Review and provide comments on the draft QAPP;
• Accept (by signature of a company representative) the final QAPP prior to test
initiation;
• Provide one unit with two measurement sensors of their technology for evaluation
during the verification test;
• Provide all equipment/supplies/reagents/consumables needed to operate their
technology for the duration of the verification test;
• Provide an appropriately trained person to operate their technology for the duration of
the verification test;
• Provide maintenance and repair support for their technology, on-site if necessary,
throughout the duration of the verification test;
-------
Verification of Picometrix THz System
QAPP
Page 13 of 52
Version 1.0
Date: 2/23/11
• Provide data from testing in an understandable format from their technology to the
Battelle VTC or Battelle technical staff;
• Review and provide comments on the draft verification report and statement for their
technology; and
• Provide any applicable documentation related to testing of the vendor's technology,
including calibration, testing results and observations, final images, supply or reagent
certificate of authenticity (COA), etc.
A5.3 EPA
EPA's responsibilities in the AMS Center are based on the requirements stated in the
"Environmental Technology Verification Program Quality Management Plan" (EPA ETV
9 _
QMP). The roles of specific EPA staff are as follows:
Ms. Michelle Henderson (or a qualified alternate) will act as the EPA's AMS Center QA
Manager. For the verification test, the EPA AMS Center QA Manager will:
• Review the draft and approve the final QAPP;
• Attend the verification kick-off meeting, as available;
• Review checklists, reports, report responses, and closure statements of TSA,
performance evaluation (PE) audits, and audits of data quality systems (ADQs)
conducted by Battelle;
• Perform an external TSA of field and/or laboratory activities, PE audits, and/or an
audit of data quality during the verification test;
• Notify the EPA AMS Center Project Officer of the need for a stop work order if
evidence indicates that data quality is being compromised;
• Prepare and distribute an assessment report summarizing results of any external audit
performed;
• Review the first day of data from the verification test and provide immediate
comments if concerns are identified; and
• Review the draft and approve the final verification report and verification statement.
Dr. John McKernan is EPA's Project Officer for the AMS Center. Dr. McKernan will:
• Review the draft QAPP;
-------
Verification of Picometrix THz System
QAPP
Page 14 of 52
Version 1.0
Date: 2/23/11
• Approve the final QAPP;
• Attend the verification kick-off meeting, as available;
• Be available during the verification test to review and authorize any QAPP deviations
by phone and provide the name of a delegate to the Battelle AMS Center Manager
should he not be available during the testing period;
• Review the draft verification report and verification statement;
• Oversee the EPA review process for the QAPP, verification report, and verification
statement;
• Coordinate the submission of the verification report and verification statement for
final EPA approval; and
• Post the QAPP, verification report, and verification statement on the ETV web site.
A5.4 Nuclear Gauge and Laboratory Reference Measurements
This test will require the use of a sealed source nuclear gauge and laboratory methods
based on TAPPI standards as the reference instrument. Appleton Paper (Appleton), which
routinely uses nuclear gauge technology and TAPPI laboratory measurements for their business
and which maintains an appropriate safety license to operate a nuclear gauge, will conduct all
reference measurements. The responsibilities of Appleton include the following:
• Provide a site and associated site equipment for conducting a test at an Appleton
facility;
• Conduct all field reference measurements of materials using a nuclear gauge;
• Conduct all laboratory reference measurements of materials generated during the
testing process, as specified in Appleton internal protocols;
• Supply all necessary equipment to obtain reference measurements using a nuclear
gauge and laboratory protocols;
• Supply a licensed technician to operate the nuclear gauge and staff to conduct the
laboratory measurements;
• Provide nuclear gauge as well as laboratory measurements resulting from this
verification test to Battelle, including an accompanying description or analysis of the
measurements as appropriate, including any QA/QC procedures that were followed;
• Perform testing activities and data acquisition as specified in this QAPP; and
-------
Verification of Picometrix THz System
QAPP
Page 15 of 52
Version 1.0
Date: 2/23/11
Allow the Battelle and/or EPA Quality Managers to perform a technical systems audit
at their facility.
A5.5 Verification Test Stakeholders
This QAPP and the verification report and verification statement based on testing
described in this document will be reviewed by experts in the fields related to nuclear gauges and
alternatives to sealed-source technologies. The following experts have been providing input to
this QAPP and have agreed to provide a peer review.
• Michael Barlament, Kimberly-Clark Corporation
• Paul Thomas, 3M Company
• Temeka Taplin, U. S. DOE
• Madeleine Nawar, U.S. EPA
The responsibilities of verification test stakeholders include the following.
• Review and provide input to the QAPP;
• Review and provide input to the verification report/verification statement.
A6 BACKGROUND
A6.1 Technology Need
Radioisotopes, such as sealed sources of Strontium-90, Krypton-85 or Promethium-147,
can be found in industrial devices used to measure the mass per unit area (basis weight) of
various manufactured products such as sheet metal, textiles, paper, photographic film, and other
pressed or sheet (flat) material. Devices widely used in industrial and commercial applications
are often small in size, and could potentially be lost, stolen, abandoned, or improperly disposed.
In some instances sealed radioactive sources can be replaced by an alternative non-radioactive
source of energy to accomplish the same function.
Currently, nuclear gauges are being used to measure or control material density, flow,
level, thickness or basis weight. The gauges contain sealed sources that radiate through the
substance being measured to a readout or controlling device. The radiation that is emitted from
the gauge is attenuated by the matter between the radioisotope and the detector. The radiation
reduction can be correlated to the quantity of matter between the source and the detector. The
-------
Verification of Picometrix THz System
QAPP
Page 16 of 52
Version 1.0
Date: 2/23/11
nuclear radiation is attenuated by the number of atoms between the source and the detector. The
physical measurement is the magnitude of radiation that passes through the material. The actual
amplitude of the detected signal must be compensated for a number of factors (e.g. moisture, air
in the intervening space, geometry of the source and detector); however these factors are well
understood and implemented in commercial nuclear gauges.
Nuclear gauges have been widely used (since the 1940's) for the measurement of basis
weight and quantities that can be derived from basis weight (e.g. thickness, coat weight) because
of their ability to penetrate most materials, their long life (depending on isotope) and their low
cost. At this point in time they are a well established, low risk gauge that is well known in the
industry. However, the radioactive sources in these gauges can present a safety risk. Minimizing
the number of radioactive sources in the public domain will decrease the opportunity for
individuals to use the radioisotopes for unintended, harmful (i.e. terrorist) purposes. In an effort
to do so, the EPA's Office of Radiation and Indoor Air in the Office of Air and Radiation (OAR),
established the EPA's Alternative Technologies Initiative (ATI)
(http://www.epa.gov/radiation/source-reduction-management/alt-technologies.html). Part of the
EPA-ATI is fostering the acceptance and voluntary market adoption of non-radioactive
technologies; i.e., alternative technologies to those that currently use sealed sources. The EPA-
ATI is focusing primarily on alternative technologies for devices with Category 3 and 4
radioactive sources as classified by the International Atomic Energy Agency (IAEA) .
Commercial-ready or available alternatives to nuclear gauges (such as infrared, laser, and other
technologies) are being considered. As with any new technology, the likelihood of acceptance
can be significantly increased by independent evaluation and verification of a technology's
capabilities.
The purpose of this QAPP is to specify procedures for a verification test applicable to
commercially available alternatives to nuclear gauges that can replace technologies using sealed
radioactive sources for the measurement of basis weight, the measurement of mass per unit area
of a material. The purpose of the verification test is to evaluate the performance of the
Picometrix T-Ray 4000® Time-Domain Terahertz (THz) System in an actual production
environment for the measurement of basis weight. In performing the verification test, Battelle
will follow the technical and QA/QC procedures specified in this QAPP and will comply with
the data quality requirements in the AMS Center QMP.1 Because this verification will be
referenced by the Office of Air and Radiation's Alternative Technology Initiative, it was decided
-------
Verification of Picometrix THz System
QAPP
Page 17 of 52
Version 1.0
Date: 2/23/11
to establish the testing as a Quality Category II, requiring a QA review of 25% of the test data
and additional peer-reviewers.
A6.2 Technology Description
In the last twenty years, significant advancements in the THz systems signal-to-noise
performance have been realized, making the deployment of such systems to research laboratories
and industrial factories possible. The most commonly deployed terahertz systems are Time-
Domain Terahertz (TD-THz) units. Professors Daniel Mittleman (Rice University) and Daniel
Grischkowski (Oklahoma State University) have published numerous articles and books on this
method. Dan Mittleman maintains a list of researchers and groups active in THz study
(http: //www-ece. ri ce. edu/~dani el/group s. html).
TD-THz systems emit and detect a very narrow (<1 picosecond [ps]) electromagnetic
(EM) pulse that forms photons in the terahertz frequency range. TD-THz systems measure the
electrical field strength of the EM pulse as a function of time. The TD-THz frequency range
falls between microwaves and the Far-IR. One very important aspect of these frequencies is that
most dielectric materials are quite transparent in the region of study with TD-THz (0.05 - 3
THz). Plastics (regardless of color), paper, textiles, dry wood, packaging materials, rubbers,
foams, non-polar liquids (oils), paints (including low observable "radar absorbing") and other
coatings are all very transparent. Polar liquids (water, alcohols) are all strongly absorbing over
the complete TD-THz region.
The THz pulse is low energy (approximately 1 microwatt [|J,W]), completely safe, and
can be focused, reflected and treated essentially in the same manner as any pulsed "light" source.
After this pulse has interacted with matter (transmission, reflection, scatter), the changes in the
pulse have lead to two methods of analysis. Spectroscopic methods of investigation are possible
with THz. The Fourier transform of the time-domain data allows frequency (i.e., spectroscopic)
analysis. The second common methods of analysis is to directly study the time-domain data
measuring changes in the Time-of-Flight (ToF) of the pulse as it interacts with matter.
The time-domain analysis of THz pulses (ToF) has demonstrated the ability to measure
the basis weight (mass per unit area) of manufactured products. This is important because the
measurement of basis weight is the most common use for nuclear gauges in industry.
-------
Verification of Picometrix THz System
QAPP
Page 18 of 52
Version 1.0
Date: 2/23/11
A laboratory basis weight measurement consists of the measurement, with an analytical
balance, of the mass of a specific area of a sheet product (e.g., paper). The specific area cut from
the sheet product varies depending on the product. For paper products, a common unit area is
800 square inches. Example basis weight units of measure are pounds per square yard or grams
per square meter. For the paper industry, a common unit is pounds per ream, where a ream
represents 3300 square feet of paper.
The THz method to measure a material's basis weight is to measure the increase of the
Time-of-Flight (ToF) of the EM pulse as it transmits through the material of interest. This ToF
value is calibrated against accepted values of the material's basis weight. The THz method is a
time-based measurement, as opposed to the amplitude attenuation-based measurement method of
nuclear gauges.
The material's ToF value is found in the following manner: when photons transmit
through a material, the transit time of the photon will be increased due to the increased refractive
index (RI) of the material compared to the RI of photons in air or vacuum (~1). The ratio of the
velocity of photons in a vacuum to the velocity of photons in the material of interest, defines the
RI for that material. Because the velocity of the EM is less in a material, the amount of time
required for the EM to transmit through the material will be longer. The difference in time
between the EM pulse transmitting through the material, compared to the same transmission
through air, can be precisely measured with THz instrumentation. This difference is the Time-
of-Flight (ToF) delay. This ToF delay (typically in ps) is calibrated against accepted values of
basis weight for the sample material, which will be supplied by Appleton as a result of their
laboratory measurements.
A wide range of material basis weight values (5 grams per square meter (gsm) to greater
than 100,000 gsm) can be measured with this single source instrument. The terahertz system
directly measures the time delay increase due to the pulse passing through the material under
test. Formally this is the volume of material in the beam times the index of refraction of the
material at the terahertz frequency minus 1. Not to be tested in this plan, but of interest, is the
ability of THz to make simultaneous non-contact sample total and individual layer thickness
measurements and percent moisture content. The total sample thickness measurement, again a
time based measurement, requires the use of an External Reference Structure (ERS). Finally, the
amplitude of the transmitted THz pulse does contain simultaneous complementary information
regarding chemical and physical properties of the sample, as mentioned, the moisture content. In
-------
Verification of Picometrix THz System
QAPP
Page 19 of 52
Version 1.0
Date: 2/23/11
this work we distinguish between a measurement of thickness (sometimes called caliper
thickness) or the physical dimension of the material under test, and the basis weight, which is the
mass per unit area. A nuclear gauge measures the amount of matter between the source and
detector, which is most directly converted to a basis weight. In circumstances where the material
has a uniform density, the nuclear gauge measurement can also be correlated to physical
thickness.
Most THz measurements are made in reflection, as this geometry simplifies the system
configuration and reduces cost. Often, a fixed metal plate is installed behind the sample. The
THz pulse, reflected off a rear metal plate, will have transmitted through the sample twice. This
measurement mode is equivalent to double pass transmission and will have the measured ToF
delay increased by a factor of two.
The use of a beam splitter in the reflection sensor allows the transmitting and reflecting
THz pulses to remain collinear throughout the inspection (see Figure 2A). Therefore, the sensor
operates best when aligned orthogonal to the inspection surface. However, for illustration
purposes, an angle is often shown between the transmitter and receiver (see Figure 2B). This
display method helps to clearly separate the incoming and reflecting THz pulses and thus better
illustrates the origin and timing of the reflection pulses.
-•[
ra
.
Figure 2A. Collinear THz Reflection Sensor Figure 2B. Illustration of Pulse's Origins
Again, the most common use of nuclear gauges in industrial settings is the measurement
of basis weight. Thus, this verification test will address the comparison of the two measurement
systems only for basis weight measurements.
The fundamental method of nuclear gauges to find mass per unit area (basis weight) is to
calibrate the measured attenuation of the nuclear particle flux when passing through a sample
-------
Verification of Picometrix THz System
QAPP
Page 20 of 52
Version 1.0
Date: 2/23/11
against a standard-method-determined or accepted basis weight value. If the average density of
the product is known and remains constant, then the average sample thickness can be calculated
from the measured basis weight value.
The fundamental measurement with a THz sensor is the delay in the ToF of the THz
pulse as it passes through a sample. In a transmission measurement, the delay can be directly
measured and then calibrated against basis weight, a similar process to nuclear gauges. The
sample's ToF is calibrated against a standard method determined or accepted basis weight
values.
A7 VERIFICATION TEST DESCRIPTION AND SCHEDULE
A7.1 Verification Test Schedule
Table 1 shows the planned schedule of testing and data analysis/reporting activities to
be conducted in this verification test. As shown in Table 1, preparation to test the technology
began in January 2011. Testing will be conducted over 2-3 days during February, 2011. The
exact test dates will be determined the week before testing. After testing, an ETV verification
report and verification statement will be drafted for the participating technology. The report will
be reviewed by the technology vendor and peer reviewers that will be selected from the paper
and coatings industry and the EPA. The final report and statement will be submitted to EPA for
final signature, and these documents will be made publicly available on both the EPA/ETV and
the Battelle AMS Center websites.
-------
Verification of Picometrix THz System
QAPP
Page 21 of 52
Version 1.0
Date: 2/23/11
Table 1. Planned Verification Test Schedule
Month Year
January 20 11
February
2011
March/April
2011
May 2011
June 2011
July 20 11
Testing Activities
• Coordinate with vendor representative
• Coordinate schedules for technologies
and reference testing
• Perform testing
Data Analysis and Reporting
• Begin preparation of ETV report template
• Compile data
• Compile testing environment conditions
• Collect and analyze data from reference samples
• Analyze and finalize all data
• Complete common sections of reports
• Prepare draft report/verification statement
• Internal review of draft report/verification
statement
• Vendor review of draft report/verification
statement
• Revise draft report/verification statement
• Peer review of draft report/verification statement
• Revise draft report/verification statement
• Submit final report/verification statement for
EPA approval
A7.2 Test Sites
Testing will be conducted at Appleton Paper (Appleton) in Appleton, Wisconsin. In
performing this verification test, Battelle will ensure that the vendor and Appleton staff follow
the procedures specified in this QAPP and will comply with quality requirements in the AMS
Center QMP.1
A7.3 Health and Safety
The vendor and Appleton staff will conduct all verification testing and reference
measurements following the safety and health guidelines in place for the Appleton facility. This
includes maintaining a safe work environment and a current awareness of radiation exposure
potential. Testing involving the release of radiation will be performed by appropriately trained
personnel according to the Appleton facility safety guidelines.
-------
Verification of Picometrix THz System
QAPP
Page 22 of 52
Version 1.0
Date: 2/23/11
A8 QUALITY OBJECTIVES AND CRITERIA FOR MEASUREMENT DATA
The data quality objectives of this verification test are to:
• Evaluate the ability of THz sensor(s) to obtain on-line measurements of the mass per unit
area (basis weight) of different weight coated paper products; Picometrix expects that the
accuracy will be within 10% of an existing nuclear gauge (MeasureX company)
measurement system;
• Evaluate if THz sensors are able to replace nuclear sensors in the measurement of basis
weight; and
• Evaluate the ability of a THz sensor to make off-line, static measurements of the mass per
unit area (basis weight) of different weight coated paper products in a controlled
laboratory environment; Picometrix expects that these to be made with precision and an
accuracy within 10% of laboratory measurements.
The verification of the THz performance will be compared to laboratory measurements of
basis weight using TAPPI standard methods. Comparisons will also be made versus results
obtained using the nuclear (beta) gauges currently used as the in-line process monitoring
device at Appleton.
Data quality indicators (DQIs) ensure that this verification test provides suitable data for
a robust evaluation of performance. DQIs have been established for paper basis weight
measurements and the laboratory analyses that must be controlled for this performance-based
measurement. The DQI for these supporting measurements are quantitatively defined in Table 2
along with the acceptance criteria. Quantitative performance parameters for vendor technology
performance are discussed in Section B.
Appleton staff will operate the nuclear gauges and perform the laboratory testing; vendor
staffer vendor-appointed representatives will operate the THz gauges. These staff will follow
the manufacturer's recommended set-up, operations, and maintenance and any internal operating
procedures to ensure that the data generated during testing are representative of gauge
capabilities. Because the THz and nuclear gauges will not be operated or handled by Battelle
staff, Battelle staff will be present throughout testing to observe testing and document the
procedures and performance of gauges as well as the laboratory equipment used for the basis
weight determination (reference method). The requirements for these observations are described
in the discussion of documentation requirements and data review, verification, and validation
-------
Verification of Picometrix THz System
QAPP
Page 23 of 52
Version 1.0
Date: 2/23/11
requirements for this verification test. System model, nuclear gauge model, isotope type and
current activity of the Appleton nuclear gauge will be noted.
The Battelle Quality Manager will perform a technical systems audit (TSA] of the
THz, nuclear gauge, and laboratory testing activities to verify that testing and analysis were
performed according to the QAPP and internal Appleton test methods. The EPA quality
manager also may conduct an independent TSA, at their discretion. TSAs are described in
detail in Section C1.2.
Battelle will rely on the vendor's data quality indicators for their THz technology in order
to insure that the technology is performing properly during testing. The technology data quality
relies on proper operation and maintenance of the THz technology. Vendor DQIs specific to this
test are provided in Table 2. The results from the technology are expected to be quantitative and
will be reported as basis weight and compared to measurements of paper sampled during the test
for analysis at Appleton's in-house laboratory. Results from the nuclear gauges will also be
compared to the THz data as well. There are realistic factors in the production environment
which will limit how ideal the overlap of laboratory and gauge measurements will be.
Laboratory measurements of a production sample's basis weight can only be made on samples at
the end of the production roll. There is no method to perfectly synchronize the gauge
measurements to the paper roll position.
Table 2. DQI and Criteria for Critical Supporting Measurements
Phase
Laboratory1
Confirmation
Laboratory
Confirmation
Laboratory
Confirmation
DQI/Critical
Measurement
Accuracy /
Balance2
Accuracy /
TAPPI Room
Temperature
Accuracy /
TAPPI Room
Relative
Humidity
Method of
Assessment
Certified
weights
Thermometer
Hygrometer
Frequency
Quarterly by
professional
balance service
and
Prior to testing
Continuously
during testing
Continuously
during testing
Acceptance
Criteria
NIST tolerances
for analytical
balances
23±1.0°C
(73±1.8°F)
50%t2.0%
Corrective
Action
Maintenance
and
recalibrate
Repeat
measurement
Perform
maintenance
Repeat
measurement.
Apply
correction
factor
-------
Verification of Picometrix THz System
QAPP
Page 24 of 52
Version 1.0
Date: 2/23/11
Phase
Laboratory
Confirmation
Laboratory
Confirmation
In-line
Production
In line
Production
In line
Production
Off-sheet
Production
DQI/Critical
Measurement
Hygrometer
Thermometer
Completeness/
Amount of THz
data collected
per second
Accuracy/ THz
instrument
calibration
Accuracy/
Amount of
accurate data
collected
Accuracy/ mass
in chamber
Method of
Assessment
ISO 17025
Certified
Laboratory
ISO 17025
Certified
Laboratory
Time stamp of
THz data
stream
Calibration
standard
results vs.
initial
calibration
Review data
for anomalies
(>±5%from
average)
Off-sheet
check of
optics
Frequency
Annually and
within one
week of testing
Annually and
within one
week of testing
Each run:
number of data
points/ second
Daily Mid-day
check
Each run
Before and
after each run
Acceptance
Criteria
±1.0%at23°C
and 50% RH
Graduated to
0.20 °C
(0.50 °F)
>99%
±20"15sec
<20%ofdata
average
< ±50"15 sec
difference
between
readings
Corrective
Action
Replace
Replace
Investigate
and correct;
repeat run
Investigate
and correct;
repeat run
Confirm
data
filtering;
correct if
necessary;
repeat run
Investigate
for sensor
assembly
damage; if
none, use
new value
laboratory Confirmation is the reference method for this test.
2 Basis weight determined using Appleton Spec. No. 10001.00 which is based on TAPPIT 410.
A9 SPECIAL TRAINING/CERTIFICATION
Documentation of training related to technology testing, field testing, data analysis, and
reporting is maintained for all Battelle technical staff in training files at their respective
locations. The Battelle Quality Manager may verify the presence of appropriate training records
prior to the start of testing. Battelle technical staff supporting this verification test will have a
minimum of a bachelor's degree in science/engineering or equivalent years of experience.
The verification test described in this QAPP will be performed at an Appleton site. All
participants in this verification test (i.e., Battelle, EPA, and vendor staff) will adhere to the health
and safety requirements of the Appleton site. Operation of each technology (THz technology
-------
Verification of Picometrix THz System
QAPP
Page 25 of 52
Version 1.0
Date: 2/23/11
and nuclear gauge) will be carried out by a trained vendor and Appleton representative during
testing. Due to restrictions in the testing agreement, no Battelle or EPA personnel are permitted
to touch or handle the Appleton or vendor equipment. Vendor staff will operate only their THz
technology during the verification test. They are not responsible for, nor permitted to operate the
Appleton test site equipment (i.e. nuclear gauge and laboratory instruments). Appleton staff
operating the nuclear gauge and laboratory test equipment must be trained, experienced
operators. Data generated by the nuclear gauge and the THz gauge will not be shared between
operators during testing.
A10 DOCUMENTATION AND RECORDS
The documents for this verification test will include the QAPP, vendor instructions,
reference methods, verification report, verification statement, and audit reports. The project
records will include copies of calibration records, sample forms, records laboratory record books
(LRB; log book) containing the data collection forms, electronic files (both raw data and
spreadsheets), and QA audit files and all other quality records. All of these documents and
records will be maintained at the test site during the test, will be transferred to the VTC at the
end of testing, then to permanent storage at Battelle's Records Management Office (RMO) at the
conclusion of the verification test. Electronic documents and records will also be uploaded to a
SharePoint site designated for this test and will be provided to EPA upon request. All Battelle
project raw data files are stored for at least 10 years by RMO. EPA will be notified before
disposal of any files. Section BIO further details the data recording practices and
responsibilities.
A bound, pre-paginated test logbook that contains the data recording forms created for
this test (Attachment 1) will be used to document all data and observations made by Battelle
during the test. All data generated during the conduct of this project will be recorded directly,
promptly, and legibly in ink. All data entries will be dated on the date of entry and signed or
initialed by the person entering the data. Any changes in entries will be made so as not to
obscure the original entry, will be dated and signed or initialed at the time of the change and
shall indicate the reason for the change. Project-specific data forms will be developed prior to
testing to ensure that all critical information is documented in real time. The draft forms will be
provided to EPA prior to testing.
-------
Verification of Picometrix THz System
QAPP
Page 26 of 52
Version 1.0
Date: 2/23/11
Table 3. Summary of Data Recording Process
Data to Be
Recorded
Dates, times, and
details of test
events, technology
maintenance, down
time, ease of use,
etc.
Technology
calibration
information
Technology
readings
(THz and nuclear
gauges)
Hobo continuous-
monitoring
temperature and
relative humidity
records
Reference method
analysis
conditions,
procedures,
calibrations, QC,
etc.
Reference method
results
Where Recorded
ETV LRBs or data
recording forms
ETV LRBs, data
recording forms, or
electronically
Recorded
electronically by
the technology and
downloaded to an
independent
computer, or hard
copy data printed
by the technology
and taped into the
ETV LRB, or hand
entered into ETV
LRBs or data
recording forms
Recorded
electronically by
the instrument and
downloaded to
independent
computer after
each data
collection event
LRBs, or other
data recording
forms (Battelle will
retain copies of all
applicable
Appleton records
for these
measurements)
Electronically or
manually into ETV
LRBs or data
recording forms
How Often
Recorded
Start/end of test
procedure, and at
each change of a
test parameter or
change of
technology status
At technology
calibration or
recalibration
Recorded
continuously for
electronic data,
printed after each
measurement for
hard copy print-
outs, or recorded
manually with each
reading
Recorded
continuously
during each
collection event
(production run,
laboratory
measurement
event)
Throughout
reference analysis
Every weight
measurement
By Whom
Battelle or
technology
operator
Technology
operator
Technology
operator
Battelle Quality
Manager
Collaborator,
Battelle, or others
assisting in
reference analysis
Collaborator,
Battelle, or other
reference analysis
technician
Disposition of Data
Used to organize
and check test
results; manually
incorporated in
data spreadsheets
as necessary
Incorporated in
verification report
as necessary
Converted to or
manually entered
into spreadsheet
for statistical
analysis and
comparisons
Used to verify the
temperature and
relative humidity
of the production
and TAPPI rooms
Retained as
documentation of
sample collection
or reference
method
performance
Transferred to
spreadsheets for
calculation of
results, and
statistical analysis
and comparisons
-------
Verification of Picometrix THz System
QAPP
Page 27 of 52
Version 1.0
Date: 2/23/11
SECTION B
MEASUREMENT AND DATA ACQUISITION
Bl EXPERIMENTAL DESIGN
A non-sealed source THz technology (Picometrix T-Ray 4000 Time-Domain Terahertz
System) will be tested at Appleton. This will allow for performance evaluation under real world
manufacturing conditions. Overall, the performance of the THz technology will be verified
based on the following factors:
• Accuracy
• Precision
• Comparability
• Operational factors (ease of use, sampling time, sampling costs).
The verification test is planned to take place over a period of 2-3 days. The evaluations
will be performed according to the vendor's approved procedures as described in the user's
instructions or manual and will be carried out by a trained and licensed operator provided by the
vendor. Similarly, calibration and maintenance of the technologies will be performed by the
vendor or vendor-provided operator. If possible, calibration of the vendor technology will be
conducted onsite so that Battelle technical staff can observe the procedure. The technology will
be evaluated based on comparable metrics to the nuclear gauge. Results from the technology
being verified will be recorded manually by the operator on appropriate data sheets or captured
in an electronic data system and then transferred manually or electronically to spreadsheets
and/or forms for further data analysis.
Bl.l Test Procedures
The following sections describe the test procedures that will be used to evaluate the
fEl
Picometrix T-Ray 4000 Time-Domain Terahertz System at the Appleton facility.
The proposed comparison testing method is to temporarily set up a THz sensor at a fixed
position near the edge of a moving paper sheet in the Appleton production facility. The existing
production nuclear gauge system is on a scanner frame and can either be moved back-and-forth
across the sheet or "parked" at a position. The THz sensor will be aligned with the nuclear
-------
Verification of Picometrix THz System
QAPP
Page 28 of 52
Version 1.0
Date: 2/23/11
gauge. Both the THZ sensor and nuclear gauge will remain stationary during testing. Figure 3
shows the positioning of the gauges.
Beta
Gauge
^^H^^l
sneet
Sheet
Mocfiine
Dnection
THz
Figure 3. Relative Position of Sample Sheet, Nuclear (Beta) Gauge and THz Sensor
The nuclear gauge and THz sensor will be placed as close to each other as reasonably
possible (less than 2-3 feet) at the point in production after the last coating application
(inspection position). The THz sensor will remain static (two to four inches from the edge of the
paper). The nuclear gauge will both be scanning across the sheet and later parked aligned with
the THz sensor. The Appleton production line used for testing involves multiple coating
applications.
The nuclear scanning will be needed by the Appleton production staff for the first few
(approximately three) rolls once a new product is beginning its production run. Once production
conditions permit the nuclear gauge to be temporarily taken off-line, the nuclear gauge will be
"parked" in a fixed position aligned with the THz gauge. Continuous overlapping measurement
data will be collected and logged by both the gauge and sensor over at least a five minute period.
The current nuclear gauge acquires measurements at a 1/5 Hz rate. Thus, a five minute period
will result in a log of 60 nuclear gauge measurements. The THz system acquires measurements
at a 100 Hz rate, thus 30,000 measurements will be collected over the five minute period.
At the mid-point of each day of testing, the THz system time measurement calibration
will be checked. This calibration procedure will follow protocols established by the vendor for
-------
Verification of Picometrix THz System
QAPP
Page 29 of 52
Version 1.0
Date: 2/23/11
the technology. The reason for the test is to confirm the measured ToF delay through a
calibration block of fused silica is within specified tolerances. The nuclear gauge will be
operated and calibrated according to manufacturer recommendations and Appleton established
protocols.
At the start of a new roll of the material under test, the nuclear gauge will operate
normally; scanning across the product sheet and providing feedback to the production equipment
and staff. If feasible, THz measurements will be acquired frequently during this period for later
exploratory analysis. Once the Appleton production staff has become confident of the operation
stability, the nuclear gauge may be moved to a static position near the edge of the roll for trial
testing. The range of times required to coat an entire roll of paper varies from 15-30 minutes.
The nuclear gauge will be available to monitor the coating for the second half of a roll. Appleton
production staff requires the use of the nuclear gauge during the first half of a roll coating
process to ensure proper product quality. All steps within a single measurement run will be
completed within a single roll of product. Five rolls each of two different products, a midrange
and heavyweight, will be measured.
For all measurement runs, the general data collection procedure is as follows:
once proper manufacturing stability is confirmed by the Appleton production representation, the
nuclear scanner will be moved and parked at a predetermined position to align with the THz
sensor, somewhere between 2 and 4.25 inches in from the edge of the sheet. The sheet edge
position will vary from product to product, hence the variation in the distance from sheet edge.
The THz sensor will be positioned as close as possible, within 2-3 feet, directly in front of or
behind the nuclear gauge.
Before sample data collection, the THz sensor assembly will be rotated "off-sheet", i.e.,
so the product is not in the measurement path. This step simulates a scanning operation; where
the sensor moves past the paper sheet (off-sheet) before reversing direction to continue scanning.
In this off-sheet position, a system measurement check will be conducted. This check will
involve measuring the "open-air space" ToF value. This data will be collected for two seconds,
which represents a typical time for a sensor to go off-sheet, reverse direction, and move back
onto the sheet. This measurement will be stored for use in the basis weight calculation. The
sensor will then be rotated back to the fixed position aligned with the nuclear gauge. On-line
data collection should begin within one minute after the completion of this procedure.
Data will then be simultaneously collected and logged for both the nuclear and THz
-------
Verification of Picometrix THz System
QAPP
Page 30 of 52
Version 1.0
Date: 2/23/11
systems for a continuous period of at least five minutes. The THz data collection is triggered
with a key press. The method of triggering the logging of the nuclear gauge data will be
determined by Appleton staff, and will be started at the same time as the THz system. If the
logging start of the nuclear gauge is imprecise, a slightly longer data set will be collected to
ensure there is at least five minutes of overlapping data between the nuclear and THz gauges. If
necessary, a hand trigger can be used to mark the THz data at the start of nuclear gauge
measurements. The clocks for both technologies will be synchronized, and data from both the
gauge and sensor will be time stamped to allow the results to be more precisely overlapped. The
measurement system clocks used to timestamp the data will be compared within ±10 minutes of
each data collection run.
The nuclear gauge must move off-sheet (i.e., ending on-line measurements) 16 seconds
before the end of a roll. This step is required to prevent the individual that pastes the end of the
previous paper roll to the beginning of the next roll from being caught in the relatively narrow
gap between the top and bottom plates of the nuclear gauge. Once the start of the new
production roll is past the inspection point the nuclear gauge will return to scanning the product.
At the end of all rolls for which an inspection was completed, an end-of-roll "tear-off
paper sample will be collected. These samples will all be marked and saved for laboratory and
later THz inspection.
Within one minute of the end of the data collection, the THz sensor assembly will again
be rotated off-sheet and a two second data set of 'open-air space' (i.e., nothing between the THz
transmitter and receiver) ToF values will be recorded.
This process will be conducted at least five times in the production run of the two
products (i.e., midrange and heavyweight stock) to be tested.
The laboratory results from the first data collection run will be used to temporarily
calibrate the THz sensor to output basis weight values. All "raw" THz measurement values will
be saved allowing this calibration to be refined once all measurement results are available.
Testing is planned for two to three days during the week of February 14, 2011. Additional
days may be required for system setup and to complete the Technical Systems Audits described
in Section C. A summary of the testing is as follows:
• Five data collection runs will be completed per product grade. In general, data logging
will begin as soon as reasonably possible once the nuclear gauge can be taken off-line for
trial testing.
-------
Verification of Picometrix THz System
QAPP
Page 31 of 52
Version 1.0
Date: 2/23/11
• All data collection runs will collect and compare simultaneous nuclear / THz and THz /
laboratory measurement results, but not nuclear / laboratory results. The paper sample
collected for laboratory analysis will also be analyzed off-line with the THz technology
in order to further establish measurement accuracy and precision.
Specific steps of the data collection process for Appleton and Picometrix personnel are:
-> An Appleton representative will be required to park the nuclear gauge and begin
and end data logging.
-> A Picometrix and Appleton representative will be required to rotate the THz
sensor off-sheet, collect a 2-second air space ToF measurement and return the
sensor to on-sheet.
-> A Picometrix representative will be required to begin and end data logging, and to
press a trigger to mark the THz data when the nuclear gauge goes off-sheet
Appleton personnel will collect a typical end of roll sample for laboratory basis weight
measurement for later off-line THz measurements.
Conditions of testing:
The materials manufactured at this facility are coated paper products. All samples tested
will be coated final products.
• The tested parameter will be product basis weight.
• Two differing bare stock basis weight materials, a mid-range and heavyweight, will be
tested.
TAPPI standard method guidance will be used to determine basis weight.
• The nuclear gauge will be required to be scanning and operating (i.e., not available for
trial testing) at the start of every new roll. This is to guarantee production quality control.
This period of use will be variable depending on the amount of time necessary to
establish confidence in the product quality. A typical roll time is 15-20 minutes. It is
assumed that the second half of the roll can be used for trial testing.
The nuclear gauge will go off-sheet 16 seconds before the end of roll, preventing the
"paster" that attaches the two rolls from being caught in the relatively narrow gap of the
nuclear gauge.
-------
Verification of Picometrix THz System
QAPP
Page 32 of 52
Version 1.0
Date: 2/23/11
There could be a significant imprecision in removing the exact sample section from the
paper roll. Assuming a manufacturing rate of 3000 ft/min, a 1-second offset between the
data logging and selected sample position will result in a mismatch of 50 feet (600
inches).
The laboratory results for the first measurement run on each product grade will be used to
temporarily calibrate the THz sensor to provide real-time measurement results for
subsequent runs of that product.
All THz "raw" measurement results (i.e., Time-of-Flight values) for all runs will be
logged to allow a separate comparison of the on-line and off-line measurements.
It is assumed that the static THz sensor configuration (1.5" gap) can remain in place
through the end of all rolls.
Comparisons of on-line measurements between the nuclear and THz technologies are
easy to perform, and are possible whenever the nuclear gauge is available for trial testing
purposes.
It is assumed that the THz sensor configuration can be rotated so as to simulate going off-
sheet. While off-sheet, calibration checks of the THz sensor will be made.
It is assumed that a THz sensor can easily be used in the "TAPPI room" laboratory
testing area.
Additional accuracy and all precision THz measurements will be made in the TAPPI
room on the same samples used for the laboratory basis weight testing. In paper
products, the paper "formation" creates basis weight variation on a short length scale.
The large spot and web movement creates natural averaging of the nuclear gauge's
measurements so that formation does not have an effect. In the laboratory measurements,
sheets of paper are weighed to determine basis weight, again naturally averaging over
variations due to formation. Due to the relatively smaller inspection spot of the THz
sensor, multiple measurements over an area will be performed to avoid any issues with
the product's formation. Another option is to cut and stack multiple sheets of the
laboratory samples for inspection with the THz sensor. The stacking of samples should
reduce formation induced product basis weight variability.
-------
Verification of Picometrix THz System
QAPP
Page 3 3 of 52
Version 1.0
Date: 2/23/11
Data Collection Runs:
On-line Data Collection
1) For a mid-range basis weight product:
a) Run 1: Acquire simultaneous nuclear and THz sensor measurements for at least the
last five minutes of a production roll. The nuclear gauge will move off-sheet (and
thus end its data collection) 16 seconds before the end of the roll. The THz sensor
will continue to collect data past the end of the roll.
i) Use a hand trigger switch to mark the THz data when nuclear gauge starts to
move off-sheet, thus aligning the end of nuclear gauge measurements to the THz
results.
ii) Collect an end-of-sheet tear off sample for later laboratory and THz analysis. The
laboratory results can be simultaneously compared to the nuclear and THz sensor
results.
iii) Use the laboratory results from this first data run to temporarily calibrate THz
sensor to provide real-time results for subsequent measurement runs.
iv) Use laboratory sample to perform static THz measurements to further confirm
accuracy and establish static measurement precision.
b) Runs 2-5: Are the same as Runl except the THz system will now be programmed to
directly report an estimated basis weight value (while still logging the ToF raw data).
c) Use laboratory samples to perform static THz measurements to further confirm
accuracy and establish static measurement precision.
2) Repeat all above steps for heavyweight product.
A typical single product production run can range from 12 hours to > 50 hours. The five
repeated measurements will occur within these ranges and will be spread out to encompass as
large a variation of the two basis weight materials (medium and heavy) as possible. The timing
of the repeated measurements will be determined by Appleton.
There will not be any control of the environmental conditions during testing. That is,
both gauges will be operating in a manufacturing factory environment. Environmental
conditions present during testing will be noted based on information provided by Appleton staff.
-------
Verification of Picometrix THz System
QAPP
Page 34 of 52
Version 1.0
Date: 2/23/11
Battelle staff present at the test site will also provide a temperature and relative humidity gauge
to record conditions on the production floor as well as in the TAPPI conditioning room.
All laboratory results will be collected and run per Appleton protocols, which are based
on TAPPI standards3. The protocols are considered business confidential information. In
general, the samples will be bagged after collection and allowed to condition in TAPPI rooms on
the manufacturing floor. The environment of these rooms is tightly controlled (72 °F and 50%
RH). The length of conditioning time is determined by the sample type. Laboratory
measurements will be collected on tear-off samples collected for each of the five runs for each
paper type. Once all on-line data collection is completed in the production facility, a THz sensor
will be moved to the TAPPI room to allow static off-line THz measurements.
Off-line Data Collection
Once laboratory basis weight analyses are complete for a given run, the five physical
paper samples will be repeatedly measured with a static THz sensor in the TAPPI room to further
establish measurement accuracy and precision. The steps to prepare the five paper samples
follow.
1) For each sample, there are four sheets, all 12.5" by 9.5". These sheets will be cut into
two inch by two inch pieces resulting in 96 pieces.
2) The suggested best method to establish a calibration factor, and thus the accuracy, is to
stack a number (e.g., 12 pieces yielding eight stacks) of individual sheet sections in a
sample holder and measure the total ToF. This method improves the measurement
confidence in two ways:
i. By stacking the sheets, any changes in the material's formation (short distances
variations in sample mass / density / consistency) will be averaged away with a
single measurement. Note: it is not expected that the eight measurements will
yield the same result, even with this spatial averaging procedure. A coated paper
product will have appreciable spatial variation in basis weight over fractions of
the dimensions of the total sheet sample.
ii. The increase of total ToF will reduce the contribution of any fixed noise in the
measurement.
3) Each of the eight sample stacks will be measured for an integration period of about 30
seconds.
-------
Verification of Picometrix THz System
QAPP
Page 3 5 of 52
Version 1.0
Date: 2/23/11
4) All individual sample stack measurement results will then be averaged and used to
calculate a THz ToF to basis weight conversion factor.
5) One randomly selected sample stack will be installed in the sample holder and measured
twice. This will be used to determine the reproducibility of the measurement.
6) Unfortunately, static off-line nuclear gauge measurements are not possible during this
trial. Thus, accuracy and precision comparisons to the nuclear gauge will not be made.
7) Alternatively, a large number of one second average measurements over a number of
different positions on a single sheet can be collected. This method would be preferred if
it is demonstrated that the formation of the product does not adversely affect the
measurement.
The THz "raw" data will consist of a .csv data file of Time-of-Flight values for every
measurement (100 per second). The calibration factor to convert these measurements to basis
weight will be temporarily determined from the laboratory results from the first data run for a
new product. That calibration factor will then be used for subsequent data runs to provide real-
time basis weight values and logged as the measurement results. Because the ToF values will be
saved for every run, it will be possible to recalibrate the THz measurements to align with the
method of choice; on-line THz measurements versus laboratory measurements, off-line static
THz measurements versus laboratory measurements or on-line THz versus on-line nuclear
gauge.
The data format of the nuclear gauge will be determined by Appleton. There exist
possible proprietary limitations on the use of data streams of the nuclear system when in parked
mode. Appleton will determine how to best log and report the data to eliminate these issues.
As every measurement of each gauge will be logged, for some analyses the averaging
interval for both sets of data can be altered in post analysis, filtered and smoothed. The majority
of the processed THz data sets will be made available by the end of that testing day. The
availability of the nuclear gauge data will be determined at the start of testing.
-------
Verification of Picometrix THz System
QAPP
Page 36 of 52
Version 1.0
Date: 2/23/11
Bl.1.3 Testing Parameters
A total of 10 data sets each will be taken by the THz technology and the nuclear gauge
technology on the Appleton production line. Quantitative assessments will be made. The
following sections describe in more detail the evaluation of the testing parameters.
Bl. 1.3.1 Accuracy
Accuracy will be assessed by evaluating the results of the THz gauge against the results
from the standard laboratory method for the determination of basis weight. The comparison of
accuracy may be limited for this production measurement trial. The limitation is the collection
of laboratory samples (i.e., the standard measurement method) in that the exact overlap between
the on-line and physical paper sample cannot be guaranteed. In order to address this possible
complication, off-line static THz measurements are planned on the paper samples. By making a
large number of measurements over the spatial area of the lab sample, the accuracy of the THz
measurement can be assessed.
Bl.1.3.2 Precision
The precision of the THz system will be assessed with static off-line measurements of
paper samples used for laboratory basis weight determination. With repeated measurements on a
static sample, the measurement precision can be assessed.
Bl.1.3.3 Comparability
This test will assess the performance of the THz technology based on the measurement of
whole sample basis weight. The measurement of that whole sample basis weight accounts for
the vast majority of nuclear gauge use. If a nuclear gauge is extended to measure an additional
parameter (e.g., thickness), the additional measurement is dependent on the same results to
obtain basis weight. Thus, comparing the basic measurement of basis weight should allow the
best comparison between the two technologies. There are, however, some limitations in the
comparison of the nuclear and THz gauge measurement of basis weight that should be noted.
One limitation in the comparison between the techniques is the operational need to
remain static for this testing. That is, cross sheet scanning will not be conducted. Though cross
sheet scanning is typically how the nuclear gauge would be operated in the Appleton production
environment, due to practical limitations, cross sheet scanning is beyond the scope of this
verification test. An additional factor limiting comparison is the difference in measurement spot
-------
Verification of Picometrix THz System
QAPP
Page 37 of 52
Version 1.0
Date: 2/23/11
size between the two technologies. A typical nuclear gauge inspection spot is 25 millimeter
(mm), while a typical THz sensor inspection spot is 2 mm. At a production line speed of 4000
feet/minute, these "spots" become spread to 25 mm x 1041 mm for the nuclear gauge's 50
millisecond measurement integration time and 2 mm x 125 mm with an 80 mm gap between THz
measurements. Clearly, these two systems will be inspecting the product in different ways. The
nuclear gauge covers a larger area and thus could provide an improved result for the average
basis weight value. The higher measurement rate of the THz system somewhat compensates for
this difference. In addition, the smaller inspection spot allows for better streak detection and
could possibly provide information on the formation (uniformity of basis weight) of the sample.
Bl. 1.3.4 Operational Factors
Operational factors such as maintenance needs, power needs, calibration frequency, data
output, consumables used, ease of use, repair requirements, training and certification
requirements, safety requirements and image throughput will be evaluated based on Battelle staff
testing observations and input provided from the vendor. Input will either be provided by the
vendor on-site during the verification test and be recorded by Battelle staff or will be provided in
documentation to the VTC after completion of the verification test. To the extent possible,
Battelle technical staff will also observe and record their own observations of these operational
factors. Examples of information to be recorded include the daily status of diagnostic indicators
for the technology, use or replacement of any consumables, use and nature of power supply
needed to operate the technology, the effort or cost associated with maintenance or repair, vendor
effort (e.g., time on site) for repair or maintenance, the duration and causes of any technology
down time or data acquisition failure, observations about technology startup, ease of use, clarity
of the user's instruction manual, user-friendliness of any needed software, overall convenience of
the technologies and accessories/consumables, the safety hazard associated with the use of the
technology, or the number of images that could be taken and processed per hour or per day.
These observations will be summarized to aid in describing the technology performance in the
verification report.
B1.2 Statistical Analysis
The statistical methods and calculations used for evaluating quantitative performance
parameters are described in the following sections. Continuous measurement data will be
-------
Verification of Picometrix THz System
QAPP
Page 3 8 of 52
Version 1.0
Date: 2/23/11
collected and logged by both the nuclear gauge and THz sensor for at least a five minute period
during each paper roll test. This will be done over at least five separate runs (Runs 1 - 5 for each
paper type). A five minute period will result in a log of 60 nuclear gauge measurements and
30,000 measurements from the THz technology.
As noted in Section Bl.l, cross sheet scanning is typically performed by the nuclear
gauge during production at the Appleton facility. The scanning is required to monitor the
uniformity of the coating process. As the nuclear gauge scans across the sheet, the product is
divided into a number of cross direction "bins". The nuclear gauge will scan across the web at
approximately 6 inches per second, producing 96 bin measurements for each single 15 second
scan across the product. Then, a number of repeated scans are averaged before a measurement
result for that bin is reported. This means for routine operation of a nuclear gauge, a single bin
measurement is an average of a number of sets of measurements with each individual
measurement covering a certain amount of product. These values are running averages. These
average results are then reported through the system to help monitor the production process. The
averaging of both across the sheet (cross direction) measurements and along the production
direction (machine direction) measurements is used to maintain controlled measurement
feedback to the coating devices.
However, for this test, the gauges must remain in "parked" mode. Thus, the cross web
averaging cannot be accomplished. However, the data acquired during this test is the closest
approximation to the typical operation of the nuclear and THz gauges in a controlled setting.
B 1.2.1 Accuracy
The accuracy of the results will be assessed by calculating the percent error between the
laboratory measurements and the results from the nuclear gauge and THz technologies. Percent
error will be calculated using the following:
\Technology - Lab
%Error = ]- - — - xlOO
Lab
The first sample collection run (Run 1) will be used to establish the calibration factor
used to calculate basis weight. The remaining runs (Run 2-5) end of roll sample collection can
-------
Verification of Picometrix THz System
QAPP
Page 3 9 of 52
Version 1.0
Date: 2/23/11
be used for an accuracy comparison. However, as previously mentioned, there are identified
issues with using end of roll samples to compare to on-line measurements.
In an attempt to better qualify accuracy, off-line static THz measurement of samples
collected for laboratory analysis is proposed. By measuring the laboratory samples used for the
basis weight measurement, at least the overlap issue is addressed. The remaining consideration
is how measurements over what percentage of the 475 square inch samples should be measured
with THz. In order to integrate a larger sample area and to reduce the effect of formation
(variability in the sample's basis weight), the off-line tests will be made with a large number
(e.g., 12) of stacked sections cut from the large laboratory sample sheet. If the sample sheet was
completely cut into two inch by two inch square pieces, then eight stacks of 12 sheets per stack
could be used for static measurements.
It is important to note that the product under study, coated paper, is inherently non-uniform.
Thus, the ToF delay measurements for each "stack" are expected to have different values
because they are taken from different positions of the laboratory test sheets. The need to average
out this non-uniformity is the logic for selection sample form all positions of the four lab test
sheets. The goal is to measure the whole area of laboratory samples, as illustrated in Figure 4.
Red squares represent 12.5 by 9.5 inch sheets cut from
the tear off. The four sheets combine for 475 square
inches of sample. The black dots represent the position
of a THz measurement when the cut sections are stacked
into eight groups of 12 sections.
Figure 4 - Illustration of Laboratory Sample Sectioning to Achieve Spatial Sampling of
Test Sheets
Bl. 2.3 Precision
The cut samples used in the above testing will also be used to determine measurement
precision. The precision will calculated as the standard deviation of repeated measurements.
-------
Verification of Picometrix THz System
QAPP
Page 40 of 52
Version 1.0
Date: 2/23/11
N-l
Bl.2.2 Comparability
One simple measure of comparability between the technologies will be assessed by
calculating the percent difference between the measurements made by the THz technology and
the nuclear gauge while in the on-line production mode. This evaluation will help in assessing
the performance of the THz technology in relation to that of the instrument typically used for
production control processes (i.e., the nuclear gauge). Percent difference will be calculated using
the following:
„ „ ^ ... (Terahertz Technology Result - Nuclear Gauge Result] , ^
^Difference = ± — '- x 100 (2)
Nuclear Gauge Result
Comparability results will be calculated for each separate run evaluated during the verification
test. Average percent difference will also be determined across the five runs.
Some exploratory data analysis procedures will also be examined to evaluate the
comparability of the two technologies. In particular, the raw data from each technology will be
plotted together. These plots will allow for examination of how well the measurements from the
two gauges track each other. These plots might also be used to explore variations in basis weight
measurements over time. The raw data may also be smoothed for the exploratory analysis, such
as using locally weighted scatterplot smoothing. The use of histograms, or other techniques such
as Grubbs, Ferguson, Dixon, and Hofer-Rickard methods, will be utilized to study the data
variability (i.e., outliers).
More advanced statistical methods of comparing the variance between the two data sets
will also be considered. A t-test and/or an ANOVA will be considered to evaluate the agreement
and variance between the data from the nuclear gauge and the THz sensor.
B1.3 Reporting
Following completion of the data evaluations, a draft verification report and verification
statement will be prepared, stating the verification test procedures and documenting the
-------
Verification of Picometrix THz System
QAPP
Page 41 of 52
Version 1.0
Date: 2/23/11
performance observed. For example, descriptions of the data acquisition procedures,
consumables used, repairs and maintenance needed, and the nature of any problems will be
presented in the draft report. The report will briefly describe the ETV Program, the AMS
Center, and the procedures used in verification testing. The results of the verification test will be
stated quantitatively, without comment on the acceptability of the technology's performance.
The draft verification report will be submitted for review by the technology vendor, by EPA, and
other peer reviewers. Comments on the draft report will be addressed in revisions of the report.
The reporting and review process will be conducted according to the requirements of the AMS
Center QMP.1
B2 SAMPLING REQUIREMENTS
Laboratory samples will be collected at the end of each roll for which a set of nuclear and
THz gauge measurements were collected. Thus, five laboratory samples will be collected from
the mid-weight paper stock and five from the heavyweight stock. The sample will be an
approximately 400 square inch piece of material. After a sample is cut from the end of a roll, it
is immediately bagged per the pertinent Appleton protocol and/or TAPPI standard. The samples
will be transported and stored in a specialty TAPPI room (i.e., tightly controlled environment)
within the Appleton production facility. The samples will then condition for a set time in the
TAPPI room before measurements are taken. Conditions for the conditioning room will be
followed per established Appleton and/or TAPPI protocols.
B3 SAMPLE HANDLING AND CUSTODY REQUIREMENTS
The paper samples collected for laboratory measurements will be handled per the
appropriate Appleton paper protocol.3 Each paper sample will be labeled and assigned a unique
sample ID, known as the "Reference Roll Number", generated by Appleton. This will identify
the Roll No., Run No., Paper Type, Date/Time of Collection, and other pertinent processing
parameters. The unique sample ID will also be documented in the test LRB or associated data
collection sheet(s). Sample custody information that documents collection, possession, and
transfer of samples will be documented directly on the sample collection forms.
-------
Verification of Picometrix THz System
QAPP
Page 42 of 52
Version 1.0
Date: 2/23/11
Once conditioned for the appropriate amount of time in the TAPPI room, the
measurements of interest will be collected using the Appleton laboratory basis weight
measurement protocol3. Once measurements are complete, unless otherwise specified, the
samples will be available for further study by the THz gauge. Samples will not be archived for
this test or removed from the Appleton facility.
B4 REFERENCE METHOD
Laboratory analysis of basis weight at the Appleton laboratory will serve as the reference
method for this test. Laboratory procedures are described in Appleton Standard Test Method
10001.00 Basis Weight - Laboratory Determination of Coated and Uncoated Paper ^ This
method is based on TAPPI Method T 410. Each sample will be approximately 800 square
inches and must be tested, cut and weighed at the TAPPI Room conditions (defined in Table 2).
The laboratory measurements will be performed by Appleton staff. Laboratory measurement
results will be provided to Battelle. The QA/QC requirements for the performance of the
analytical method are described as data quality indicators (DQI) in Section A8.
B5 QUALITY CONTROL REQUIREMENTS
Table 2 presents the DQIs and criteria for the laboratory reference method measurements.
The reference method measurement quality will be assured by adherence to these measurement
quality and DQI criteria. All laboratory equipment will be maintained, calibrated, and operated
according to Appleton internal requirements and TAPPI standards.
B6 INSTRUMENT/EQUIPMENT TESTING, INSPECTION, AND MAINTENANCE
The equipment for the reference method for this test includes balance, pre-scribed cutting
device (guillotine cutter) (indicating the cut dimensions of the type of paper under study),
temperature measuring device, and hygrometer. This equipment will be tested, inspected, and
maintained by Appleton staff according to the manufacturer's recommendations and any
appropriate TAPPI standards to ensure that the performance requirements established in this
document can be achieved. Battelle staff will observe that the balance, temperature measuring
device, and hydrometer have been tested and verified to be operating with the Table 2 criteria
-------
Verification of Picometrix THz System
QAPP
Page 43 of 52
Version 1.0
Date: 2/23/11
prior to use. Operation of the guillotine cutter will be observed during laboratory tests but
cutting accuracy is not a DQI. The TSA will confirm compliance with the QAPP. A Hobo
continuous-recording monitor that has been calibrated at Battelle's ISO 17025-certified
instrument laboratory will collect temperature and relative humidity measurements during
production test runs and while measurements are made in the TAPPI room.
B7 INSTRUMENT CALIBRATION AND FREQUENCY
The balance, temperature measuring device, and hygrometer, will be calibrated according
to the manufacturer's specifications and Appleton procedures prior to testing. Table 2 defines
calibration frequency. Calibration of the TAPPI room equipment will be verified during the
TSA. Certified analytical balance weights will be used to verify balance calibration and a Hobo
continuous-recording monitor will be used to verify the accuracy of the TAPPI room temperature
and relative humidity equipment. The nuclear gauges, MeasureX, MX Open system, will have
been calibrated by the Appleton plant operator. The THz technology will be calibrated by the
vendor according to the technology's specified procedures. This calibration will be performed
onsite prior to testing and confirmed once every mid-day.
B8 INSPECTION/ACCEPTANCE OF SUPPLIES AND CONSUMABLES
It is not anticipated that any materials, supplies, or consumables will be ordered by the
VTC for this test. Supplies used by Appleton and the vendor will be received, inspected, and
documented according to their internal procedures. For this test, additional documentation may
be required. This includes: source, certification of analysis or verification, and date of receipt for
supplies that could impact data quality. If this documentation is required, Appleton staff and
vendor representatives will be asked to supply the appropriate paperwork.
B9 NON-DIRECT MEASUREMENTS
No non-direct measurements will be used during this verification test.
-------
Verification of Picometrix THz System
QAPP
Page 44 of 52
Version 1.0
Date: 2/23/11
BIO DATA MANAGEMENT
Various types of data will be acquired and recorded electronically or manually by
Battelle during the verification test. Electronic data, including the JOHN NEW COMMENT:
THz sensor, nuclear gauge, and Hobo monitoring data, will be saved with unique file names that
are directly traceable to the test records for each test run date/time and activity. Table 3
summarizes the types of data to be recorded. All maintenance activities, repairs, and calibrations
relevant to the technology operation will be requested from Appleton staff and the technology
vendor. All observations relevant to the technology operation during the course of verification
testing will be documented in LRBs or on data sheets by Battelle technical staff present during
testing. Results from the reference method, including raw data, analyses, and final results, will
be documented by the Appleton laboratory and compiled by Battelle. Battelle will obtain copies
of all results (hard copy and electronic) in real time to ensure the integrity of the data.
Records received by or generated by any technical staff during the verification test will
be reviewed by a Battelle staff member within two weeks of generation or receipt, before the
records are used to calculate, evaluate, or report verification results. If a Battelle staff member
generated the record, this review will be performed by a Battelle technical staff member involved
in the verification test, but not the staff member who originally generated the record. The review
will be documented by the person performing the review by adding his/her initials and date to the
hard copy of the record being reviewed. A technical review of 100% of the test and reference
data produced will be conducted. In addition, any calculations performed by technical staff will
be spot-checked by Battelle QA and/or technical staff to ensure that calculations are performed
correctly. Calculations to be checked include any statistical calculations described in this QAPP.
Data obtained during the verification test will be maintained confidentially at Battelle,
and used only for purposes of the technology evaluation. Data reporting in the final report will
consist of tabular results of the calculations in Section B.
It is anticipated that testing of the THz technology will take place over 2-3 days during
the week of February 14th, 2011. As such, Battelle will provide technology test data and
associated reference data (including records; data sheets; notebook records) within 2 weeks of
receipt to EPA. The goal of this data delivery schedule is prompt identification and resolution of
any data collection or recording issues.
-------
Verification of Picometrix THz System
QAPP
Page 45 of 52
Version 1.0
Date: 2/23/11
SECTION C
ASSESSMENT AND OVERSIGHT
Cl ASSESSMENTS AND RESPONSE ACTIONS
Every effort will be made in this verification test to anticipate and resolve potential
problems before the quality of performance is compromised. One of the major objectives of this
QAPP is to establish mechanisms necessary to ensure this. The procedures described in this
QAPP, which is peer reviewed by a panel of outside experts, implemented by the technical staff
and monitored by the VTC, will give information on data quality on a day-to-day basis. The
responsibility for interpreting the results of these checks and resolving any potential problems
resides with the VTC, who will contact the Battelle AMS Center Manager, Battelle AMS Center
QM, EPA AMS Center Project Officer, and EPA AMS Center QM if any deviations from the
QAPP are observed. The VTC will describe the deviation in a teleconference or by email, and
once a path forward is determined and agreed upon with EPA, the deviation form will be
completed. Technical staff have the responsibility to identify problems that could affect data
quality or the ability to use the data. Any problems that are identified will be reported to the
VTC, who will work with the Battelle Quality Manager to resolve any issues. Action will be
taken to control the problem, identify a solution to the problem, and minimize losses and correct
data, where possible. Independent of any EPA QA activities, Battelle will be responsible for
ensuring that the following audits are conducted as part of this verification test.
Cl.l Performance Evaluation Audits
No independent standard of basis weight is available to provide an independent, certified
standard against which laboratory results can be compared.
C1.2 Technical Systems Audits
The Battelle Quality Manager will perform a technical systems audit (TSA) during this
verification test. The purpose of this audit is to ensure that the verification test is being
performed in accordance with the AMS Center QMP,1 this QAPP, and any Standard Operating
Procedures (SOPs) used by Appleton. Prior to the TSA, a project-specific checklist based on the
-------
Verification of Picometrix THz System
QAPP
Page 46 of 52
Version 1.0
Date: 2/23/11
QAPP, Appleton SOP, and TAPPI requirements will be prepared to guide the ISA. The ISA
will include a review of the test location and general testing conditions; observe the testing
activities; and review test documentation at both the Appleton plant and laboratory. During the
ISA, the Battelle Quality Manager or a designee will specifically review the reference method
procedures, compare actual test procedures to those specified or referenced in this plan, and
review data acquisition and handling procedures.
The Battelle Quality Manager will prepare an initial TSA report and will submit the
report to the EPA QA Manager (with no corrective actions documented) and VTC within 10
business days after completion of the audit. A copy of each final TSA report (with corrective
actions documented) will be provided to the EPA AMS Center Project Officer and Quality
Manager within 20 business days after completion of the audit. At EPA's discretion, EPA QA
staff may also conduct an independent on-site TSA during the verification test. The TSA
findings will be communicated to technical staff at the time of the audit and documented in a
TSA report.
C1.3 Data Quality Audits
The Battelle Quality Manager, or designee, will audit at least 25% of the sample results
data acquired in the verification test and 100% of the calibration and QC data versus the QAPP
requirements. Two audit of data quality (ADQ) will be conducted for this project: data collected
during the test will be audited within 10 business days of receipt by the Quality Manager or
designee and assessed using a project-specific checklist. The synthesized data and verification
report will be audited within 10 business days of receipt of all test data and the report. During
these audits, the Battelle quality manager, or designee, will trace the data from initial acquisition
(as received from the Appleton or the vendor's technology), through reduction and statistical
comparisons, to final reporting. Data must undergo a 100% validation and verification by
technical staff (i.e., VTC, or designee) before it will be assessed as part of the ADQ. All QC
data and all calculations performed on the data undergoing the audit will be checked by the
Battelle Quality Manager or designee. Results of each ADQ will be documented using the
checklist and reported to the VTC and EPA within 10 business days after completion of the
audit. The final ADQ will assess overall data quality, including accuracy and completeness of
the technical report.
-------
Verification of Picometrix THz System
QAPP
Page 47 of 52
Version 1.0
Date: 2/23/11
C1.4 QA/QC Reporting
Each audit will be documented in accordance with Sections 3.3.4 and 3.3.5 of the AMS
Center QMP.1 The results of the audits (both ISA and ADQ) will be submitted to EPA within
10 business days as noted above. Audit reports will include the following:
• Identification of any adverse findings or potential problems
• Response to adverse findings or potential problems
• Recommendations for resolving problems
• Confirmation that solutions have been implemented and are effective
• Citation of any noteworthy practices that may be of use to others.
C2 REPORTS TO MANAGEMENT
During the laboratory evaluation, any QAPP deviations will be reported immediately to
EPA. The Battelle Quality Manager and/or VTC, during the course of any assessment or audit,
will identify to the technical staff performing experimental activities any immediate corrective
action that should be taken. A summary of the required assessments and audits, including a
listing of responsibilities and reporting timeframes, is included in Table 4. If serious quality
problems exist, the Battelle Quality Manager will notify the AMS Center Manager, who is
authorized to stop work. Once the assessment reports have been prepared, the VTC will ensure
that a response is provided for each adverse finding or potential problem and will implement any
necessary follow-up corrective action. The Battelle Quality Manager will ensure that follow-up
corrective action has been taken. The QAPP and final report are reviewed by the EPA AMS
Center Quality Manager and the EPA AMS Center Project Officer. Upon final review and
approval, both documents will then be posted on the ETV website (www.epa.gov/etv).
-------
Table 4. Summary of Assessment Reports1
Verification of Picometrix THz System
QAPP
Page 48 of 52
Version 1.0
Date: 2/23/11
Assessment
ISA
ADQ
(raw data)
ADQ
(Synthesized
data and
verification
report)
Prepared
By
Battelle
Battelle
Battelle
Report Submission
Timeframe
TSA response is due to QM
within 10 business days
TSA responses will be
verified by the QM and
provided to EPA within 20
business days
ADQ will be completed
within 10 business days after
receipt of first data set
ADQ will be completed
within 10 business days after
completion of the
verification report review
Submitted To
EPA ETV AMS Center
EPA ETV AMS Center
EPA ETV AMS Center
Any QA checklists prepared to guide audits will be provided with the audit report.
-------
Verification of Picometrix THz System
QAPP
Page 49 of 52
Version 1.0
Date: 2/23/11
SECTION D
DATA VALIDATION AND USABILITY
Dl DATA REVIEW, VERIFICATION, AND VALIDATION REQUIREMENTS
The key data review requirements for the verification test are stated in Section BIO of
this QAPP. In general, the data review requirements specify that the data generated during this
test will be reviewed by a Battelle technical staff member within two weeks of data generation.
The reviewer will be familiar with the technical aspects of the verification test, but will not be
the person who generated the data. This process will serve both as the data review and the data
verification, and will ensure that data have been recorded, transmitted, and processed properly.
The data validation requirements for this test involve an assessment of the quality of the
data relative to the DQIs and Measurement Quality Objectives for this test referenced in Table 2.
Any deficiencies in these data will be flagged and excluded from any statistical comparisons,
unless these deviations are accompanied by descriptions of their potential impacts on the data
quality.
D2 VERIFICATION AND VALIDATION METHODS
Data verification is conducted as part of the data review, as described in Section BIO of
this QAPP. A visual inspection of handwritten data will be conducted to ensure that all entries
were properly recorded or transcribed and that any erroneous entries were properly noted (i.e.,
single line through the entry with an error code and the initials of the recorder and date of entry).
Electronic data from the technologies and other instruments used during the test will be inspected
to ensure proper transfer from the data logging system. Data manually incorporated into
spreadsheets for use in calculations will be checked against handwritten data to ensure that
transcription errors have not occurred. All calculations used to transform the data will be
reviewed to ensure the accuracy and the appropriateness of the calculations. Calculations
performed manually will be reviewed and repeated using a handheld calculator or commercial
software (e.g., Excel). Calculations performed using standard commercial office software (e.g.,
Excel) will be reviewed by inspecting the equations used in calculations and verifying selected
calculations by handheld calculator. Calculations performed using specialized commercial
-------
Verification of Picometrix THz System
QAPP
Page 50 of 52
Version 1.0
Date: 2/23/11
software (i.e., for the nuclear and THz gauges) will be reviewed by inspection and, when
feasible, verified by handheld calculator, or standard commercial office software.
To ensure that the data generated from this test meet the goals of the test, a number of
data validation procedures will be performed. Sections B and C of this QAPP provide a
description of the validation safeguards employed for this verification test. Data validation and
verification efforts include the completion of QC activities and the performance of TSA as
described in Section C. The data from this test will be evaluated relative to the measurement
DQIs described in Section A4 and B5 of this QAPP. Data failing to meet these criteria will be
flagged in the data set and not used for evaluation of the technologies, unless these deviations are
accompanied by descriptions of their potential impacts on the data quality.
A data quality audit will be conducted by the Battelle Quality Manager to ensure that data
review, verification, and validation procedures were completed, and to assure the overall data
quality.
D3 RECONCILIATION WITH USER REQUIREMENTS
The purpose of this verification test is to evaluate the performance of a commercial
technology that measures basis weight (mass per unit area). This test evaluates the non-
radioactive source THz technology capabilities. This evaluation will include comparisons of the
results from the technology to results from the standard laboratory technique, which is being
used as the reference method for this test and the results of a nuclear gauge. To meet the
requirements of the user community, the data obtained in such a verification test will include
thorough documentation of the technology's performance during the verification test. The data
review, verification, and validation procedures described above will assure that verification test
data meet these requirements, are accurately presented in the verification reports generated from
the test, and that data not meeting these requirements are appropriately flagged and discussed in
the verification reports. Additionally, all data generated using the reference method, which are
used to evaluate technology results during the verification test, should meet the QA requirements
of any applicable standard operating procedures or instrumentation instruction manuals.
This QAPP and any resulting ETV verification report generated following procedures
described in this QAPP will be subjected to review by the participating technology vendor, ETV
AMS Center staff, test collaborators, EPA, and external expert peer reviewers. These reviews
-------
Verification of Picometrix THz System
QAPP
Page 51 of 52
Version 1.0
Date: 2/23/11
will assure that this QAPP, verification test, and the resulting report meet the needs of potential
users and regulators. The final report will be submitted to EPA in 508 compliant Adobe Portable
Document Format (pdf) and subsequently posted on the ETV website.
-------
Verification of Picometrix THz System
QAPP
Page 52 of 52
Version 1.0
Date: 2/23/11
SECTION E
REFERENCES
El REFERENCES
1. Quality Management Plan for the ETV Advanced Monitoring Systems Center, Version
7.0, U.S. EPA Environmental Technology Verification Program, Battelle, Columbus,
Ohio, November 2008.
2. Environmental Technology Verification Program Quality Management Plan, EPA/600/R-
08/009, U.S. Environmental Protection Agency, Cincinnati, Ohio, January 2008.
3. Appleton Standard Test Method, "Basis Weight - Lab Determination of Coated and
Uncoated Paper," Spec. No. 10001.00, Revision 1, July 1993.
------- |