Response to Findings RTI International Audit Report 40 Code of Federal Regulations (CFR) Part 58 Technical Systems Audit (TSA) of Clean Air Status and Trends Network (CASTNET) Program Ozone Monitoring Process December 20, 2012


December 20, 2012

Response to Findings:

RTI International Audit Report -

40 Code of Federal Regulations (CFR)

Part 58 Technical Systems Audit (TSA)
of Clean Air Status and Trends Network (CASTNET)

Program
Ozone Monitoring Process

Prepared by:

AMEC Environment & Infrastructure, Inc.

Prepared for

U.S. Environmental Protection Agency (EPA),
Clean Air Markets Division

-------
Response to Findings: December 20, 2012
RTI International Audit Report — 40CFR, Part 58 Technical Systems Audit ofCASTNET Ozone Monitoring Process

Below please find the AMEC response to findings documented in the RTI International (RTI)
report for the technical systems audit performed for the Clean Air Status and Trends Network
(CASTNET) ozone laboratory facility and two remote monitoring sites between August and
October 2012. Responses are organized according to sections that list findings in the RTI report.

Section 2: General Program

FINDING 1:

During the site visits to CND125 and BFT142 [in North Carolina], the site operators could not provide
the RTI auditor an electronic or hard copy of the current QAPP or Field Operation SOPs.

Discussion:

Based on conversation (by phone) with Mr. Michael Smith [Assistant Field Operations Manager], hard
copies of the Field SOPs (CASTNET QAPP Appendix 1) and Health and Safety Plan (CASTNET QAPP
Appendix 5) are sent to the site operators annually. Appendix 5 is sent with a signature approval form for
the site operators to sign and date and send back to AMEC.

During the laboratory visit, Mr. [Marcus] Stewart [Quality Assurance Manager] gave the auditor a copy
of a memorandum dated April 8, 201 lthat was sent to site operators with a package that included:

• A CD of the Health and Safety Plan. Site Operator Handbook, and SOP for ozone air
monitoring

• Signature pages of acknowledgement for site operator to sign for safety plan and destruction
for obsolete SOPs.

He also provided a table showing the site operators and backup operators have signed and returned the
signature pages to AMEC. The RTI auditor was also given a copy of each site operator's signature
approval for the CND125 and BFT142.

RECOMMENDATION:

There seems to be some confusion based on the discussions with the site operators regarding the signature
approval page that is returned to AMEC. The signature approval page is a good idea for documentation,
but it does not solve the problem that the site operators do not have or could not provide the RTI auditor
with the current quality documents (QAPP and field SOPs). RTI recommends that AMEC develop a
reviewing mechanism to be performed during the 6-month field calibration or the annual performance
evaluation (PE) audit conducted by EPA subcontractor, EE&MS, wherein the calibrator or auditor request
the current QAPP and field SOPs from the site operator. This mechanism can be performed through a
checklist that is signed and dated by the auditor/calibrator and site operators and submitted with the
calibration/audit report to the AMEC QA Manager for filing.

AMEC RESPONSE:

The acknowledgement pages signed most recently by the site operators included acknowledgment for all
documents including handbook and SOPs (see Figure 1).

Page 2 of 8

-------
Response to Findings: December 20, 2012
RTI International Audit Report — 40CFR, Part 58 Technical Systems Audit ofCASTNET Ozone Monitoring Process

figure 1

AMEC will ensure that EPA-sponsored monitoring sites are systematically inspected for the presence of
obsolete project documents during 6-month field calibration visits. Obsolete documents found will be
removed. Current documents will be supplied if they are not present. Formal documentation of site
inspection and of provision or removal of documents (if required) will be completed for each site visited.
Additionally, site operators will be trained to find and utilize current site documentation. This training
will be formally documented. This process will be completed by the end of September 2013.

FINDING 2:

Section 4 of the Quality Assurance Handbook for Air Pollution Measurement Systems, Volume II,
December 2008 (QA Handbook) discusses the need for a formalized training program. During the field
site visits, no site operator could provide training records documenting their satisfactory completion of
training on the current ozone collection system or any evidence of a follow up training program for senior
staff (site operators) for new instrumentation (Thermo 49i ozone analyzer). Section 2.3 of the CASTNET
QAPP states:

lA record of the training is established in the site logbook. If needed, refresher training is given
during the biannual calibration and maintenance visits.

Site operators received additional support and training during the Tuesday call to the FOM,
during each biannual calibration visit, and through technical tips and informative articles
provided by the CASTNET site operator newsletter, which is delivered electronically two to three
times per year"

The SOP titled "Site Selection Procedures, Site Installation, Initiation, and Operator Training" states
there is a training seminar onsite (neither of these site operators attended) and the Station Initiation Team
performs follow up training with the site operator. Currently, there is no continual training when
instrumentation changes or new site operators/backup operators begin conducting the field work.

This could lead to two problems. First, the site logbooks are either not maintained at the site due to
storage space (BFT142) or are shipped to the AMEC Field Operations Laboratory when completely filled
(the CND125 site had site logbooks onsite dated back to 2004). In either scenario, there is no permanent
record maintained at the site to provide evidence for an auditor that the site operators were properly
trained or were provided follow up training for any job deficiencies or new instrumentation brought
online such as the Thermo 49i ozone analyzer. A second problem is not having the documentation at the
AMEC Laboratory (QA Manager) that the site operators have received the necessary training and can
provide evidence of the training to an auditor.

Page 3 of 8

-------
Response to Findings: December 20, 2012
RTI International Audit Report — 40CFR, Part 58 Technical Systems Audit ofCASTNET Ozone Monitoring Process

Discussion:

Based on conversation with Mr. [Kevin] Mishoe [Field Operations Manager] and Mr. Smith, the site
operators are given instructions during the initial site set up or when equipment change outs occur. When
an issue or problem occurs at the site, the site operators contact Mr. Mishoe and Mr. Smith by telephone
(each site has a phone line) and they provide technical assistance to the site operators and document the
issues/problems and corrective action in the Call Log. But if the phone system is down the inexperienced
site operators that are not properly trained are left to try and solve the problem on their own, or have gaps
in data acquisition until Mr. Mishoe or Mr. Smith are reached.

Based on conversations with Mr. Stewart, in the CASTNET QAPP Appendix I, Section III, Subsection
6.14, the 6-month field technician is also supposed to assist with providing training information and
explanations. At the CND125 site, the site operators stated they leave when the field technician
(calibrator) arrives and at the BFT142 site the site operator wants to learn more and stays during the
calibration.

There was a contrast in understanding the basic operations of the Thermo 49i ozone analyzer. At the
CND125 monitoring site, the site operators had difficulty explaining the basic operations of the Thermo
49i ozone analyzer. The day-to-day operations seem to be functioning as long as there are no issues or
problems. At the BFT142 site, the site operator was more knowledgeable of the Thermo 49i analyzer
because he also participates in a NOy study using the Thermo 42i oxides of nitrogen analyzer. At neither
site could the site operator assist the RTI auditor to download any raw ozone data from the laptop.
Downloading of electronic data is not part of the normal activities performed by the site operators, but it
could be useful particularly when being audited by a third party.

RECOMMENDATION:

AMEC needs to develop a mechanism (tracking system) to ensure that all site operators have been trained
(training method and date trained); provide a certificate (electronic on the site operator's laptop or hard
copy) of training completion that the site operator can maintain at the site; and a training record log
(electronic or hard copy) maintained by the QA Manager. This mechanism can also be used to include
follow up training. A training program will ensure that new staff are proficient and that existing staff are
able to keep skills current, to learn of emerging technologies and capabilities. The training program
should also include the requirements of field operators stated in the current QAPPs and SOPs. To assess
the capabilities of the site operators, a checklist could be prepared by the QA Manager addressing
questions relating to the field operations at the site. During the 6-month calibration, the calibrator could
discuss the checklist with the site operators and assess their field operation knowledge. The results of the
checklist should be maintained as part of the site operator's training records. Any previous training
performed on the phone by Mr. Mishoe or Mr. Smith can be added to the checklist. The overall training
record package for the field operators should be maintained by the QA Manager, Mr. Stewart and
available for internal and external review.

AMEC RESPONSE:

The proficiency of operators of EPA sponsored sites will be evaluated via questionnaire during the first
quarter of 2013. Deficiencies observed will be addressed through targeted training. Training certificates
denoting trainer, trainee, area(s) of training and date of training will be provided to document satisfactory
completion of their questionnaire and/or follow up training. Copies of the certificates will be kept on file
at AMEC. This process will be completed by the end of September 2013 and will be repeated annually.

FINDING 3:

Page 4 of 8

-------
Response to Findings: December 20, 2012
RTI International Audit Report — 40CFR, Part 58 Technical Systems Audit ofCASTNET Ozone Monitoring Process

During the field site visits, the auditor found several obsolete checklists and field SOPs at each site. In
some cases, the site operator identified these checklist/SOPs as the procedure that they followed for
sample removal and documentation.

In Table 1-2 of the CASTNET QAPP, the RTI auditor could not determine who was responsible for the
removal of obsolete documents from the monitoring sites. Section 1.7.6 of the QAPP discusses updating,
distribution, version control, and archiving, but there is no discussion for handling obsolete documents
(field or in the laboratory).

Discussion:

During the laboratory visit, Mr. Stewart gave the auditor a copy of a memorandum dated April 8, 2011
that was sent to site operators with an enclosed package that included:

• A CD of the Health and Safety Plan, Site Operator Handbook, and SOP for ozone air monitoring

• Signature pages of acknowledgement for site operator to sign for safety plan and destruction for
obsolete SOPs.

RECOMMENDATION:

Based on the discussion with Mr. Stewart, the site operator is given directions to remove and destroy
obsolete documentation, but there seems to be some confusion based on conversations with the site
operators of the two sites visited regarding the destruction of obsolete SOPs. RTI recommends that
AMEC develop a reviewing mechanism that during the 6-month field calibration or the annual PE audit
conducted by EE&MS, the calibrator or auditor should inspect that obsolete SOPs or checklists have been
removed from the site. This mechanism can be part of the checklist discussed in Findings 1. Also,
describe in the CASTNET QAPP, the process and who is responsible for removal of obsolete document at
the monitoring sites and the AMEC laboratories.

AMEC RESPONSE:

The Quality Assurance Manager (QAM) ensures distribution of updated SOPs and checklists to the
AMEC laboratory. The QAM also ensures the removal of obsolete documents from the laboratory. This
will be documented in revision 8.1 of the CASTNET QAPP.

Section 4: Field Operations

FINDING 1:

There is an Installation/Implementation Checklist for EPA regulatory Ozone Monitoring (see Appendix
F) that describes the steps for installing and implementing the ozone monitoring at the field site. But there
are no place marks for the site operator's signature and date nor are there any checklist statements

Page 5 of 8

-------
Response to Findings: December 20, 2012
RTI International Audit Report — 40CFR, Part 58 Technical Systems Audit ofCASTNET Ozone Monitoring Process

showing training of the site operator.

As stated in the CASTNET Appendix 1 Field Standard Operating Procedures (Section 6.5.1 of Section I
Site Selection Procedures, Subsection A Site Installation, Initiation, and Operator Training).

It is essential that the site operator(s) be onsite during the configuration and installation of the
equipment. The system as a whole is covered in detail during installation. The site operator's
assistance expedites the initiation process and provides valuable training. During such onsite
assistance, the Station Initiation Team reviews all phases of training with the site operator.
Following training, the Station Initiation Team requires the site operator to perform all site
tasks as though routine operations were underway.

Discussion:

In recent years, there have been only a few new installations of CASTNET sites. Thus, there was no
completed Installation/Implementation Checklist for EPA regulatory Ozone Monitoring to review during
the audit; only the blank copy shown in Appendix F.

RECOMMENDATION:

If training of the site operator occurs during the installation/implementation process, a statement should
be added to this checklist describing the training and place marks for the site operator's signature and
date. The signature of the trainer and date training performed should also be included. All information
regarding training should be added to the site operator's training record.

AMEC RESPONSE:

FINDING 2:

At the CND125 monitoring site, neither of the ozone analyzers were connected to the uninterruptable
power supply and both desiccant canisters need to be replaced with desiccant. When the RTI auditor
connected the uninterruptable power supply to a receptacle, there was no indication that the supply was
working.

Discussion:

The site operator stated that she would advise the Field Calibration Laboratory that she needs more
desiccant. Regarding the uninterruptable power supply, there was no discussion.

RECOMMENDATION:

RTI recommends for the AMEC staff at the Field Calibration Laboratory replace the uninterruptable
power supply at this site. The RTI auditor followed up with the site operator's request for more desiccant
from the Field Calibration Laboratory and the site operator has made the request.

AMEC RESPONSE:

The uninterruptable power supply (UPS) at CND125 will be evaluated during the next site calibration
visit and replaced if necessary.

The UPS at CASTNET sites are routinely tested during 6-month calibrations and removed from service if

Page 6 of 8

-------
Response to Findings: December 20, 2012
RTI International Audit Report — 40CFR, Part 58 Technical Systems Audit ofCASTNET Ozone Monitoring Process

not properly functioning. The Field Operations Manager or his designee is notified to facilitate decision to
repair or replace the unit.

Desiccant consumption is monitored each Tuesday during site operator visits and reported to an AMEC
Field Calibration Laboratory representative. The percentage of unspent material (level of blue color vs
pink) is recorded in the site call-in log (see Figure 2). Desiccant is replaced when overall consumption
reaches 50 percent for sites equipped with single canisters. Sites equipped with dual canisters of desiccant
may refill when the percentage of unspent material in the upstream canister reaches 25 percent (the
downstream canister will exhibit zero consumption in this case).

Figure 2

Site' CND125 Problem Status: {OPEN Observations j Site Operator j i j Close

Section 6: Data and Data Management

FINDING 1:

There was difficulty locating the "AIRNow website" cited in various places in the CASTNET QAPP, for
example, QAPP Sec. 2.5.2 states (emphasis added):

"... Each CASTNET site is polled hourly to retrieve hourly averages and status files. 03, meteorological,
and flow data are reviewed daily by data operations personnel as part of the data validation process
(Section 4.0). For sites with EPA supplied CR3000 data logger and 49i ozone analyzers, data are polled
hourly with Campbell's LoggerNet and uploaded to the EPA AIRNow Web site.. ."

Based on this information, the auditor initially searched the public AIRNow website http://airnow.gov/

which did not reveal any data for the two audited CASTNET sites. After an inquiry by RTI, AMEC

provided the full address of the EPA FTP site to which these hourly data are uploaded:

ftp:/upload.epa.gov/incoming/CASTNET/data where data for the two audited sites were successfully

located.

RECOMMENDATION:

Update the CASTNET QAPP and relevant SOPs to provide the specific EPA Internet address(es) to
which the CASTNET data are being uploaded. It is also recommended that the QAPP and/or SOPs also
identify the group(s) at EPA responsible for maintaining the site to which these data are being uploaded
(e.g., AQS staff, CAMD staff, Air Now staff, etc.). Having the latter information could help future
auditors and data users obtain potentially useful information about the data storage facility, such as SOPs
and validation procedures, as well as contact information.

AMEC RESPONSE:

Revision 8.1 of the CASTNET QAPP will include the relevant internet addresses.

FINDING 2:

No major discrepancies in data were identified upon comparing data, other than the 1-hour offset that was
easily explained.

Page 7 of 8

-------
Response to Findings: December 20, 2012
RTI International Audit Report — 40CFR, Part 58 Technical Systems Audit ofCASTNET Ozone Monitoring Process	

RECOMMENDATION:

No action required.

AMEC RESPONSE:

As noted above, a satisfactory explanation for the 1-hour offset was provided during the audit. Finding
closed.

FINDING 3:

The outlier in the data that occurred at CND125 on 8/15/12 was detected quickly and was properly
documented.

RECOMMENDATION:

No action required.

AMEC RESPONSE:

As noted above, the outlier was detected quickly and properly documented. Finding closed.

Page 8 of 8

-------