U.S. ENVIRONMENTAL PROTECTION AGENCY
OFFICE OF INSPECTOR GENERAL
Catalyst for Improving the Environment
Evaluation Report
EPA Needs an Oversight Program
for Protocol Gases
Report No. 09-P-0235
September 16, 2009
-------
Report Contributors:
Jim Hatfield
Geoff Pierce
Rick Beusse
Abbreviations
CAA Clean Air Act
CAIR Clean Air Interstate Rule
CEMS Continuous Emissions Monitoring System
CFR Code of Federal Regulations
CO Carbon monoxide
CC>2 Carbon dioxide
EPA U.S. Environmental Protection Agency
GMIS Gas Manufacturer's Intermediate Standard
NAAQS National Ambient Air Quality Standards
NIST National Institute of Standards and Technology
NO Nitric oxide
NOX Nitrogen oxides
NO2 Nitrogen dioxide
NTRM NIST Traceable Reference Material
OAP Office of Atmospheric Programs
OAQPS Office of Air Quality Planning and Standards
OAR Office of Air and Radiation
OIG Office of Inspector General
ORD Office of Research and Development
PGVP Protocol Gas Verification Program
ppm parts per million
SO2 Sulfur dioxide
SRM Standard Reference Material
Cover photo:
Gas cylinders stored at the National Institute of Standards and Technology
laboratory in Gaithersburg, Maryland. (Photo courtesy National Institute of
Standards and Technology)
-------
U.S. Environmental Protection Agency
Office of Inspector General
At a Glance
09-P-0235
September 16, 2009
Catalyst for Improving the Environment
why we Did This Review EPA /Veec/s a/7 Oversight Program for Protocol Gases
The accuracy of continuous
emissions monitors is critical
to the U.S. Environmental
Protection Agency's (EPA's)
Acid Rain Program because
data from these monitors
determine the number of
allowances a utility can bank,
sell, or trade. Specialized
gases known as "EPA
Protocol Gases" are used to
calibrate and assure the quality
of these monitors. We sought
to determine whether certified
concentrations of these gases
are accurate.
Background
Vendors produce and certify
EPA Protocol Gases in high-
pressure cylinders according
to EPA procedures. EPA
regulations require the use of
these gases, or National
Institute of Standards and
Technology (NIST)-certified
reference materials, when
conducting quality assurance
for continuous emissions and
ambient air monitoring
systems. EPA's Acid Rain
Program requires that the
certified concentration of the
I gases be within ± 2 percent of
the true concentration.
For further information, contact
our Office of Congressional,
Public Affairs and Management
at (202) 566-2391.
To view the full report,
click on the following link:
www.epa.qov/oiq/reports/2009/
20090916-09-P-0235.pdf
What We Found
We purchased 87 cylinders of EPA Protocol Gases and had NIST analyze each
cylinder to determine whether the three gaseous mixtures contained in each
cylinder (carbon dioxide, nitric oxide, and sulfur dioxide) met the Acid Rain
Program's accuracy criterion. We found that 89 percent (233 components) met
the Acid Rain Program's accuracy criterion and 11 percent (28 components) did
not. Of the 28 components that did not meet the criterion, 17 were within 3.0
percent of the NIST-determined true concentration; 7 were within 3.0 to 5.0
percent; and 4 exceeded the true concentration by more than 5.0 percent.
Our sample was not designed to estimate the impact of the test results on the Acid
Rain Program. However inaccurately certified concentrations could cause system
operators to unknowingly calibrate their monitoring systems to record inaccurate
measurements. For example, if a utility overstates its emissions, it could lose the
opportunity to sell allowances to other utilities. If a utility understates its
emissions, the utility and regulators may incorrectly conclude that the source is
complying with emissions standards. With respect to ambient air monitoring, the
accuracy of these monitors is important because the data are used to determine
whether areas are in compliance with the Nation's ambient air quality standards.
EPA has conducted only two tests of the accuracy of EPA Protocol Gases since
1997, when EPA's Office of Research and Development discontinued its annual
testing program. Thus, EPA does not have reasonable assurance that the gases
used to calibrate emissions monitors for the Acid Rain Program and continuous
ambient monitors for the Nation's air monitoring network are accurate.
What We Recommend
We recommend that the Office of Air and Radiation (OAR) implement oversight
programs to assure the quality of EPA Protocol Gases used to calibrate continuous
emissions monitoring systems and ambient air monitors. We also recommend that
the Office of Research and Development (ORD) update and maintain the protocol
gas procedures to ensure that the protocol meets the objectives of the Acid Rain,
ambient air, and stationary source air programs. OAR and ORD concurred with
our recommendations. OAR has initiated efforts to conduct another gas audit later
this year. OAR also plans to propose a rule later in 2009 to establish a largely
self-supported, annual gas audit program of protocol gases used for the Acid Rain
Program. Further, OAR plans to implement a separate verification program to
address the lower concentration protocol gases used to calibrate continuous
ambient air monitors. ORD will update and maintain the protocol gas procedures.
EPA's planned actions meet the intent of our recommendations.
-------
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
OFFICE OF
INSPECTOR GENERAL
MEMORANDUM
SUBJECT:
FROM:
TO:
September 16, 2009
EPA Needs an Oversight Program for Protocol Gases
Report No. 09-P-0235
tt^ ^J^-
Wade T. Najjum A-i
Assistant Inspector General for Program Evaluation
Gina McCarthy
Assistant Administrator for Air and Radiation
Lek G. Kadeli
Acting Assistant Administrator for Research and Development
This is our report on the subject evaluation conducted by the Office of Inspector General (OIG)
of the U.S. Environmental Protection Agency (EPA). This report contains findings that describe
the problems the OIG has identified and corrective actions the OIG recommends. This report
represents the opinion of the OIG and does not necessarily represent the final EPA position.
Final determinations on matters in this report will be made by EPA managers in accordance with
established audit resolution procedures.
The estimated cost of this report - calculated by multiplying the project's staff days by the
applicable daily full cost billing rates in effect at the time, including the costs of purchasing EPA
Protocol Gases and having them analyzed by the National Institute of Standards and Technology
-is $665,846.
Action Required
In accordance with EPA Manual 2750, EPA 's Audit Management Process, the Office of Air and
Radiation should provide a written response within 90 calendar days. The Office of Air and
Radiation's response should include a corrective action plan and planned completion dates for
Recommendations 2-1 and 2-2. The Office of Research and Development submitted a corrective
action plan that sufficiently addresses Recommendation 2-3. As such, we are "closing"
Recommendation 2-3 in our tracking system upon issuance of this report. These
recommendations will be tracked to completion in the Agency's tracking system. No further
-------
response is required for Recommendation 2-3. As outlined in EPA Manual 2750, the Agency is
responsible for tracking the implementation of these actions in its Management Audit Tracking
System. We have no objections to the further release of this report to the public. This report will
be available at http://www.epa.gov/oig.
If you or your staff have any questions regarding this report, please contact me at (202) 566-0832
or najjum.wade@epa.gov, or Rick Beusse at (919) 541-5747 or beusse.rick@epa.gov.
-------
EPA Needs an Oversight Program for Protocol Gases 09-P-0235
Table of Contents
Chapters
1 Introduction 1
Purpose 1
Background 1
Noteworthy Achievements 5
Scope and Methodology 5
2 EPA Needs an Oversight Program to Assure the Accuracy of
Protocol Gases 7
EPA Does Not Have an Oversight Program to Assure the
Accuracy of Protocol Gases 7
Low Concentration NO and SO2 Gases and Lower-Cost Cylinders
Performed Worst in OurTests 8
Calibration Gases Play an Important Role in Assuring Data Quality
for EPA Air Programs 10
Conclusions 13
Recommendations 13
Agency Comments and OIG Evaluation 13
Status of Recommendations and Potential Monetary Benefits 15
Appendices
A Details on Scope and Methodology 16
B Test Results with Uncertainty Factored into Analyses 18
C OAR's Response to OIG Draft Report 20
D ORD's Response to OIG Draft Report 24
E Distribution 27
-------
09-P-0235
Chapter 1
Introduction
Purpose
The U.S. Environmental Protection Agency (EPA) Protocol Gases are used to
calibrate and test continuous emissions monitoring systems (CEMS) and ambient
air quality monitoring systems. EPA requires the use of CEMS for large utilities
covered by the Acid Rain Program and other clean air programs. Consequently,
the accuracy of the EPA Protocol Gases is critical to ensuring the integrity of
emissions trading and other EPA programs. We conducted this evaluation to
determine whether the certified concentrations of EPA Protocol Gases were
within acceptable limits for accuracy.
Background
The integrity and effectiveness of EPA's air programs depend upon collecting
accurate pollutant emissions and air quality data. EPA requires that the systems
collecting these data periodically undergo certain quality assurance procedures to
assure their accuracy. These procedures include daily calibration and accuracy
tests of the monitoring systems. Calibration tests check the accuracy of these
monitoring systems by running a gas mixture of a known concentration through
the systems and comparing the systems' readings with the known concentration.
The systems are adjusted accordingly to eliminate measurement inaccuracies.
The certified concentrations of gaseous mixtures used to calibrate and test
monitoring systems should accurately reflect the true gaseous concentration. EPA
regulations require that the gases used for calibrating and testing the accuracy of
ambient air quality analyzers and continuous emissions monitors1 be traceable to
either a National Institute of Standards and Technology (NIST) Standard
Reference Material (SRM) or a NIST Traceable Reference Material (NTRM).
EPA has established an EPA Traceability Protocol that specialty gas producers
must follow when preparing calibration gases to meet the NIST traceability
requirement. The gases produced in accordance with these standards are referred
to as EPA Protocol Gases.
This protocol was designed by EPA, NIST, and specialty gas producers so that a
laboratory analyst can analyze and certify compressed gas calibration standards
with low uncertainty and with traceability to NIST. The protocol specifies a
general analytical procedure for determining the concentration of the standards,
but it does not specify the analytical instrumentation or the gas-handling
1 Continuous emissions monitors continuously measure pollutants emitted into the atmosphere in exhaust gases from
combustion or industrial processes.
-------
09-P-0235
apparatus that the analyst must use during the analysis. In summary, the protocol
requires that (1) a NIST-certified gaseous mixture be used as the analytical
reference standard, (2) a monthly calibration curve for the instrumentation be
established, and (3) both the NIST-certified gaseous mixture and the candidate
EPA Protocol Gas be measured at least three times, with reactive gas mixtures
measured at least three additional times at least 7 days after the first set of
measurements.
The protocol has detailed statistical procedures for estimating the total uncertainty
of these standards. Although it includes procedures for estimating uncertainty,
the protocol does not specify that EPA Protocol Gases meet any given level of
uncertainty and does not establish specific accuracy criteria for the standards'
uncertainty. However, the Acid Rain Program has prescribed an uncertainty
standard for EPA Protocol Gases used to calibrate CEMS as described in the
following section.
Acid Rain Program Requirements for EPA Protocol Gases
The Acid Rain Program implements Title IV of the Clean Air Act (CAA) to
reduce emissions of sulfur dioxide (SO2) and nitrogen oxides (NOX), the primary
causes of acid rain. The program uses an allowance trading system to reduce 862
emissions. Under this system, affected utility units are allocated allowances (one
allowance = one ton of 802) that permit a unit to emit SC>2 during or after a
specified year. Utilities can buy, sell, or bank allowances. The program began in
1995. For each ton of 862 emitted in a given year, one allowance is retired and
can no longer be used. The program is phased in, with the final SC>2 cap set at
8.95 million tons in 2010. The Acid Rain Program does not use a trading system2
to reduce emissions for NOX, but instead establishes emissions limits for covered
units that are designed to achieve the program's goal of reducing NOX emissions
by 2 million tons from 1980 levels.
Continuous emissions monitoring is instrumental in ensuring that the mandated
reductions of 862 and NOX under the Acid Rain Program are achieved.
Accordingly, EPA has established requirements for continuously monitoring SO2
and NOX (and other parameters) for units regulated under the Acid Rain Program.
Appendices A and B to 40 Code of Federal Regulations (CFR) Part 75 describe
specification and test procedures for these monitors, which include daily
calibration tests and periodic accuracy tests. The regulations require that the
calibration gases used to conduct these tests must be NIST-certified reference
standards or that their concentrations must be traceable to NIST-certified
reference standards according to the EPA Traceability Protocol for Assay and
Certification of Gaseous Calibration Standards. In January 1993, EPA's Office
of Atmospheric Programs (OAP) included a 2.0 percent accuracy criterion in its
final Acid Rain Program emissions monitoring rule (58 Federal Register 3731).
2 NOX emissions are traded under other air programs, including the NOX Budget Trading Program, which is designed
to reduce NOX emissions in order to reduce ambient ozone levels.
-------
09-P-0235
Appendix A to 40 CFR Part 753 specifies that the EPA Protocol Gases must have
a certified uncertainty (95 percent confidence interval) that must not be greater
than plus or minus (±) 2.0 percent of the certified concentration (tag value) of the
gas mixture.
In January 2008, EPA revised the Acid Rain Cap and Trade Program regulations
to require that producers of EPA Protocol Gases participate in EPA's Protocol
Gas Verification Program in order to certify their gases as EPA Protocol Gases.
National Ambient Monitoring Program Requirements for EPA
Protocol Gases
Appendix A to 40 CFR Part 584 describes quality assurance procedures for
national ambient air monitors. States and EPA use these monitors to determine
whether an area's air quality meets or exceeds the National Ambient Air Quality
Standards (NAAQS). The quality control procedures include biweekly one-point
checks of carbon monoxide (CO), 862, and nitrogen dioxide (NC^), and annual
performance evaluations that test the monitors against gases of known
concentrations for at least three different ranges of concentrations. Appendix A to
this regulation requires that cylinders of compressed gas used to obtain test
concentrations must be traceable to either a NIST Traceable Reference Material
or a Gas Manufacturer's Intermediate Standard, certified in accordance with
EPA's Traceability Protocol for Assay and Certification of Gaseous Calibration
Standards. Further, vendors advertising certification with the procedures and
distributing gases as "EPA Protocol Gas" must participate in the EPA Protocol
Gas Verification Program or not use "EPA" in any form of advertising.
Unlike the Acid Rain Program, the ambient monitoring regulations do not specify
accuracy criteria for the EPA Protocol Gases used to calibrate ambient air
monitors.
Stationary Source Performance Standards Requirements for EPA
Protocol Gases
The CAA required EPA to establish emissions standards5 for certain stationary
sources that cause or contribute significantly to air pollution. These standards are
intended to promote use of the best air pollution control technologies and apply to
sources that have been constructed or modified since the proposal of the standard.
These standards are published in 40 CFR Part 60, and may require the source to
install CEMS to demonstrate compliance.
3 Appendix A to Part 75—Specifications and Test Procedures.
4 Appendix A to Part 58—Quality Assurance Requirements for State and Local Air Monitoring Stations, Special
Purpose Monitors, and Prevention of Significant Deterioration Air Monitoring.
5 Referred to as New Source Performance Standards.
-------
09-P-0235
Part 60 contains varied requirements for using calibration gases depending upon
the EPA method used to conduct emissions testing. Specifically, the methods for
determining SC>2 (Method 6C) and NOX (Method 7E) emissions from stationary
sources require that the calibration gases be prepared according to the EPA
protocol or certified by the tester using a specified wet chemical test method. The
electric utility, petroleum, and municipal waste incineration industries are the
largest industries (in terms of emissions and the number of sources) subject to
continuous emissions monitoring requirements and the use of EPA Protocol Gases
for Part 60 emissions standards.
EPA's Protocol Gas Verification Program
From 1985 to 1997, EPA's Office of Research and Development (ORD)
conducted a series of analytical tests6 of EPA Protocol Gases sold by specialty gas
producers. In 1996, ORD issued a final draft exit strategy that transferred the
administration of this and other quality assurance programs to the Office of Air
Quality Planning and Standards (OAQPS) of the Office of Air and Radiation
(OAR). ORD provided OAR with resource support during the transition, but
provided no further resources after the transfer was completed in 1998. The
annual tests were discontinued after 1997, and no additional tests occurred until
2003. Due to a lack of analytical tests and a concern that EPA Protocol Gas
quality may have declined, the OAP conducted a test in 2003. OAP's contractor
purchased 42 EPA Protocol Gas cylinders containing tri-blend mixtures of carbon
dioxide (CO2). nitric oxide (NO), and SO2 from a total of 14 specialty gas vendors
nationwide. A third party purchased the cylinders for EPA so that the gas vendors
would not know that EPA was analyzing the cylinders. A gas passed the test if
the "true" concentration of the gas differed from the certified concentration by ± 2
percent or less. Overall, 89 percent of the gases passed on a gas component basis;
11 percent did not pass the test.
OAP conducted another test in 2006 but decided not to publish the results because
of concerns about the representativeness of some of the sample gases. Instead of
purchasing gases directly from vendors, OAP arranged for end users (i.e.,
utilities) to provide EPA with the Protocol Gases for testing. However, EPA
encountered several problems with this approach, including over- and
under-representation of manufacturers and production sites, as well as old and
used samples.
Approximately 6 years ago, EPA started developing an ongoing EPA Protocol
Gas Verification Program (PGVP). After a development process involving U.S.
specialty gas companies, NIST, the Institute of Clean Air Companies, and EPA,
tentative agreement was reached on a specialty gas company-funded, ongoing
draft PGVP. EPA promulgated a requirement for producers to participate in this
6 EPA calls these tests EPA Protocol Audits. We refer to these audits as tests in this report to avoid confusion with
the term "audit" as defined by the Government Accountability Office's Government Auditing Standards, and
applicable to work conducted by the Office of Inspector General (OIG).
-------
09-P-0235
program in a January 24, 2008, final rule. However, in March 2008, one specialty
gas vendor petitioned EPA for reconsideration of the final rule. The vendor cited
several concerns including the nature of the funding for the PGVP. Since that
time, EPA has been working to resolve these concerns, but still had not funded the
program as of May 2009. EPA had planned to have EPA Protocol Gas vendors
fund the verification program through payments of fees to a third party.
However, EPA has determined such a process could violate the Miscellaneous
Receipts Act.7
Noteworthy Achievements
Recognizing the importance of EPA Protocol Gases, the OAP conducted two
protocol gas assessments, one in 2003 and another in 2006. Although the sample
size in each of these assessments was not statistically significant and market share
information was not available, both assessments indicated that some specialty gas
vendors were not meeting the EPA Protocol Gas accuracy criterion of ± 2.0
percent. OAP posted the results, including the names of the specialty gas
companies, on an Agency Website. Based on comments by specialty gas
companies, the 2003 accuracy check resulted in some vendors making
improvements in personnel, training, quality, and equipment. When we originally
approached EPA with our proposal to assess the accuracy of EPA Protocol Gases,
OAR supported our plans and cooperated fully with our evaluation team to
develop a useful assessment of the EPA Protocol Gas program.
Scope and Methodology
We conducted this evaluation to test the accuracy of EPA Protocol Gases used to
calibrate continuous emissions monitoring equipment. We designed our sampling
methodology to test at least one set of three EPA Protocol Gas cylinders from
every specialty gas manufacturing location in the country and to maintain the
confidentiality of the tests so that the gases we purchased would represent those
sold to utilities and other customers.8 In order to maintain the confidentiality of
the tests, we contracted with a third party to purchase the gases and deliver them
to NIST for analysis.
Our contractor purchased tri-blend mixtures of NO, CO2, and SO2, since tri-blend
gases are often used by utilities. We identified 11 gas producers nationwide using
18 manufacturing locations to produce EPA Protocol Gases. Our contractor
purchased at least one set of EPA Protocol Gas cylinders produced from each of
the 18 manufacturing locations. In all, we purchased 87 EPA Protocol Gas
cylinders from 14 specialty gas vendors (i.e., 11 producers and 3 distributors), and
7 The Miscellaneous Receipts Act was passed in order to ensure that government agencies did not bypass the
appropriations authority of Congress by augmenting their budgets via other means, such as user fees, fees for
training courses, parking fees, contract and lease fees and revenues, monetary awards in court cases involving the
agencies, court costs and fees, or civil penalties.
8 A set consisted of three cylinders in low-, medium-, and high-range concentrations.
-------
09-P-0235
18 manufacturing locations. Our contractor delivered the sample gases to NIST
for analysis. We selected NIST because of its extensive experience in gas
metrology and its reputation for analytical accuracy. NIST analyzed the gaseous
concentrations of the 87 tri-blend cylinders (i.e., 261 separate component
analyses) to determine whether the "true" concentrations were within ± 2 percent
of the concentration certified by the vendor.9
We reviewed documentation related to EPA's past assessments of EPA Protocol
Gases, including past results, program funding and implementation, and current
plans for implementing a revised EPA Protocol Gas Verification Program. We
reviewed federal regulations to identify requirements for the use of EPA Protocol
Gases. We interviewed staff and managers from ORD, OAQPS, and OAP.
We conducted this evaluation in accordance with generally accepted government
auditing standards. Those standards require that we obtain sufficient, appropriate
evidence to provide a reasonable basis for our findings and conclusions based on
our evaluation objectives. We believe that the evidence obtained provides a
reasonable basis for our findings and conclusions based on our objectives. We
conducted our work from June 2007 to July 2009.
Appendix A provides more detailed information on our scope and methodology.
9 Vendors typically certify gaseous concentrations using either a NIST Traceable Reference Material (NTRM), a
Standard Reference Material (SRM), or a Gas Manufacturer's Intermediate Standard (GMIS). NTRMs are
reference material produced by a commercial supplier with a well-defined traceability linkage to NIST. These
reference materials are allowed to bear the NIST NTRM certification mark and were developed to allow NIST to
respond to increasing needs for high-quality reference materials. SRMs are reference materials produced by NIST
and are certified to have specific chemical or physical properties. A GMIS is a NIST traceable standard produced
by the gas vendor. A NIST traceable standard is certified with an instrument calibrated through direct traceability
with a NIST standard such as an SRM or NTRM.
-------
09-P-0235
Chapter 2
EPA Needs an Oversight Program to
Assure the Accuracy of Protocol Gases
EPA does not have reasonable assurance that EPA Protocol Gases used to
calibrate CEMS for the Acid Rain Program and continuous ambient monitors for
the Nation's air monitoring network are accurate. EPA has conducted only two
assessments since 1997, when ORD discontinued its program to test the accuracy
of EPA Protocol Gases. We analyzed 261 gaseous components and found that
89 percent (233 components) met the Acid Rain Program's accuracy criterion for
calibration gases, while 11 percent (28 components) did not meet the accuracy
criterion. Calibration gases must be accurate because inaccurate calibration gases
can cause a system operator to calibrate a measurement system to produce
inaccurate measurements. In turn, inaccurate CEMS measurements can affect the
integrity of the computed allowances available for trade on the open market.
EPA Does Not Have an Oversight Program to Assure the Accuracy of
Protocol Gases
EPA does not have a viable oversight program in place to assure the quality of
EPA Protocol Gases. Since 1997, EPA oversight to assure the accuracy of these
gases has consisted of two analytical tests conducted by OAP in 2003 and 2006.
However, EPA did not publish the 2006 test results because of concerns over the
sampling methodology. Without an oversight program, EPA does not have
reasonable assurance that EPA Protocol Gases meet the accuracy criterion
certified to by the vendors and required for the Acid Rain Program.
In January 2008, EPA promulgated a final rule requiring EPA Protocol Gas
producers to participate in an EPA PGVP. EPA anticipated implementing a
verification program to assess the accuracy of Protocol Gases on an ongoing
basis. However, implementation was delayed when a specialty gas vendor
petitioned EPA to reconsider the rule. As of May 2009, EPA's OAP was still
working to resolve legal issues raised in the petition and deciding how to fund the
program. As of May 2009, OAR's OAQPS was developing a similar but separate
verification program to test the accuracy of the EPA Protocol Gases used to
calibrate continuous ambient air monitors.
Even though the Acid Rain Program and the ambient air monitoring program
generally use different concentrations of EPA Protocol Gases in their programs,
the gases must be produced and certified under the same EPA Protocol. Although
ORD originally developed the Protocol, a co-author of the Protocol informed us
that it was not clear who currently has responsibility for maintaining the Protocol.
Although the 2003 EPA Protocol Gas assessment report and specialty gas
-------
09-P-0235
producers have identified several analytical issues not addressed by the current
Protocol, the Protocol has not been revised since 1997.
Low Concentration NO and SO2 Gases and Lower-Cost Cylinders
Performed Worst in Our Tests
We purchased and analyzed 87 cylinders of tri-blend EPA Protocol Gases to
determine whether each of the three gaseous concentrations (per cylinder) met the
Acid Rain Program's accuracy criterion. Of the 261 gaseous components
analyzed, 89 percent met the Acid Rain Program's accuracy criterion for EPA
Protocol Gases, while 28 (11 percent) did not meet the accuracy criterion.
Of the 28 components that did not meet the criterion, 17 were within 3.0 percent
of the NIST-determined true concentration; 7 were within 3.0 to 5.0 percent; and
4 exceeded the true concentration by more than 5.0 percent. All exceptions were
for gaseous components from lower-cost cylinders (i.e., under $380). Twelve of
the gas components' "true" concentrations were higher than the certified
concentration, and 16 were lower than the certified concentration.
The gaseous components with the most exceptions were low concentration NO
and SO2. The certified concentrations for low range NO and SO2 gases exceeded
the NIST analyzed or "true" concentration by more than ± 2 percent for 24 and 17
percent of the samples, respectively. The following table summarizes the test
results for each range of gaseous components analyzed.
Table 2-1: Results by Gas and Concentration
Gas
NO
CO2
SO2
Range
Low
Med
High
Low
Med
High
Low
Med
High
Total
No. of
Samples
29
29
29
29
29
29
29
29
29
261
Met the Accuracy
Criterion
No.
22
29
25
26
28
29
24
25
25
233
Percentage
76
100
86
90
97
100
83
86
86
89 a
Did Not Meet the
Accuracy Criterion
No.
7
0
4
3
1
0
5
4
4
28
Percentage
24
0
14
10
3
0
17
14
14
11a
Source: OIG, developed from data in NIST Report of Analysis, December 4, 2008
a Represents the overall percentage, not the column total.
We also assessed the test results on a cost-per-cylinder and production-facility
basis. This analysis revealed that all gaseous components not meeting the
accuracy criterion were from cylinders costing under $380. All of the gaseous
components not meeting the accuracy criterion came from 7 of the 18 (39 percent)
facilities. These seven facilities accounted for 108 of the 261 components we
tested, and 28 of these 108 components did not meet the Acid Rain Program's
-------
09-P-0235
accuracy criterion. All 153 gaseous components associated with cylinders costing
over $380 met the accuracy criterion. The following table shows the total tests
conducted, the cost per cylinder, and the test results for each production facility.
Table 2-2: Test Results by Cost and Production Facility
Production
Facility
A
B
N
D
K
Q
R
Subtota
C
D
E
F
G
H
1
J
L
M
O
P
Cost Per
Cylinder
<380
<380
<380
<380a
<380
<380
<380
<380
>380
>380a
>380
>380
>380
>380
>380
>380
>380
>380
>380
>380
Subtotal >380
Totals
Total
Tests
18
18
18
18
9
18
9
108
9
9
9
9
18
18
9
9
9
18
18
18
153
261
No. of Tests
Meeting the
Accuracy
Criterion
15
17
9
15
7
10
6
80
9
9
9
9
18
18
9
9
9
18
18
18
153
233
No. of Tests
Not Meeting the
Accuracy
Criterion
3
1
9
2
2
8
3
28
0
0
0
0
0
0
0
0
0
0
0
0
0
28
Source: OIG table developed from data in NIST Report of Analysis, December 4, 2008.
3 Three sets of cylinders (a set consists of a low-, a medium-, and a high-range cylinder) were
produced at this location but were purchased from two different vendors. Two sets of cylinders
cost under $380 per cylinder and one set cost over $380 per cylinder.
We further analyzed the test results, taking into account the vendor's analytical
technique and the reference standard used by the vendor to certify its gaseous
concentrations. This analysis showed that low-cost cylinders certified with a Gas
Manufacturer's Intermediate Standard (GMIS) performed the worst in our tests.
Specifically, 24 of the 28 (86 percent) gaseous components that did not meet the
Acid Rain Program's accuracy criterion were from lower-cost cylinders that were
certified using GMIS. Although most of the gaseous components not meeting the
accuracy criterion were certified using GMIS, many GMIS-certified components
did meet the accuracy standard. However, all of these gaseous components were
from cylinders costing over $380. The following table shows the test results by
certification standard and cylinder cost.
-------
09-P-0235
Table 2-3: Pass/Fail Rates By Analytical Reference Standard
Reference Standard
Used and Cost of Gas
Cylinder
NTRM/SRM < $380
NTRM/SRM > $380
GMIS < $380
GMIS > $380
Unknown3 < $380
Totals
No. of Gaseous
Components
Certified with this
Standard
8
103
96
50
4
261
Components Not Meeting
the Acid Rain Program's
Accuracy Criterion
No.
2
0
24
0
2
28
Percentage
25
0
25
0
50
11"
Source: OIG table developed from data in NIST Report of Analysis, December 4, 2008.
a Vendor documentation did not specify type of reference standard used to certify the gaseous
concentration.
b Represents the overall percentage, not the column total.
Calibration Gases Play an Important Role in Assuring Data Quality for
EPA Air Programs
Calibration gases play an important role in helping to assure the quality of data
used in EPA programs. Inaccurate calibration gases can negatively affect EPA
programs that rely upon accurate emissions or ambient air quality measurements
by causing system operators to calibrate their measurements systems inaccurately.
For example, if a calibration gas used by a utility was certified to contain 100
parts per million (ppm) of 862, but only contained 96 ppm, the system operator
would unknowingly calibrate the CEMS to read 96 ppm as 100 ppm. This would
result in the CEMS overestimating emissions. Conversely, if the true
concentration of the calibration gas were more than the certified concentration,
the system operator would unknowingly calibrate the CEMS to underestimate
emissions.
The number of monitoring systems potentially using EPA Protocol Gases is
significant. For example, over 3,500 electric generating units were covered by the
SC>2 allowance trading program in 2007. A subset of almost 1,000 of these 3,500
units was also covered by the NOx provisions of the Acid Rain Program. Further,
approximately 1,300 ambient air monitors collect data on CO, SO2, and NOX
concentrations. An EPA contractor surveyed Protocol Gas use in preparation for
EPA's analytical tests of Protocol Gases in 2003. The contractor estimated that
utilities purchased from 37,500 to 75,000 Protocol Gas cylinders annually and
missions testing companies purchased from 16,500 to 55,000 cylinders annually.
The survey did not estimate annual purchases by State and local agencies, but one
large State reported using about 75 Protocol Gases a year for its ambient
monitoring network.
Due to the nature of our sample selection and a lack of data on vendor market
shares, we were unable to develop a sampling methodology to ensure that the
number of samples selected from each producer represented the number of
cylinders produced and sold to consumers. Consequently, we did not project our
test results to the universe of EPA Protocol Gases, or estimate the impact that
10
-------
09-P-0235
these results could have on the Acid Rain Program. Despite this limitation, our
tests represent the largest one-time assessment of the Protocol Gas industry, and
revealed some potential areas of concern with respect to the quality of less-
expensive gases certified using GMIS.
The following sections describe the importance of accurate monitoring data to the
Acid Rain and Air Monitoring programs. These data are collected by
measurement systems whose accuracy is assured by quality assurance activities
such as calibrations using EPA Protocol Gases.
Calibration Gases Help Assure
Accuracy of Emissions Data for
Trading Programs
For sources participating in emissions
trading programs, overestimating
emissions can result in lost opportunities
to sell or bank allowances.
Underestimating emissions could result
in banking or selling allowances that did
not really exist, and provide an
inaccurate picture of a facility's progress
in reducing emissions.
The volume of SC>2 allowances traded
under the Acid Rain Cap and Trade
Program is significant. According to
EPA's Acid Rain and Related
Programs: 2007 Progress Report, the
total value of the SC>2 allowance market was over $5.1 billion in 2007.1U In that
same year, 4,700 private transactions involved 16.9 million allowances.
In addition to the Acid Rain Cap and Trade Program, utilities and other large
combustion sources participate in EPA's NOx Budget Trading Program. This
program is designed to reduce emissions of NOx to improve air quality in the
eastern part of the United States. Under this program, facilities in the eastern
States can buy and sell NOx emissions allowances. As noted in the NOxBudget
Trading Program Compliance and Environmental Results report for 2007,
99 percent of the NOx emissions measured under the NOx Budget Program were
measured by CEMS. In 2007, approximately 700,000 NOx allowances were
traded in private transactions.11 The price of NOx allowances varied from about
$500 to $1,000 per ton during 2007, with the year-end closing price at $825 per
ton. The total value of NOx allowances traded in 2007 exceeded $350 million.
Importance of Quality Assurance
Tests for CEMS
"Sources are required to conduct
stringent quality assurance tests of
their monitoring systems, such as
daily and quarterly calibration tests
and a semiannual or annual relative
accuracy test audit. These tests ensure
that sources report accurate data and
provide assurance to market
participants that a ton of emissions
measured at one facility is equivalent
to a ton measured at a different
facility."
Source: NOx Budget Trading Program
Compliance and Environmental Results
2007, EPA-430-R-08-008, December 2008
10
10 Based on total annual volume of 15,776,130 tons at an average nominal price of $325 per ton.
11 Private transactions include all transfers initiated by authorized account representatives for any compliance or
general account purposes.
11
-------
09-P-0235
The importance of calibration gases could increase as EPA expands the use of
emissions trading programs. On March 10, 2005, EPA announced the Clean Air
Interstate Rule (CAIR).12 CAIR caps emissions for NOx and SO2 for 28 eastern
States and the District of Columbia. States must achieve required emissions
reductions by requiring power plants to participate in an interstate cap and trade
program or meet individual State emissions budget by implementing its own
measures. Under CAIR, States may choose to participate in an EPA-administered
regional trading program.
In order to address climate change, Congress is considering a cap and trade
program to control greenhouse gas emissions. The primary greenhouse gases
include CO2, methane, nitrous oxide, and synthetically produced fluorinated
gases.13 Implementing a greenhouse gas emissions cap and trade program could
greatly expand the use and importance of EPA Protocol Gases if such a program
included monitoring and calibration requirements similar to the Acid Rain
Program.
Calibration Gases Help Assure the Accuracy of Ambient Monitoring
Measurements
EPA Protocol Gases also help assure the accuracy of ambient air quality monitors
used to collect ambient air quality data. EPA Protocol Gases are needed to
calibrate the monitoring systems for three National Ambient Air Quality
Standards - CO, NO2, and SO2. Based on the ambient monitoring data, EPA
determines whether the air quality in an area meets or exceeds the standard. If the
air quality exceeds the standard, the economic cost to the community for control
measures needed to reduce emissions and comply with the standard can be in the
hundreds of millions of dollars. Conversely, if EPA were to determine that an
area met the standard because of monitoring data that produced inaccurately low
data, control measures may not be implemented to protect the public health.
Consequently, it is extremely important that the monitoring data are accurate so
that EPA can make the proper attainment decision.
The EPA Protocol Gases we tested were tri-blend cylinders with gaseous
concentration levels generally used by utilities to calibrate CEMS. Ambient air
monitoring systems operators would generally use EPA Protocol Gases of lower
concentrations to calibrate their monitors, and perform other quality assurance
tests. For example, low concentration tri-blend EPA Protocol Gas cylinders are
used by OAQPS when they conduct the National Performance Audit Program to
assess the accuracy of ambient air quality monitors. Since we did not test gases of
the lower concentration range used by OAQPS for the National Performance
Audit Program, we cannot comment on their quality.
On December 23, 2008, the DC Circuit Court of Appeals reversed an earlier decision and granted EPA's petition
to remand the rule to EPA without vacatur. This ruling means that the regulation remains in effect but EPA must
remedy CAIR's flaws in accordance with the Court's July 11, 2008, opinion in the case.
13 Hydrofluorocarbons, perfluorocarbons, and sulfur hexafluoride.
12
-------
09-P-0235
Conclusions
Acid Rain regulations specify accuracy criterion for EPA Protocol Gases and
require vendors to participate in an EPA verification program. Ambient
monitoring regulations also require the use of EPA Protocol Gases and gas vendor
participation in a verification program. However, EPA has not implemented a
program to provide reasonable assurance that EPA Protocol Gases are accurate.
In addition, the Protocol for preparing EPA Protocol Gases has not been revised
since 1997 and programmatic responsibilities for maintaining the Protocol are
unclear.
Recommendations
We recommend that the EPA Assistant Administrator for Air and Radiation:
2-1 Implement an oversight program to provide reasonable assurance of the
quality of EPA Protocol Gases used to calibrate continuous emissions
monitoring systems for EPA's Acid Rain Cap and Trade Program, and
other stationary source air programs.
2-2 Implement an oversight program to provide reasonable assurance of the
quality of EPA Protocol Gases used to calibrate continuous ambient air
monitors for the NAAQS monitoring program.
We recommend that the EPA Assistant Administrator for Research and
Development:
2-3 Update and maintain the EPA Traceability Protocol to meet the defined
objectives of the Acid Rain, NAAQS, and other stationary source air
programs.
Agency Comments and OIG Evaluation
OAR and ORD agreed with the report's recommendations.
• For Recommendation 2-1, OAR plans to propose a rule for public comment
in fall 2009 to establish a largely self-supported, annual gas audit program of
protocol gases used for the Acid Rain program. In the interim, OAR has
already initiated efforts to conduct another OAR-funded test of EPA Protocol
Gases gas later in 2009.
• For Recommendation 2-2, OAR plans to implement a separate verification
program to address the lower concentration protocol gases used to calibrate
continuous ambient air monitors. Using analytical support from EPA
Regions 2 and 7, OAR plans to verify EPA Protocol Gases supplied to State,
13
-------
09-P-0235
local, and tribal agencies for calibration of continuous, ambient gaseous
pollutant monitors. OAR plans to launch this program by March 2010.
• For Recommendation 2-3, ORD will update and publish, as guidance, a
revised protocol to reduce the percentage of EPA Protocol Gases that do not
attain the Acid Rain Program's accuracy criterion. ORD plans to complete
and issue the revised protocol by October 1, 2010. ORD also provided a
corrective action plan in response to Recommendation 2-3 (see Appendix D).
The general actions outlined in OAR's and ORD's responses meet the intent of our
recommendations. Recommendations 2-1 and 2-2 will remain open in our tracking
system pending our receipt and approval of OAR's final corrective action plan.
ORD's proposed corrective action plan sufficiently addresses Recommendation 2-3.
As such, we are closing Recommendation 2-3 in our tracking system upon issuance
of this report. These recommendations will be tracked to completion in the
Agency's tracking system.
With respect to the ambient monitoring program, OAR commented that "the report
demonstrates that the gases tested met the expectations of the EPA Traceability
Protocol with 95 percent confidence limit. We point out that the gases used for the
ambient air program rely mainly on the traceability protocol. Thus, there are no
findings demonstrating a concern with using protocol gases for that program." We
disagree with this characterization. Our tests found that the low-range concentration
gases performed the worst in meeting the 2 percent accuracy criterion for the Acid
Rain Program. The ambient monitoring program uses even lower concentration
gases than the ones we tested during our evaluation. The past two Protocol Gas
assessments conducted by EPA have not included these lower concentration gases.
We believe the lack of test data to assure the accuracy of the lower concentration
gases used for the ambient monitoring program is a concern that should be addressed
by EPA.
After receiving OAR's response to the draft report, the OIG met with OAR to
discuss its response. OAR requested that we revise Recommendation 2-2 to state
that OAR should implement an oversight program to provide reasonable assurance of
the quality of EPA Protocol Gases. We made this change to Recommendation 2-2.
OAR and ORD also provided several technical clarifications and comments to the
report. We made changes to the final report based on these comments, as
appropriate. OAR and ORD's responses to the draft report are contained in
Appendices C and D.
14
-------
09-P-0235
Status of Recommendations and
Potential Monetary Benefits
RECOMMENDATIONS
Rec. Page
No. No.
Subject
Status1
Action Official
Planned
Completion
Date
POTENTIAL MONETARY
BENEFITS (in $OOOs)2
Claimed Agreed To
Amount Amount
2-1 13 Implement an oversight program to provide
reasonable assurance of the quality of EPA
Protocol Gases used to calibrate continuous
emissions monitoring systems for EPA's Acid
Rain Cap and Trade Program, and other
stationary source air programs.
2-2 13 Implement an oversight program to provide
reasonable assurance of the quality of EPA
Protocol Gases used to calibrate continuous
ambient air monitors for the NAAQS program.
2-3 13 Update and maintain the EPA Traceability
Protocol to meet the defined objectives of the
Acid Rain, NAAQS, and other stationary source
air prog rams.
Assistant Administrator for
Air and Radiation
Assistant Administrator for
Air and Radiation
Assistant Administrator for 10/01/2010
Research and
Development
O = recommendation is open with agreed-to corrective actions pending;
C = recommendation is closed with all agreed-to actions completed;
U = recommendation is undecided with resolution efforts in progress.
Identification of potential monetary benefits was not an objective of this evaluation.
15
-------
09-P-0235
Appendix A
Details on Scope and Methodology
To test the accuracy of EPA Protocol Gases, we contracted with a third party to purchase EPA
Protocol Gases directly from the specialty gas producers. Our intent was to purchase at least one
set of three EPA Protocol Gas cylinders from each gas producer and manufacturing location in
the country. After purchasing at least one cylinder from each identified producer14 or
manufacturing location, we purchased additional cylinders from each producer to maximize our
sample size and obtain a better representation of the facilities' production capability. Our
contractor purchased 87 EPA Protocol Gas tri-blend cylinders from 11 different producers.
NIST analyzed these cylinders and compared the results with each cylinder's certified tag values.
Chain of custody procedures were employed to account for each sample cylinder throughout the
process. Both our contractor and NIST conducted their work in accordance with the OIG-
approved quality assurance project plans.
Procedures for Purchasing Sample Gases
We tasked our contractor with surveying the industry to identify EPA Protocol Gas vendors and
manufacturing locations. To accomplish this survey, the contractor developed a set of keywords
to electronically search the Internet and the Thomas Register® for potential vendors of EPA
Protocol Gases. Our contractor also obtained industry information through conversations with
gas vendor personnel during the process of obtaining vendor quotes. The survey identified a
total of 11 gas producers and 18 manufacturing locations.
We authorized our contractor to purchase gas cylinders from each of the identified vendors with
an initial goal of purchasing at least three tri-blend cylinders of a low-, medium-, and high-
concentration range from each vendor manufacturing location. After accomplishing this goal,
we purchased additional cylinders from vendors with a goal of obtaining additional cylinders
from a cross section of vendor types taking into account such factors as cylinder cost and
potential market share. An initial set of three cylinders was purchased from all 18 manufacturing
locations, an additional set of cylinders was purchased from 9 of the 18 locations, and a third set
was purchased from one of the 18 locations. In all, 87 cylinders (29 sets of 3) were purchased.
Our contractor purchased the gases from January 30, 2008, to April 29, 2008. The following
table shows the number of cylinders by pollutant and concentration range.
Table A-1: Summary of Sample Gases Purchased
No. of
Cylinders
29
29
29
Range
Type
High
Mid
Low
C02 (%)
18.0
12.0
5.00
NO
(ppm)
900
400
50.0
SO2
(ppm)
1000
500
50.0
Source: EPA Protocol Gas Industry Survey and Blind Audit,
Final Report, February 2009
14
To ensure cylinders were purchased from every producer, sets of cylinders were purchased from three distributors.
16
-------
09-P-0235
NIST Analysis
Under an interagency agreement with the OIG, NIST was tasked with analyzing the
concentrations of the sample cylinders and comparing the test results to the manufacturers'
certified value. NIST's primary objective in conducting its analysis was to achieve a calculated
uncertainty of ± 0.5 percent or better for its analysis to achieve a 4:1 or better ratio between the
acceptance criterion (i.e., the Acid Rain Program's ± 2 percent accuracy criterion) being tested
and the uncertainty of NIST's analysis. A general standard of practice in metrology is for this
ratio - referred to as TAR or test to accuracy ratio - to be greater than or equal to 4:1. NIST's
analytical uncertainty ranged from ± 0.42 percent to ± 0.68 percent at the 95 percent confidence
level, depending upon the gas analyzed and the concentration.
To be consistent with the reporting of past EPA Protocol Gas tests conducted by EPA, Chapter 2
of this report presents the analytical results without adjustment for the uncertainty of the
analyses. Appendix B presents NIST's results with uncertainty.
Limitations
We instructed all parties involved in this evaluation not to discuss our plans with specialty gas
vendors or persons not needing such information to implement the evaluation. However, several
months prior to purchasing the gas cylinders, we were informed that at least one specialty gas
producer had become aware our intended evaluation. We do not know whether additional
vendors also became aware of our planned evaluation, or whether they would have been able to
identify the orders intended for the tests. If a specialty gas vendor was aware of our evaluation,
the results may not necessarily represent the quality of gases such a vendor routinely produces
and sells to users, but only indicates its ability to produce quality EPA Protocol Gases.
Because of a lack of data on the specialty gas vendor market share, we were unable to design a
sample that would allow us to project our results to the entire universe of EPA Protocol Gases
with an acceptable level of confidence and precision. Since we were unable to select gas
cylinders for testing in a manner that would ensure a statistically representative sample of the
EPA Protocol Gas market, we did not perform any inferential statistical analysis to project the
sample test results to the overall EPA Protocol Gas market.
17
-------
09-P-0235
Appendix B
Test Results With Uncertainty Factored into Analyses
NIST calculated the total uncertainty of its analysis at the 95 percent confidence level for each of
the component ranges it tested. NIST's calculated analytical uncertainty ranged from ± 0.42
percent to ± 0.68 percent. To assess the results taking into account the uncertainty of NIST's
analysis, we adjusted the test limits to account for uncertainty using the following formula:
Test limit = Specification (Acid Rain Program ± 2 percent criterion) - ± Uncertainty (NIST's analytical uncertainty)
The following table shows how the test limits were adjusted using a NIST analytical uncertainty
of ± 0.42 percent as an example.
Table B-1: Example of Test Limits Adjusted for Uncertainty
Test Limit (Expressed as
Percentage Difference)
< - 2.42
-2.42 to -1.58
-1.57 to 1.57
1.58 to 2.42
>2.42
Conclusion if Sample Falls in This Range
Failed with 95 percent confidence
Within the range of uncertainty
Passed with 95 percent confidence
Within the range of uncertainty
Failed with 95 percent confidence
Source: OIG table developed from data in NIST Report of Analysis, December 4, 2008.
When the test results are adjusted for uncertainty, the number of components meeting or
exceeding the Acid Rain Program's accuracy criterion is smaller than the unadjusted test
results presented in Chapter 2. For example, when we factor in uncertainty, we can only
say with 95 percent confidence that 82 percent (214 of 261) of the components met the
criterion, whereas the unadjusted test results show that 89 percent of the components met
the accuracy criterion. Similarly, when we factor in uncertainty, we can only state with 95
percent confidence that 5 percent (12 of 261) of the components did not meet criterion.
The remaining 13 percent (35 of 261) of the components were within the range of NIST's
analytical uncertainty. Table B-2 on the next page summarizes the test results adjusted for
uncertainty.
18
-------
09-P-0235
Table B-2: Summary of Test Results
Category
Met the Acid
Did Not Meet
Test Results
Rain Accuracy Criterion
the Acid Rain Accuracy Criterion
Within the Range of Analytical Uncertainty
Total
Test Results
No.
214
12
35
261
Percentage
82
5
13
100
Source: Developed from data in NIST Report of Analysis, December 4, 2008.
Both the unadjusted and adjusted test results showed that the lower-cost cylinders in our
sample performed worse than higher-cost cylinders. The adjusted results show that 11 of
the 12 gaseous mixtures not meeting the Acid Rain Program's accuracy criterion were
lower-cost cylinders (< $380) certified with GMIS. The following table summarizes the
tests meeting the accuracy criterion by cost and reference standard used to certify the
concentration.
Table B-3: Results by Vendor's Analytical Reference Standard and Cost
Reference Standard
Used and Cost of Gas
NTRM/SRM < $380
NTRM/SRM > $380
GMIS < $380
GMIS > $380
Unknowna<$380
Totals
No. of Components
Certified with this Standard
8
103
96
50
4
261
Components Meeting the Acid Rain
Accuracy Criterion
No.
6
100
59
47
2
214
Percentage
75
97
61
94
50
82"
Source: Developed from data in NIST Report of Analysis, December 4, 2008
a Vendor documentation did not specify type of reference standard used to certify the gaseous concentration.
b Represents the overall percentage, not the column total.
19
-------
09-P-0235
Appendix C
OAR's Response to OIG Draft Report
August 5, 2009
MEMORANDUM
SUBJECT: Comments on Draft IG Report: "EPA Needs an Oversight Program for Protocol
Gases" Assignment number 2007-000877
FROM: Gina McCarthy /s/
Assistant Administrator for Air and Radiation
TO: Wade T. Najjum
Assistant Inspector General for Program Evaluation
The Office of Air and Radiation (OAR) appreciates the Office of Inspector General's
(OIG) effort to work with our staff on this unique testing program to produce the subject Draft
Final Report "EPA Needs an Oversight Program for Protocol Gases" to improve the quality
assurance of protocol gases. We generally agree with the data described in the draft final report.
However, the issues raised apply differently to the Acid Rain Program's (ARP's) source
emissions monitoring systems and National Ambient Air Quality measurement network systems.
Therefore, this memorandum is structured in two parts - an Acid Rain Program section and an
Ambient Air Program section.
Acid Rain Program
Our major comments are presented below. We include both major and detailed
comments in the attached Word document in Track Changes. After you've reviewed our
comments, we'd like to have another meeting with you as soon as possible.
• At the end of the first paragraph in "At a Glance", please add: "However, 96 percent were
within 3 percent of the true value; and there was no bias associated with the 28 gas
components that did not meet the ARP performance specification. Based on the analysis
results, there are no anticipated nationwide environmental consequences of the calibration
gases used in the ARP. However, the ARP is addressing potential source-level problems in
two ways: (1) by performing an EPA-funded gas audit in 2009; and (2) by proposing a rule
establishing a largely self-supported, ongoing, annual gas audit program."
• Add the following paragraph to the "At a Glance" section:
"Due to the nature of our sample selection and a lack of data on vendor market shares, we
were unable to develop a sampling methodology to ensure that the number of samples selected from
each producer represented the number of cylinders produced and sold to consumers. Consequently,
20
-------
09-P-0235
we could not project our test results to the universe of EPA Protocol Gases, or estimate the potential
impact that gases not meeting the Acid Rain Program's accuracy criteria could have on that
program. Despite this limitation, our tests represent the largest one-time assessment of the Protocol
Gas industry. The final results of this study identified some potential areas of concern with respect
to the quality of less expensive gases certified using GMIS."
• Add the following paragraph to the "Noteworthy Achievements" section:
"The ARP is currently conducting a third gas audit, covering all known producers of EPA
Protocol gas. As in past ARP audits, the results will be released to the public. We expect results by
early 2010. To provide a cost-effective, long term audit capability, OAR is also developing a
proposed rule package to implement a largely self-supporting, annual gas audit program. This
proposed rule is expected to be published for public comment in the Federal Register in the fall of
2009."
ARP concurs with all of the OIG recommendations. However, we suggest a slight
rewording for recommendation #2-1 as follows:
2-1 Implement a better oversight program to assure the quality of EPA Protocol Gases used
to ensure the accuracy of continuous emissions monitoring systems for EPA's Acid Rain
Program, and other stationary source air programs.
Ambient Air Programs
[OIG NOTE: The comments initially submitted by the Ambient Air Program on August 5, 2009
were superseded by the amended comments provided September 2, 2009, in the following
memorandum.]
Thank you again for the opportunity to comment on the draft final report. If you have
questions regarding our comments, please contact Dawn Roddy, OAR Audit Follow-up
Coordinator, at 202-564-1228.
21
-------
09-P-0235
September 2, 200915
MEMORANDUM
SUBJECT: Amended Comments on Draft IG Report: "EPA Needs an Oversight Program for
Protocol Gases" Assignment Number 2007-000877
FROM: Gina McCarthy /s/
Assistant Administrator for Air and Radiation
TO: Wade T. Najjum
Assistant Inspector General for Program Evaluation
The Office of Air and Radiation (OAR) appreciates the Office of Inspector General's (OIG)
effort to work with our staff on this unique testing program to verify the quality assurance of
protocol gases as published in the subject Draft Final Report "EPA Needs an Oversight Program
for Protocol Gases". In a follow-up meeting between our staffs, your office explained that
recommendation 2-2 was directed toward the program under development and outlined by the
Office of Air Quality Planning and Standards (OAQPS) during the audit. Consequently, we
agree with recommendation 2-2 and provide the following amended paragraph titled "Ambient
Air Program", along with new recommendations to amend our August 5, 2009 response
concerning the ambient air program's protocol gases.
Ambient Air Program
The report demonstrates that the gases tested met the expectations of the EPA Traceability
Protocol with 95 percent confidence limit. We point out that the gases used for the ambient air
program rely mainly on the traceability protocol. Thus, there are no findings demonstrating a
concern with using protocol gases for that program. Determination of compliance with an ambient
air standard is completed with consideration of many factors including the quality of the data. We
believe updating the traceability protocol is the most efficient and effective corrective action to
improve the quality of gases and subsequently the data collected for the ambient air program.
Revising the protocol will also benefit stationary source emissions measurements. We, therefore,
agree with recommendation 2-2 and have already started creation of an appropriate oversight
program to reasonably assure the quality of protocol gases for the ambient air monitoring program.
We amend our August 5, 2009 response:
1. Replace the sentence in the "At A Glance" section that says, "However, as of April 2009,
EPA had not implemented a verification program, primarily due to indecision over how to
fund the program." with "However, as of April 2009, OAQPS, with commitments by EPA
Regions 2 and 7 for analytical support, had begun development of a program to provide
independent verification of EPA Protocol Gas cylinders used for continuous ambient air
monitors."
' The OAR memorandum was undated but was received by OIG on September 2, 2009.
22
-------
09-P-0235
2. Add the following paragraph to the "Noteworthy Achievements" section: "OAQPS has
planned and begun development of the Protocol Gas Verification Program to independently
verify "EPA Protocol Gases" supplied to state, local and Tribal agencies for calibration of
continuous, ambient gaseous pollutant monitors. EPA Regions 2 and 7 have committed to
providing analytical support. The program is expected to be launched in March 2010.
Finally, we suggest the following rewording for recommendation 2-2: "Implement an oversight
program, which reasonably assures the quality of EPA Protocol Gases used to calibrate continuous
ambient air monitors for the National Ambient Air Quality Standards monitoring program as
specified in 40 CFR part 58 Appendix A."
Thank you again for the opportunity to comment on the draft final report. If you have
questions regarding our comments, please contact Dawn Roddy, OAR Audit Follow-up
Coordinator, at 202-564-1228.
23
-------
09-P-0235
Appendix D
ORD's Response to OIG Draft Report
July 30, 2009
MEMORANDUM
SUBJECT: ORD Response to OIG's Draft Evaluation Report: EPA Needs an Oversight
Program for Protocol Gases (Assignment No. 2007-00877)
FROM: Lek G. Kadeli /s/
Acting Assistant Administrator (8101R)
TO: Wade T. Najjum
Assistant Inspector General for Program Evaluation (2460T)
Thank you for the opportunity to provide comments on the draft Office of the Inspector
General (OIG) report entitled "EPA Needs an Oversight Program for Protocol Gases."
Attached are the Office of Research and Development's (ORD) specific comments on the
draft report. In general, ORD concurs with the recommendation that ORD should update and
maintain the EPA Traceability Protocol for Gaseous Calibration Standards. We appreciate your
consideration of our comments. If there are any questions please do not hesitate to call me at 202-
564-6620 or Jorge Rangel at 202-564-1606.
Attachment
24
-------
09-P-0235
Office of Research and Development Comments on
Office of Inspector General Draft Audit Report, "EPA
Needs an Oversight Program for Protocol Gases",
Assignment No. 2007- 000877, July 8, 2009
This document is comprised of four sections:
1. General comments regarding the OIG recommendations
2. Table of ORD Response to OIG recommendations
3. Proposed detailed schedule of corrective actions
4. Specific comments by page number
1. General Comments regarding the OIG Recommendations
Thank you for the opportunity to comment on the draft Office of the Inspector General
(OIG) report entitled "EPA Needs an Oversight Program for Protocol Gases." EPA's Office of
Research and Development (ORD) concurs with the recommendation that ORD should update and
maintain the EPA Traceability Protocol for Gaseous Calibration Standards. We believe that the
scientific expertise within ORD puts it in a unique position to understand the gas metrological issues
that need to be addressed in the update and to prepare the update.
2. Table of ORD Response to OIG Recommendations
Rec.
No.
2-3
OIG Recommendation
Update and maintain the
EPA Traceability Protocol
to meet the defined
objectives of the Acid
Rain, NAAQS, and other
stationary source air
programs.
Action
Official
Assistant
Administrator
for Research
and
Development
ORD
Corrective
Actions
Update and
publish (as
guidance) a
revised protocol
to reduce the
percentage of
EPA Protocol
Gases that do not
attain the CAMD
accuracy criteria
Planned
Completion
Date
10/01/2010
3. Proposed Detailed Schedule of Corrective Actions
Corrective Action Taken
Final audit report disseminated to public
List of potential protocol changes* sent to
NIST and specialty gas producers for
Planned Completion Date
October 1, 2009
October 15, 2009
25
-------
09-P-0235
Corrective Action Taken
review with request for their comments,
suggested revised text, and any
concentration stability data supporting
changes to allowable gas composition,
concentration ranges, minimum cylinder
pressures, and certification periods
Receive all comments and stability data
Complete internal review draft of protocol
Receive EPA comments on internal draft
Complete external review draft of protocol
by NIST and specialty gas producers
Receive external comments on revisions
Revised protocol published as guidance
Planned Completion Date
November 15, 2009
May 15,2010
June 1,2010
July 1, 2010
August 15,2010
October 1, 2010
* In 2005, EPA sent the following list of informal change suggestions to NIST and specialty gas producers:
- include a procedure specifically for the use of Fourier Transform infrared (FTIR) spectroscopy;
- improve statistical procedures for uncertainty calculations and revise the Excel spreadsheet;
- include mercury calibration standards in the protocol;
- eliminate (or retain) gas manufacturers intermediate standards (GMISs);
- lengthen the certification periods for EPA Protocol Gases;
- change the cylinder pressure limitations for EPA Protocol Gases;
- include a procedure for analyzing and certifying zero gases;
- include gas dilution systems in the protocol;
- tighten documentation requirements;
- develop audit participant identification number for regulatory data reporting purposes;
- require ISO 17025 accreditation for producers of environmental calibration gases;
- include provision for preparation and analysis of batches of EPA Protocol Gases;
- require on-site visits of specialty gas producers by EPA representatives; and
- provide for technical assistance/outreach to specialty gas producers by NIST representatives;
4. Specific comments by page number
[OIG NOTE: ORD also provided several technical clarifications and comments to the draft
report. We made changes to the final report based on these comments, as appropriate.]
26
-------
09-P-0235
Appendix E
Distribution
Office of the Administrator
Assistant Administrator for Air and Radiation
Acting Assistant Administrator for Research and Development
Deputy Assistant Administrator for Air and Radiation
Deputy Assistant Administrator for Research and Development
Director, Office of Air Quality Planning and Standards, Office of Air and Radiation
Director, Office of Atmospheric Programs, Office of Air and Radiation
Director, National Risk Management Research Laboratory, Office of Research and Development
Agency Follow-up Official (the CFO)
Agency Follow-up Coordinator
General Counsel
Associate Administrator for Congressional and Intergovernmental Relations
Associate Administrator for Public Affairs
Audit Follow-up Coordinator, Office of Air and Radiation
Audit Follow-up Coordinator, Office of Research and Development
Acting Inspector General
27
------- |