U.S. ENVIRONMENTAL PROTECTION AGENCY
         OFFICE OF INSPECTOR GENERAL
                            Catalyst for Improving the Environment
Evaluation Report
       EPA Needs an Oversight Program
       for Protocol Gases

       Report No. 09-P-0235
       September 16, 2009

-------
Report Contributors:
                  Jim Hatfield
                  Geoff Pierce
                  Rick Beusse
Abbreviations

CAA        Clean Air Act
CAIR        Clean Air Interstate Rule
CEMS       Continuous Emissions Monitoring System
CFR         Code of Federal Regulations
CO          Carbon monoxide
CC>2         Carbon dioxide
EPA         U.S. Environmental Protection Agency
GMIS        Gas Manufacturer's Intermediate Standard
NAAQS      National Ambient Air Quality Standards
NIST        National Institute of Standards and Technology
NO          Nitric oxide
NOX         Nitrogen oxides
NO2         Nitrogen dioxide
NTRM       NIST Traceable Reference Material
OAP         Office of Atmospheric Programs
OAQPS      Office of Air Quality  Planning and Standards
OAR        Office of Air and Radiation
OIG         Office of Inspector General
ORD        Office of Research and Development
PGVP        Protocol Gas Verification Program
ppm         parts per million
SO2         Sulfur dioxide
SRM        Standard Reference Material
Cover photo:
Gas cylinders stored at the National Institute of Standards and Technology
laboratory in Gaithersburg, Maryland.  (Photo courtesy National Institute of
Standards and Technology)

-------
                    U.S. Environmental Protection Agency
                    Office of Inspector General

                   At  a  Glance
                                                              09-P-0235
                                                       September 16, 2009
                                                                   Catalyst for Improving the Environment
why we Did This Review      EPA /Veec/s a/7 Oversight Program for Protocol Gases
The accuracy of continuous
emissions monitors is critical
to the U.S. Environmental
Protection Agency's (EPA's)
Acid Rain Program because
data from these monitors
determine the number of
allowances a utility can bank,
sell, or trade. Specialized
gases known as "EPA
Protocol Gases" are used to
calibrate and assure the quality
of these monitors. We sought
to determine whether certified
concentrations of these gases
are accurate.

Background

Vendors produce and certify
EPA Protocol Gases in high-
pressure cylinders according
to EPA procedures.  EPA
regulations require the use of
these gases, or National
Institute of Standards and
Technology (NIST)-certified
reference materials, when
conducting quality assurance
for continuous emissions and
ambient air monitoring
systems. EPA's Acid Rain
Program requires that the
certified concentration of the
I gases be within ± 2 percent of
the true concentration.

For further information, contact
our Office of Congressional,
Public Affairs and Management
at (202) 566-2391.

To view the full report,
click on the following link:
www.epa.qov/oiq/reports/2009/
20090916-09-P-0235.pdf
 What We Found
We purchased 87 cylinders of EPA Protocol Gases and had NIST analyze each
cylinder to determine whether the three gaseous mixtures contained in each
cylinder (carbon dioxide, nitric oxide, and sulfur dioxide) met the Acid Rain
Program's accuracy criterion. We found that 89 percent (233 components) met
the Acid Rain Program's accuracy criterion and 11 percent (28 components) did
not. Of the 28 components that did not meet the criterion, 17 were within 3.0
percent of the NIST-determined true concentration; 7 were within 3.0 to 5.0
percent; and 4 exceeded the true concentration by more than 5.0 percent.

Our sample was not designed to estimate the impact of the test results on the Acid
Rain Program.  However inaccurately certified concentrations could cause system
operators to unknowingly calibrate their monitoring systems to record inaccurate
measurements. For example, if a utility overstates  its emissions, it could lose the
opportunity to sell allowances to other utilities.  If a utility understates its
emissions, the utility and regulators may incorrectly conclude that the source is
complying with emissions standards.  With respect to ambient air monitoring, the
accuracy of these monitors is important because the data are used to determine
whether areas are in compliance with the Nation's ambient air quality standards.

EPA has conducted only two tests  of the accuracy of EPA Protocol Gases since
1997, when EPA's Office of Research and Development discontinued its annual
testing program. Thus, EPA does not have reasonable assurance that the gases
used to calibrate emissions monitors for the Acid Rain Program and continuous
ambient monitors for the Nation's  air monitoring network are accurate.
 What We Recommend
We recommend that the Office of Air and Radiation (OAR) implement oversight
programs to assure the quality of EPA Protocol Gases used to calibrate continuous
emissions monitoring systems and ambient air monitors. We also recommend that
the Office of Research and Development (ORD) update and maintain the protocol
gas procedures to ensure that the protocol meets the objectives of the Acid Rain,
ambient air, and stationary source air programs.  OAR and ORD concurred with
our recommendations. OAR has initiated efforts to conduct another gas audit later
this year.  OAR also plans to propose a rule later in 2009 to establish a largely
self-supported, annual gas audit program of protocol gases used for the Acid Rain
Program.  Further, OAR plans to implement a separate verification program to
address the lower concentration protocol gases used to calibrate continuous
ambient air monitors. ORD  will update and maintain the protocol gas procedures.
EPA's planned actions meet the intent of our recommendations.

-------
                    UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
                                  WASHINGTON, D.C. 20460
                                                                           OFFICE OF
                                                                       INSPECTOR GENERAL
MEMORANDUM
SUBJECT:
FROM:
TO:
                                  September 16, 2009
EPA Needs an Oversight Program for Protocol Gases
Report No. 09-P-0235


                   tt^ ^J^-
Wade T. Najjum               A-i
Assistant Inspector General for Program Evaluation

Gina McCarthy
Assistant Administrator for Air and Radiation

Lek G. Kadeli
Acting Assistant Administrator for Research and Development
This is our report on the subject evaluation conducted by the Office of Inspector General (OIG)
of the U.S. Environmental Protection Agency (EPA). This report contains findings that describe
the problems the OIG has identified and corrective actions the OIG recommends.  This report
represents the opinion of the OIG and does not necessarily represent the final EPA position.
Final determinations on matters in this report will be made by EPA managers in accordance with
established audit resolution procedures.

The estimated cost of this report - calculated by multiplying the project's staff days by the
applicable daily full cost  billing rates in effect at the time, including the costs of purchasing EPA
Protocol Gases and having them analyzed by the National Institute of Standards and Technology
-is $665,846.

Action Required

In accordance with EPA Manual 2750, EPA  's Audit Management Process, the Office of Air and
Radiation should provide a written response  within 90 calendar days. The Office of Air and
Radiation's response should include a corrective action plan and planned completion dates  for
Recommendations 2-1 and 2-2.  The Office of Research and Development submitted a corrective
action plan that sufficiently addresses Recommendation 2-3. As such, we are "closing"
Recommendation 2-3 in our tracking system upon issuance of this report. These
recommendations will be tracked to completion in the Agency's tracking system.  No further

-------
response is required for Recommendation 2-3. As outlined in EPA Manual 2750, the Agency is
responsible for tracking the implementation of these actions in its Management Audit Tracking
System. We have no objections to the further release of this report to the public.  This report will
be available at http://www.epa.gov/oig.

If you or your staff have any questions regarding this report, please contact me at (202) 566-0832
or najjum.wade@epa.gov, or Rick Beusse at (919) 541-5747 or beusse.rick@epa.gov.

-------
EPA Needs an Oversight Program for Protocol Gases                         09-P-0235
                     Table  of Contents
Chapters
   1   Introduction	   1

           Purpose	   1
           Background	   1
           Noteworthy Achievements	   5
           Scope and Methodology	   5

   2   EPA Needs an Oversight Program to Assure the Accuracy of
       Protocol Gases	   7

           EPA Does Not Have an Oversight Program to Assure the
                Accuracy of Protocol Gases	   7
           Low Concentration NO and SO2 Gases and Lower-Cost Cylinders
                Performed Worst in OurTests	   8
           Calibration Gases Play an Important Role in Assuring Data Quality
               for EPA Air Programs 	  10
           Conclusions	  13
           Recommendations	  13
           Agency Comments and OIG Evaluation	  13

   Status of Recommendations and Potential Monetary  Benefits	  15
Appendices
   A   Details on Scope and Methodology	  16

   B   Test Results with Uncertainty Factored into Analyses	  18

   C   OAR's Response to OIG Draft Report 	  20

   D   ORD's Response to OIG Draft Report	  24

   E   Distribution	  27

-------
                                                                              09-P-0235
                                 Chapter 1
                                  Introduction
Purpose
             The U.S. Environmental Protection Agency (EPA) Protocol Gases are used to
             calibrate and test continuous emissions monitoring systems (CEMS) and ambient
             air quality monitoring systems. EPA requires the use of CEMS for large utilities
             covered by the Acid Rain Program and other clean air programs.  Consequently,
             the accuracy of the EPA Protocol Gases is critical to ensuring the integrity of
             emissions trading and other EPA programs.  We conducted this evaluation to
             determine whether the certified concentrations of EPA Protocol Gases were
             within acceptable limits for accuracy.

Background

             The integrity and effectiveness of EPA's air programs depend upon collecting
             accurate pollutant emissions and air quality data.  EPA requires that the systems
             collecting these data periodically undergo certain quality assurance procedures to
             assure their accuracy.  These procedures include daily calibration and accuracy
             tests of the monitoring systems.  Calibration tests check the accuracy of these
             monitoring systems by running a gas mixture of a known concentration through
             the systems and comparing the systems' readings with the known concentration.
             The systems are adjusted accordingly to eliminate measurement inaccuracies.

             The certified concentrations of gaseous mixtures used to calibrate and test
             monitoring systems should accurately reflect the true gaseous concentration. EPA
             regulations require that the gases used for calibrating and testing the accuracy of
             ambient air quality analyzers and continuous emissions monitors1 be traceable to
             either a National Institute of Standards and Technology (NIST) Standard
             Reference Material (SRM) or a NIST Traceable Reference Material (NTRM).
             EPA has established an EPA Traceability Protocol that specialty gas producers
             must follow when preparing calibration gases to meet the NIST traceability
             requirement.  The gases produced in accordance with these standards are referred
             to as EPA Protocol Gases.

             This protocol was designed by EPA, NIST, and specialty gas producers so that a
             laboratory analyst can analyze and certify compressed gas calibration standards
             with low uncertainty and with traceability to NIST. The protocol specifies a
             general analytical procedure for determining the concentration of the standards,
             but it does not specify the analytical instrumentation or the gas-handling

1 Continuous emissions monitors continuously measure pollutants emitted into the atmosphere in exhaust gases from
combustion or industrial processes.

-------
                                                                               09-P-0235
              apparatus that the analyst must use during the analysis. In summary, the protocol
              requires that (1) a NIST-certified gaseous mixture be used as the analytical
              reference standard, (2) a monthly calibration curve for the instrumentation be
              established, and (3) both the NIST-certified gaseous mixture and the candidate
              EPA Protocol Gas be measured at least three times, with reactive gas mixtures
              measured at least three additional times at least 7 days after the first set of
              measurements.

              The protocol has detailed statistical procedures for estimating the total uncertainty
              of these standards. Although it includes procedures for estimating uncertainty,
              the protocol does not specify that EPA Protocol Gases meet any given level of
              uncertainty and does not establish specific accuracy criteria for the standards'
              uncertainty. However, the Acid Rain Program has prescribed an uncertainty
              standard for EPA Protocol Gases used to calibrate CEMS as described in the
              following section.

              Acid Rain Program Requirements for EPA Protocol Gases

              The Acid Rain Program implements Title IV of the Clean Air Act (CAA) to
              reduce emissions of sulfur dioxide (SO2) and nitrogen oxides (NOX), the  primary
              causes of acid rain.  The program uses an allowance trading system to reduce 862
              emissions. Under this system, affected utility units are allocated allowances (one
              allowance = one ton of 802) that permit a unit to emit SC>2 during or after a
              specified year. Utilities can buy, sell, or bank allowances. The program began in
              1995.  For each ton of 862 emitted in a given year, one allowance is retired and
              can no longer be used.  The program is phased in, with the final SC>2 cap  set at
              8.95 million tons in 2010.  The Acid Rain Program does not use a trading system2
              to reduce emissions for NOX, but instead establishes emissions limits for covered
              units that are designed to achieve the program's goal of reducing NOX emissions
              by 2 million tons from  1980 levels.

              Continuous emissions monitoring is instrumental in ensuring that the mandated
              reductions of 862 and NOX under the Acid Rain Program are achieved.
              Accordingly, EPA has established requirements for continuously monitoring SO2
              and NOX (and other parameters) for units regulated under the Acid Rain Program.
              Appendices A and B to 40 Code of Federal Regulations (CFR) Part 75 describe
              specification and test procedures for these monitors, which include daily
              calibration tests and periodic accuracy tests.  The regulations require that the
              calibration gases used to conduct these tests must be NIST-certified reference
              standards or that their concentrations must be traceable to NIST-certified
              reference standards according to the EPA Traceability Protocol for Assay and
              Certification of Gaseous Calibration Standards.  In January 1993, EPA's Office
              of Atmospheric Programs (OAP) included a 2.0 percent accuracy criterion in its
              final Acid Rain Program emissions monitoring rule (58 Federal Register 3731).
2 NOX emissions are traded under other air programs, including the NOX Budget Trading Program, which is designed
to reduce NOX emissions in order to reduce ambient ozone levels.

-------
                                                                               09-P-0235


              Appendix A to 40 CFR Part 753 specifies that the EPA Protocol Gases must have
              a certified uncertainty (95 percent confidence interval) that must not be greater
              than plus or minus (±) 2.0 percent of the certified concentration (tag value) of the
              gas mixture.

              In January 2008, EPA revised the Acid Rain Cap and Trade Program regulations
              to require that producers of EPA Protocol Gases participate in EPA's Protocol
              Gas Verification Program in order to certify their gases as EPA Protocol Gases.

              National Ambient Monitoring Program Requirements for EPA
              Protocol Gases

              Appendix A to 40 CFR Part 584 describes quality assurance procedures for
              national ambient air monitors.  States and EPA use these monitors to determine
              whether an area's air quality meets or exceeds the National Ambient Air Quality
              Standards (NAAQS).  The quality control procedures include biweekly one-point
              checks of carbon monoxide (CO), 862, and nitrogen dioxide (NC^), and annual
              performance evaluations that test the monitors against gases of known
              concentrations for at least three different ranges of concentrations.  Appendix A to
              this regulation requires that cylinders of compressed gas used to obtain test
              concentrations must be traceable to either a NIST Traceable Reference Material
              or a Gas Manufacturer's Intermediate Standard, certified in accordance with
              EPA's Traceability Protocol for Assay and Certification of Gaseous Calibration
              Standards. Further, vendors advertising certification with the procedures and
              distributing gases as "EPA Protocol Gas" must participate in the EPA Protocol
              Gas Verification Program or not use "EPA" in any form of advertising.

              Unlike the Acid Rain Program, the ambient monitoring regulations do not specify
              accuracy criteria for the EPA Protocol Gases  used to calibrate ambient air
              monitors.

              Stationary Source Performance Standards Requirements for EPA
              Protocol Gases

              The CAA required EPA to establish emissions standards5 for certain stationary
              sources that cause or contribute significantly to air pollution. These standards are
              intended to promote use of the best air pollution  control technologies and apply to
              sources that have been constructed or modified since the proposal of the standard.
              These standards are published in 40 CFR Part 60, and  may require the source to
              install CEMS to demonstrate compliance.
3 Appendix A to Part 75—Specifications and Test Procedures.
4 Appendix A to Part 58—Quality Assurance Requirements for State and Local Air Monitoring Stations, Special
Purpose Monitors, and Prevention of Significant Deterioration Air Monitoring.
5 Referred to as New Source Performance Standards.

-------
                                                                               09-P-0235
              Part 60 contains varied requirements for using calibration gases depending upon
              the EPA method used to conduct emissions testing. Specifically, the methods for
              determining SC>2 (Method 6C) and NOX (Method 7E) emissions from stationary
              sources require that the calibration gases be prepared according to the EPA
              protocol or certified by the tester using a specified wet chemical test method. The
              electric utility, petroleum, and municipal waste incineration industries are the
              largest industries (in terms of emissions and the number of sources) subject to
              continuous emissions monitoring requirements and the use of EPA Protocol Gases
              for Part 60 emissions  standards.

              EPA's Protocol Gas Verification  Program

              From 1985 to 1997, EPA's Office of Research and Development (ORD)
              conducted a series of analytical tests6 of EPA Protocol Gases sold by specialty gas
              producers. In 1996, ORD issued a final draft exit strategy that transferred the
              administration of this  and other quality assurance programs to the Office of Air
              Quality Planning and  Standards (OAQPS) of the Office  of Air and Radiation
              (OAR).  ORD provided OAR with resource support during the transition, but
              provided no further resources after the transfer was completed  in 1998.  The
              annual tests were discontinued after 1997, and no additional tests occurred until
              2003.  Due to a lack of analytical tests and a concern that EPA Protocol Gas
              quality may have declined, the OAP conducted a test in  2003.  OAP's contractor
              purchased 42 EPA Protocol Gas cylinders containing tri-blend mixtures of carbon
              dioxide (CO2). nitric oxide (NO), and SO2 from a total of 14 specialty gas vendors
              nationwide. A third party purchased the cylinders for EPA so that the gas vendors
              would not know that EPA was analyzing the cylinders.  A gas passed the test if
              the "true" concentration of the gas differed from the certified concentration by ± 2
              percent or less.  Overall, 89 percent of the gases passed on a gas component basis;
              11 percent did not pass the test.

              OAP conducted another test in 2006 but decided not to publish the results because
              of concerns about the representativeness of some of the  sample gases. Instead of
              purchasing gases directly from vendors, OAP arranged for end users (i.e.,
              utilities) to provide EPA with the Protocol Gases for testing. However,  EPA
              encountered several problems with this approach, including over- and
              under-representation of manufacturers and production sites, as well as old and
              used samples.

              Approximately 6 years ago, EPA started developing an ongoing EPA Protocol
              Gas Verification Program (PGVP). After a development process involving U.S.
              specialty gas companies, NIST, the Institute of Clean Air Companies, and EPA,
              tentative agreement was reached on a specialty gas company-funded, ongoing
              draft PGVP.  EPA promulgated a requirement for producers to participate in this
6 EPA calls these tests EPA Protocol Audits. We refer to these audits as tests in this report to avoid confusion with
the term "audit" as defined by the Government Accountability Office's Government Auditing Standards, and
applicable to work conducted by the Office of Inspector General (OIG).

-------
                                                                               09-P-0235
              program in a January 24, 2008, final rule. However, in March 2008, one specialty
              gas vendor petitioned EPA for reconsideration of the final rule.  The vendor cited
              several concerns including the nature of the funding for the PGVP. Since that
              time, EPA has been working to resolve these concerns, but still had not funded the
              program as of May 2009.  EPA had planned to have EPA Protocol Gas vendors
              fund the verification program through payments of fees to a third party.
              However, EPA has determined such a process could violate the Miscellaneous
              Receipts Act.7
Noteworthy Achievements
              Recognizing the importance of EPA Protocol Gases, the OAP conducted two
              protocol gas assessments, one in 2003 and another in 2006. Although the sample
              size in each of these assessments was not statistically significant and market share
              information was not available, both assessments indicated that some specialty gas
              vendors were not meeting the EPA Protocol Gas accuracy criterion of ± 2.0
              percent. OAP posted the results, including the names of the specialty gas
              companies, on an Agency Website. Based on comments by specialty gas
              companies, the 2003 accuracy check resulted in some vendors making
              improvements in personnel, training,  quality, and equipment.  When we originally
              approached EPA with our proposal to assess the accuracy of EPA Protocol Gases,
              OAR supported our plans and cooperated fully with our evaluation team to
              develop a useful assessment of the EPA Protocol Gas program.
Scope and Methodology
              We conducted this evaluation to test the accuracy of EPA Protocol Gases used to
              calibrate continuous emissions monitoring equipment. We designed our sampling
              methodology to test at least one set of three EPA Protocol  Gas cylinders from
              every specialty gas  manufacturing location in the country and to maintain the
              confidentiality of the tests so that the gases we purchased would represent those
              sold to utilities and  other customers.8 In order to maintain the confidentiality of
              the tests, we contracted with a third party to purchase the gases and deliver them
              to NIST for analysis.

              Our contractor purchased tri-blend mixtures of NO, CO2, and SO2, since tri-blend
              gases are often used by utilities. We identified 11 gas producers nationwide using
              18 manufacturing locations to produce EPA Protocol Gases.  Our contractor
              purchased at least one set of EPA Protocol Gas cylinders produced from each of
              the 18 manufacturing locations. In all, we purchased 87 EPA Protocol Gas
              cylinders from  14 specialty gas vendors (i.e., 11 producers and 3 distributors),  and
7 The Miscellaneous Receipts Act was passed in order to ensure that government agencies did not bypass the
appropriations authority of Congress by augmenting their budgets via other means, such as user fees, fees for
training courses, parking fees, contract and lease fees and revenues, monetary awards in court cases involving the
agencies, court costs and fees, or civil penalties.
8  A set consisted of three cylinders in low-, medium-, and high-range concentrations.

-------
                                                                                   09-P-0235
               18 manufacturing locations.  Our contractor delivered the sample gases to NIST
               for analysis. We selected NIST because of its extensive experience in gas
               metrology and its reputation for analytical accuracy.  NIST analyzed the gaseous
               concentrations of the 87 tri-blend cylinders (i.e., 261  separate component
               analyses) to determine whether the "true" concentrations were within ± 2 percent
               of the concentration certified by the vendor.9

               We reviewed documentation related to EPA's past assessments of EPA Protocol
               Gases, including past results, program funding and implementation, and current
               plans for implementing a revised EPA Protocol Gas Verification Program.  We
               reviewed federal regulations to identify requirements for the use of EPA Protocol
               Gases.  We interviewed staff and managers from ORD, OAQPS, and OAP.

               We conducted this evaluation in accordance with generally accepted government
               auditing standards. Those standards require that we obtain sufficient, appropriate
               evidence to provide a reasonable basis for our findings and conclusions based on
               our evaluation objectives.  We believe that the evidence obtained provides a
               reasonable basis for our findings and conclusions based  on our objectives. We
               conducted our work from June 2007 to July 2009.

               Appendix A provides more detailed information on our scope and methodology.
9 Vendors typically certify gaseous concentrations using either a NIST Traceable Reference Material (NTRM), a
Standard Reference Material (SRM), or a Gas Manufacturer's Intermediate Standard (GMIS). NTRMs are
reference material produced by a commercial supplier with a well-defined traceability linkage to NIST. These
reference materials are allowed to bear the NIST NTRM certification mark and were developed to allow NIST to
respond to increasing needs for high-quality reference materials.  SRMs are reference materials produced by NIST
and are certified to have specific chemical or physical properties. A GMIS is a NIST traceable standard produced
by the gas vendor.  A NIST traceable standard is certified with an instrument calibrated through direct traceability
with a NIST standard such as an SRM or NTRM.

-------
                                                                        09-P-0235
                               Chapter  2

             EPA Needs an Oversight  Program to

           Assure the Accuracy of Protocol Gases

            EPA does not have reasonable assurance that EPA Protocol Gases used to
            calibrate CEMS for the Acid Rain Program and continuous ambient monitors for
            the Nation's air monitoring network are accurate. EPA has conducted only two
            assessments since 1997, when ORD discontinued its program to test the accuracy
            of EPA Protocol  Gases.  We analyzed 261  gaseous components and found that
            89 percent (233 components) met the Acid Rain Program's accuracy criterion for
            calibration gases, while  11 percent (28 components) did not meet the accuracy
            criterion. Calibration gases must be accurate because inaccurate calibration gases
            can cause a system operator to calibrate a measurement system to produce
            inaccurate measurements.  In turn,  inaccurate CEMS measurements can affect the
            integrity of the computed allowances available for trade on the open market.

EPA Does Not Have an  Oversight Program to Assure the Accuracy of
Protocol Gases

            EPA does not have a viable oversight program in place to assure the quality of
            EPA Protocol Gases. Since 1997, EPA oversight to assure the accuracy of these
            gases has consisted of two analytical tests conducted by OAP in 2003 and 2006.
            However, EPA did not publish the  2006 test results because of concerns over the
            sampling methodology.  Without an oversight program, EPA does not have
            reasonable assurance that EPA Protocol Gases meet the accuracy criterion
            certified to by the vendors and required for the Acid Rain Program.

            In January 2008,  EPA promulgated a final rule requiring EPA Protocol Gas
            producers to participate  in an EPA  PGVP. EPA anticipated implementing a
            verification program to assess the accuracy of Protocol Gases on an ongoing
            basis. However,  implementation was delayed when a specialty gas vendor
            petitioned EPA to reconsider the rule.  As of May 2009, EPA's OAP was still
            working to resolve legal issues raised in the petition and deciding how to fund the
            program. As of May 2009, OAR's OAQPS was developing a similar but separate
            verification program to test the accuracy of the EPA Protocol Gases used to
            calibrate continuous ambient air monitors.

            Even though the  Acid Rain Program and the ambient air monitoring program
            generally use different concentrations of EPA Protocol Gases in their programs,
            the gases must be produced and certified under the same EPA Protocol. Although
            ORD originally developed the Protocol, a co-author of the Protocol informed us
            that it was not clear who currently has responsibility for maintaining the Protocol.
            Although the 2003  EPA Protocol Gas assessment report and specialty gas

-------
                                                                             09-P-0235
             producers have identified several analytical issues not addressed by the current
             Protocol, the Protocol has not been revised since 1997.
Low Concentration NO and SO2 Gases and Lower-Cost Cylinders
Performed Worst in Our Tests

             We purchased and analyzed 87 cylinders of tri-blend EPA Protocol Gases to
             determine whether each of the three gaseous concentrations (per cylinder) met the
             Acid Rain Program's accuracy criterion. Of the 261 gaseous components
             analyzed, 89 percent met the Acid Rain Program's accuracy  criterion for EPA
             Protocol Gases, while 28 (11 percent) did not meet the accuracy criterion.

             Of the 28 components that did not meet the criterion, 17 were within 3.0 percent
             of the NIST-determined true concentration; 7 were within 3.0 to 5.0 percent; and
             4 exceeded the true concentration by more than 5.0 percent.  All exceptions were
             for gaseous components from lower-cost cylinders (i.e., under $380). Twelve of
             the gas components' "true" concentrations were higher than the certified
             concentration, and 16 were lower than the certified concentration.

             The gaseous components with the most exceptions were low concentration NO
             and SO2. The certified concentrations for low range NO and SO2 gases exceeded
             the NIST analyzed or "true" concentration by more than ± 2 percent for 24 and 17
             percent of the samples, respectively.  The following table summarizes the test
             results for each range of gaseous components analyzed.

             Table 2-1:  Results by Gas and Concentration
Gas
NO
CO2
SO2
Range
Low
Med
High
Low
Med
High
Low
Med
High
Total
No. of
Samples
29
29
29
29
29
29
29
29
29
261
Met the Accuracy
Criterion
No.
22
29
25
26
28
29
24
25
25
233
Percentage
76
100
86
90
97
100
83
86
86
89 a
Did Not Meet the
Accuracy Criterion
No.
7
0
4
3
1
0
5
4
4
28
Percentage
24
0
14
10
3
0
17
14
14
11a
             Source: OIG, developed from data in NIST Report of Analysis, December 4, 2008
             a Represents the overall percentage, not the column total.

             We also assessed the test results on a cost-per-cylinder and production-facility
             basis. This analysis revealed that all gaseous components not meeting the
             accuracy criterion were from cylinders costing under $380. All of the gaseous
             components not meeting the accuracy criterion came from 7 of the 18 (39 percent)
             facilities. These seven facilities accounted for 108 of the 261 components we
             tested, and 28 of these 108 components did not meet the Acid Rain Program's

-------
                                                                    09-P-0235
accuracy criterion. All 153 gaseous components associated with cylinders costing
over $380 met the accuracy criterion.  The following table shows the total tests
conducted, the cost per cylinder, and the test results for each production facility.

Table 2-2: Test Results by Cost and Production Facility
Production
Facility
A
B
N
D
K
Q
R
Subtota
C
D
E
F
G
H
1
J
L
M
O
P
Cost Per
Cylinder
<380
<380
<380
<380a
<380
<380
<380
<380
>380
>380a
>380
>380
>380
>380
>380
>380
>380
>380
>380
>380
Subtotal >380
Totals
Total
Tests
18
18
18
18
9
18
9
108
9
9
9
9
18
18
9
9
9
18
18
18
153
261
No. of Tests
Meeting the
Accuracy
Criterion
15
17
9
15
7
10
6
80
9
9
9
9
18
18
9
9
9
18
18
18
153
233
No. of Tests
Not Meeting the
Accuracy
Criterion
3
1
9
2
2
8
3
28
0
0
0
0
0
0
0
0
0
0
0
0
0
28
Source: OIG table developed from data in NIST Report of Analysis, December 4, 2008.
3 Three sets of cylinders (a set consists of a low-, a medium-, and a high-range cylinder) were
produced at this location but were purchased from two different vendors. Two sets of cylinders
cost under $380 per cylinder and one set cost over $380 per cylinder.

We further analyzed the test results, taking into account the vendor's analytical
technique and the reference standard used by the vendor to certify its gaseous
concentrations.  This analysis showed that low-cost cylinders certified with a Gas
Manufacturer's Intermediate Standard (GMIS) performed the worst in our tests.
Specifically, 24  of the 28 (86 percent) gaseous components that did not meet the
Acid Rain Program's accuracy criterion were from lower-cost cylinders that were
certified using GMIS.  Although most of the gaseous components not meeting the
accuracy criterion were certified using GMIS, many GMIS-certified components
did meet the accuracy standard.  However, all of these gaseous components were
from cylinders costing over $380.  The following table shows the test results by
certification standard and cylinder cost.

-------
                                                                              09-P-0235
             Table 2-3: Pass/Fail Rates By Analytical Reference Standard
Reference Standard
Used and Cost of Gas
Cylinder
NTRM/SRM < $380
NTRM/SRM > $380
GMIS < $380
GMIS > $380
Unknown3 < $380
Totals
No. of Gaseous
Components
Certified with this
Standard
8
103
96
50
4
261
Components Not Meeting
the Acid Rain Program's
Accuracy Criterion
No.
2
0
24
0
2
28
Percentage
25
0
25
0
50
11"
             Source: OIG table developed from data in NIST Report of Analysis, December 4, 2008.
             a Vendor documentation did not specify type of reference standard used to certify the gaseous
             concentration.
             b Represents the overall percentage, not the column total.

Calibration Gases Play an Important Role in Assuring Data Quality for
EPA Air Programs

             Calibration gases play an important role in helping to assure the quality of data
             used in EPA programs. Inaccurate calibration gases can negatively affect EPA
             programs that rely upon accurate emissions or ambient air quality measurements
             by causing system operators to calibrate their measurements systems inaccurately.
             For example, if a calibration gas used by a utility was certified to contain 100
             parts per million (ppm) of  862, but only contained  96 ppm, the system operator
             would unknowingly calibrate the CEMS to read 96  ppm as 100 ppm. This would
             result in the CEMS overestimating emissions. Conversely, if the true
             concentration of the calibration gas were more than the certified concentration,
             the system operator would  unknowingly calibrate the CEMS to underestimate
             emissions.

             The number of monitoring  systems potentially using EPA Protocol Gases is
             significant. For example, over 3,500 electric generating units were covered by the
             SC>2 allowance trading program in 2007. A subset of almost  1,000 of these 3,500
             units was also covered by the NOx provisions of the Acid Rain Program. Further,
             approximately 1,300 ambient air monitors collect data on CO, SO2, and NOX
             concentrations.  An EPA contractor surveyed Protocol Gas use in preparation for
             EPA's analytical tests of Protocol Gases in 2003. The contractor estimated that
             utilities purchased from 37,500 to 75,000  Protocol Gas cylinders annually and
             missions testing companies purchased from 16,500  to 55,000 cylinders annually.
             The survey did not estimate annual purchases by State and local agencies, but one
             large State reported using about 75 Protocol Gases a year for its ambient
             monitoring network.

             Due to the nature of our sample selection  and a lack of data on vendor market
             shares, we were unable to develop  a sampling methodology to ensure that the
             number of samples selected from each producer represented the number of
             cylinders produced and sold to consumers. Consequently, we did not project our
             test results to the universe of EPA Protocol Gases, or estimate the impact that
                                          10

-------
                                                                                09-P-0235
              these results could have on the Acid Rain Program.  Despite this limitation, our
              tests represent the largest one-time assessment of the Protocol Gas industry, and
              revealed some potential areas of concern with respect to the quality of less-
              expensive gases certified using GMIS.

              The following sections describe the importance of accurate monitoring data to the
              Acid Rain and Air Monitoring programs. These data are collected by
              measurement systems whose accuracy is assured by quality assurance activities
              such as calibrations using EPA Protocol Gases.
              Calibration Gases Help Assure
              Accuracy of Emissions Data for
              Trading Programs

              For sources participating in emissions
              trading programs, overestimating
              emissions can result in lost opportunities
              to sell or bank allowances.
              Underestimating emissions could result
              in banking or selling allowances that did
              not really exist,  and provide an
              inaccurate picture of a facility's progress
              in reducing emissions.

              The volume of SC>2 allowances traded
              under the Acid Rain Cap and Trade
              Program is significant.  According to
              EPA's Acid Rain and Related
              Programs: 2007 Progress Report, the
              total value of the SC>2 allowance market was over $5.1 billion in 2007.1U In that
              same year, 4,700 private transactions involved 16.9 million allowances.

              In addition to the Acid Rain Cap and Trade Program, utilities and other large
              combustion sources participate in EPA's NOx Budget Trading Program. This
              program is designed to reduce emissions of NOx to improve air quality in the
              eastern part of the United States.  Under this program, facilities in the eastern
              States can buy and sell NOx emissions allowances. As noted in the NOxBudget
              Trading Program Compliance and Environmental Results report for 2007,
              99 percent of the NOx emissions measured under the NOx Budget Program were
              measured by CEMS. In 2007, approximately 700,000 NOx allowances were
              traded in private transactions.11 The price of NOx allowances varied from about
              $500 to $1,000 per ton during 2007, with the year-end closing price at $825 per
              ton.  The total value of NOx allowances traded in 2007 exceeded $350 million.
 Importance of Quality Assurance
         Tests for CEMS

"Sources are required to conduct
stringent quality assurance tests of
their monitoring systems, such as
daily and quarterly calibration tests
and a semiannual or annual relative
accuracy test audit. These tests ensure
that sources report accurate data and
provide assurance to market
participants that a ton of emissions
measured at one facility is equivalent
to a ton measured at a different
facility."

Source: NOx Budget Trading Program
Compliance and Environmental Results
2007, EPA-430-R-08-008, December 2008
                       10
10 Based on total annual volume of 15,776,130 tons at an average nominal price of $325 per ton.
11 Private transactions include all transfers initiated by authorized account representatives for any compliance or
general account purposes.
                                            11

-------
                                                                               09-P-0235
              The importance of calibration gases could increase as EPA expands the use of
              emissions trading programs. On March 10, 2005, EPA announced the Clean Air
              Interstate Rule (CAIR).12 CAIR caps emissions for NOx and SO2 for 28 eastern
              States and the District of Columbia. States must achieve required emissions
              reductions by requiring power plants to participate in an interstate cap and trade
              program or meet individual State emissions budget by implementing its own
              measures. Under CAIR, States may choose to participate in an EPA-administered
              regional trading program.

              In order to address climate change, Congress is considering a cap and trade
              program to control greenhouse gas emissions.  The primary greenhouse gases
              include CO2, methane, nitrous oxide, and synthetically produced fluorinated
              gases.13  Implementing a greenhouse gas emissions cap and trade program could
              greatly expand the use and importance of EPA Protocol Gases if such a program
              included monitoring and calibration requirements similar to the Acid Rain
              Program.

              Calibration Gases Help Assure the Accuracy of Ambient Monitoring
              Measurements

              EPA Protocol Gases also help assure the accuracy of ambient air quality monitors
              used to collect ambient air quality data. EPA Protocol Gases are needed to
              calibrate the monitoring systems for three National Ambient Air Quality
              Standards - CO, NO2, and SO2.  Based on the ambient monitoring  data, EPA
              determines whether the air quality in an area meets or exceeds the standard.  If the
              air quality exceeds the standard, the economic cost to the community for control
              measures needed to reduce emissions and comply with the standard can be in the
              hundreds of millions of dollars.  Conversely, if EPA were to determine that an
              area met the standard because of monitoring data that produced inaccurately low
              data, control measures may not be implemented to protect the public health.
              Consequently, it is extremely important that the monitoring data are accurate so
              that EPA can make the proper attainment decision.

              The EPA Protocol Gases we tested were tri-blend cylinders with gaseous
              concentration levels generally used by utilities to calibrate CEMS.  Ambient air
              monitoring systems operators would generally use EPA Protocol Gases of lower
              concentrations to calibrate their  monitors, and perform other quality assurance
              tests. For example, low concentration tri-blend EPA Protocol Gas  cylinders are
              used by OAQPS when they conduct the National Performance Audit Program to
              assess the accuracy of ambient air quality monitors. Since we did not test gases of
              the lower concentration range used by OAQPS for the National Performance
              Audit Program, we cannot comment on their quality.
  On December 23, 2008, the DC Circuit Court of Appeals reversed an earlier decision and granted EPA's petition
to remand the rule to EPA without vacatur. This ruling means that the regulation remains in effect but EPA must
remedy CAIR's flaws in accordance with the Court's July 11, 2008, opinion in the case.
13 Hydrofluorocarbons, perfluorocarbons, and sulfur hexafluoride.
                                           12

-------
                                                                            09-P-0235


Conclusions

             Acid Rain regulations specify accuracy criterion for EPA Protocol Gases and
             require vendors to participate in an EPA verification program. Ambient
             monitoring regulations also require the use of EPA Protocol Gases and gas vendor
             participation in a verification program. However, EPA has not implemented a
             program to provide reasonable assurance that EPA Protocol Gases are accurate.
             In addition, the Protocol for preparing EPA Protocol Gases has not been revised
             since 1997 and programmatic responsibilities for maintaining the Protocol are
             unclear.

Recommendations

             We recommend that the EPA Assistant Administrator for Air and Radiation:

             2-1     Implement an oversight program to provide reasonable assurance of the
                    quality of EPA Protocol Gases used to calibrate continuous emissions
                    monitoring systems for EPA's Acid Rain Cap and Trade Program, and
                    other stationary source air programs.

             2-2    Implement an oversight program to provide reasonable assurance of the
                    quality of EPA Protocol Gases used to calibrate continuous ambient air
                    monitors for the NAAQS monitoring program.

             We recommend that the EPA Assistant Administrator for Research and
             Development:

             2-3     Update and maintain the EPA Traceability Protocol to meet the defined
                    objectives of the Acid Rain, NAAQS, and other stationary source air
                    programs.

Agency Comments and OIG  Evaluation

             OAR and ORD agreed with the report's recommendations.

                 •   For Recommendation 2-1, OAR plans to propose a rule for public comment
                    in fall 2009 to establish a largely self-supported, annual gas audit program of
                    protocol gases used for the Acid Rain program.  In the interim, OAR has
                    already initiated efforts to conduct another OAR-funded test of EPA Protocol
                    Gases gas later in 2009.

                 •   For Recommendation 2-2, OAR plans to implement a separate verification
                    program to address the lower concentration protocol gases used to calibrate
                    continuous ambient air monitors. Using analytical support from EPA
                    Regions 2 and 7, OAR plans to verify EPA Protocol Gases supplied to State,
                                          13

-------
                                                                 09-P-0235
       local, and tribal agencies for calibration of continuous, ambient gaseous
       pollutant monitors.  OAR plans to launch this program by March 2010.

   •   For Recommendation 2-3, ORD will update and publish, as guidance, a
       revised protocol to reduce the percentage of EPA Protocol Gases that do not
       attain the Acid Rain Program's accuracy criterion. ORD plans to complete
       and issue the revised protocol by October 1, 2010. ORD also provided a
       corrective action plan in response to Recommendation 2-3 (see Appendix D).

The general actions outlined in OAR's and ORD's responses meet the intent of our
recommendations. Recommendations 2-1 and 2-2 will remain open in our tracking
system pending our receipt and approval of OAR's final corrective action plan.
ORD's proposed corrective action plan sufficiently addresses Recommendation 2-3.
As such, we are closing Recommendation 2-3 in our tracking system upon issuance
of this report. These recommendations will be tracked to completion in the
Agency's tracking system.

With respect to the ambient monitoring program, OAR commented that "the report
demonstrates that the gases tested met the expectations of the EPA Traceability
Protocol with 95 percent confidence limit.  We point out that the gases used for the
ambient air program rely mainly on the traceability protocol. Thus, there are no
findings demonstrating a concern with using protocol gases  for that program."  We
disagree with this characterization. Our tests found that the  low-range concentration
gases performed the worst in meeting the 2 percent accuracy criterion for the Acid
Rain Program. The ambient monitoring program uses even  lower concentration
gases than the ones we tested during our evaluation. The past two Protocol Gas
assessments conducted by EPA have not included these lower concentration gases.
We believe the lack of test data to assure the accuracy of the lower concentration
gases used for the ambient monitoring program is a concern that should be addressed
by EPA.

After receiving OAR's response to the draft report, the OIG met with OAR to
discuss its response. OAR requested that we revise Recommendation 2-2 to state
that OAR should implement an oversight program to provide reasonable assurance of
the quality of EPA Protocol Gases. We made this change to Recommendation 2-2.
OAR and ORD also provided several technical clarifications and comments to the
report. We made changes to the final report based on these comments, as
appropriate.  OAR and ORD's responses to the draft report are contained in
Appendices C and D.
                             14

-------
                                                                                               09-P-0235
                    Status of Recommendations and
                         Potential Monetary Benefits
                                RECOMMENDATIONS
Rec.   Page
No.    No.
                          Subject
                                               Status1
                                                         Action Official
                     Planned
                    Completion
                      Date
                                 POTENTIAL MONETARY
                                  BENEFITS (in $OOOs)2
Claimed   Agreed To
Amount    Amount
2-1     13    Implement an oversight program to provide
             reasonable assurance of the quality of EPA
             Protocol Gases used to calibrate continuous
             emissions monitoring systems for EPA's Acid
             Rain Cap and Trade Program, and other
             stationary source air programs.

2-2     13    Implement an oversight program to provide
             reasonable assurance of the quality of EPA
             Protocol Gases used to calibrate continuous
             ambient air monitors for the NAAQS program.

2-3     13    Update and maintain the EPA Traceability
             Protocol to meet the defined objectives of the
             Acid Rain, NAAQS, and other stationary source
             air prog rams.
Assistant Administrator for
   Air and Radiation
Assistant Administrator for
   Air and Radiation
 Assistant Administrator for  10/01/2010
     Research and
     Development
 O = recommendation is open with agreed-to corrective actions pending;
 C = recommendation is closed with all agreed-to actions completed;
 U = recommendation is undecided with resolution efforts in progress.
 Identification of potential monetary benefits was not an objective of this evaluation.
                                                   15

-------
                                                                              09-P-0235
                                                                       Appendix A

                Details on Scope and Methodology

To test the accuracy of EPA Protocol Gases, we contracted with a third party to purchase EPA
Protocol Gases directly from the specialty gas producers. Our intent was to purchase at least one
set of three EPA Protocol Gas cylinders from each gas producer and manufacturing location in
the country.  After purchasing at least one cylinder from each identified producer14 or
manufacturing location, we purchased additional cylinders from each producer to maximize our
sample size and obtain a better representation of the facilities' production capability. Our
contractor purchased 87 EPA Protocol Gas tri-blend cylinders from 11 different producers.
NIST analyzed these cylinders and compared the results with each cylinder's certified tag values.
Chain of custody procedures were employed to account for each sample cylinder throughout the
process. Both our contractor and NIST conducted their work in accordance with the OIG-
approved quality assurance project plans.

Procedures for Purchasing Sample Gases

We tasked our contractor with surveying the industry to identify EPA Protocol Gas vendors and
manufacturing locations. To accomplish this survey, the contractor developed a set of keywords
to electronically search the Internet and the Thomas Register® for potential vendors of EPA
Protocol Gases.  Our contractor also obtained industry information through conversations with
gas vendor personnel during the process of obtaining vendor quotes. The survey identified a
total of 11 gas producers and 18 manufacturing locations.

We authorized our contractor to purchase gas cylinders from each of the identified vendors with
an initial goal of purchasing at least three tri-blend cylinders of a low-, medium-, and high-
concentration range from each vendor manufacturing location.  After accomplishing this goal,
we purchased additional  cylinders from vendors with a goal of obtaining additional cylinders
from a cross section of vendor types taking into account such factors as cylinder cost and
potential market share. An initial set of three cylinders was purchased from all 18 manufacturing
locations, an additional set of cylinders was purchased from 9 of the 18 locations,  and a third set
was purchased from one of the 18 locations. In all, 87 cylinders (29 sets  of 3) were purchased.
Our contractor purchased the gases from January 30, 2008, to April 29, 2008. The following
table shows the number of cylinders by pollutant and concentration range.

Table A-1: Summary of Sample Gases Purchased
No. of
Cylinders
29
29
29
Range
Type
High
Mid
Low
C02 (%)
18.0
12.0
5.00
NO
(ppm)
900
400
50.0
SO2
(ppm)
1000
500
50.0
Source: EPA Protocol Gas Industry Survey and Blind Audit,
Final Report, February 2009
14
  To ensure cylinders were purchased from every producer, sets of cylinders were purchased from three distributors.
                                           16

-------
                                                                                09-P-0235
NIST Analysis

Under an interagency agreement with the OIG, NIST was tasked with analyzing the
concentrations of the sample cylinders and comparing the test results to the manufacturers'
certified value.  NIST's primary objective in conducting its analysis was to achieve a calculated
uncertainty of ± 0.5 percent or better for its analysis to achieve a 4:1 or better ratio between the
acceptance criterion (i.e., the Acid Rain Program's ± 2 percent accuracy criterion) being tested
and the uncertainty of NIST's analysis. A general standard of practice in metrology is for this
ratio - referred to as TAR or test to accuracy ratio - to be greater than or equal to 4:1. NIST's
analytical uncertainty ranged from ± 0.42 percent to ± 0.68 percent at the 95 percent confidence
level, depending upon the gas analyzed and the concentration.

To be consistent with the reporting of past EPA Protocol Gas tests conducted by EPA, Chapter 2
of this report presents the analytical results without adjustment for the uncertainty of the
analyses. Appendix B presents NIST's results with uncertainty.

Limitations

We instructed all parties involved in this evaluation not to discuss our plans with specialty gas
vendors or persons not needing such information to implement the evaluation. However, several
months prior to purchasing the gas cylinders, we were informed that at least one specialty gas
producer had become aware our intended evaluation. We do not know whether additional
vendors also became aware of our planned evaluation, or whether they would have been able to
identify the orders intended for the tests. If a specialty gas vendor was aware of our evaluation,
the results may  not necessarily represent the quality of gases such a vendor routinely produces
and sells to users, but only indicates its ability  to produce quality EPA Protocol Gases.

Because of a lack of data on the specialty gas vendor market share, we were unable to design a
sample that would allow us to project our results to the entire universe  of EPA Protocol Gases
with an acceptable level of confidence and precision.  Since we were unable to select gas
cylinders for testing in a manner that would ensure a statistically representative sample of the
EPA Protocol Gas market, we did not perform any inferential statistical analysis to project the
sample test results to the overall EPA Protocol Gas market.
                                           17

-------
                                                                             09-P-0235


                                                                          Appendix B

 Test Results With Uncertainty Factored into Analyses

NIST calculated the total uncertainty of its analysis at the 95 percent confidence level for each of
the component ranges it tested.  NIST's calculated analytical uncertainty ranged from ± 0.42
percent to ± 0.68 percent.  To assess the results taking into account the uncertainty of NIST's
analysis, we adjusted the test limits to account for uncertainty using the following formula:

 Test limit = Specification (Acid Rain Program ± 2 percent criterion) - ± Uncertainty (NIST's analytical uncertainty)

The following table shows how the test limits were adjusted using a NIST analytical uncertainty
of ± 0.42 percent as an example.

Table B-1: Example of Test Limits Adjusted for Uncertainty
Test Limit (Expressed as
Percentage Difference)
< - 2.42
-2.42 to -1.58
-1.57 to 1.57
1.58 to 2.42
>2.42
Conclusion if Sample Falls in This Range
Failed with 95 percent confidence
Within the range of uncertainty
Passed with 95 percent confidence
Within the range of uncertainty
Failed with 95 percent confidence
Source: OIG table developed from data in NIST Report of Analysis, December 4, 2008.

When the test results are adjusted for uncertainty, the number of components meeting or
exceeding the Acid Rain Program's accuracy criterion is smaller than the unadjusted test
results presented in Chapter 2. For example, when we factor in uncertainty, we can only
say with 95 percent confidence that 82 percent (214 of 261) of the components met the
criterion, whereas the unadjusted test results show that 89 percent of the components met
the accuracy criterion. Similarly, when we factor in uncertainty, we can only state with 95
percent confidence that 5 percent (12 of 261) of the components did not meet criterion.
The remaining 13 percent (35 of 261) of the components were within the range of NIST's
analytical uncertainty. Table B-2 on the next page summarizes the test results adjusted for
uncertainty.
                                          18

-------
                                                                                   09-P-0235
Table B-2: Summary of Test Results
Category
Met the Acid
Did Not Meet
Test Results
Rain Accuracy Criterion
the Acid Rain Accuracy Criterion
Within the Range of Analytical Uncertainty
Total
Test Results
No.
214
12
35
261
Percentage
82
5
13
100
Source: Developed from data in NIST Report of Analysis, December 4, 2008.

Both the unadjusted and adjusted test results showed that the lower-cost cylinders in our
sample performed worse than higher-cost cylinders. The adjusted results show that 11 of
the 12 gaseous mixtures not meeting the Acid Rain Program's accuracy criterion were
lower-cost cylinders (< $380) certified with GMIS.  The following table summarizes the
tests meeting the accuracy criterion by cost and reference standard used to certify the
concentration.

Table B-3: Results by Vendor's Analytical Reference Standard and Cost
Reference Standard
Used and Cost of Gas
NTRM/SRM < $380
NTRM/SRM > $380
GMIS < $380
GMIS > $380
Unknowna<$380
Totals
No. of Components
Certified with this Standard
8
103
96
50
4
261
Components Meeting the Acid Rain
Accuracy Criterion
No.
6
100
59
47
2
214
Percentage
75
97
61
94
50
82"
Source: Developed from data in NIST Report of Analysis, December 4, 2008
a Vendor documentation did not specify type of reference standard used to certify the gaseous concentration.
b Represents the overall percentage, not the column total.
                                             19

-------
                                                                           09-P-0235


                                                                       Appendix C


              OAR's Response  to  OIG Draft Report

                                   August 5, 2009

MEMORANDUM

SUBJECT:   Comments on Draft IG Report: "EPA Needs an Oversight Program for Protocol
             Gases" Assignment number 2007-000877

FROM:      Gina McCarthy  /s/
             Assistant Administrator for Air and Radiation

TO:         Wade  T. Najjum
             Assistant Inspector General for Program Evaluation
      The Office of Air and Radiation (OAR) appreciates the Office of Inspector General's
(OIG) effort to work with our staff on this unique testing program to produce the subject Draft
Final Report "EPA Needs an Oversight Program for Protocol Gases" to improve the quality
assurance of protocol gases. We generally agree with the data described in the draft final report.
However, the issues raised apply differently to the Acid Rain Program's (ARP's) source
emissions monitoring systems and National Ambient Air Quality measurement network systems.
Therefore, this memorandum is structured in two parts - an Acid Rain Program section and an
Ambient Air Program section.

Acid Rain Program

      Our major comments are presented below. We include both major and detailed
comments in the attached Word document in Track  Changes.  After you've reviewed our
comments, we'd like to have another meeting with you as soon as possible.

•  At the end of the first paragraph in "At a Glance", please add: "However, 96 percent were
   within 3 percent of the true value; and there was no bias associated with the 28 gas
   components that did not meet the ARP performance specification. Based on the analysis
   results, there are no anticipated nationwide environmental consequences of the calibration
   gases used in the ARP. However, the ARP is addressing potential source-level problems in
   two ways:  (1) by performing an EPA-funded gas audit in 2009; and (2) by proposing a rule
   establishing a largely self-supported, ongoing, annual gas audit program."

•  Add the following paragraph to the "At a Glance" section:

      "Due to the nature of our sample selection and a lack of data on vendor market shares, we
were unable to develop a sampling methodology to  ensure that the number of samples selected from
each producer represented the number of cylinders produced and sold to consumers. Consequently,


                                        20

-------
                                                                              09-P-0235
we could not project our test results to the universe of EPA Protocol Gases, or estimate the potential
impact that gases not meeting the Acid Rain Program's accuracy criteria could have on that
program. Despite this limitation, our tests represent the largest one-time assessment of the Protocol
Gas industry.  The final results of this study identified some potential areas of concern with respect
to the quality of less expensive gases certified using GMIS."

•  Add the following paragraph to the "Noteworthy Achievements" section:

       "The ARP is currently conducting a third gas audit, covering all known producers of EPA
Protocol gas.  As in past ARP audits, the results will be released to the public. We expect results by
early 2010. To provide a cost-effective, long term audit capability, OAR is also developing a
proposed rule package to implement a largely self-supporting, annual gas audit program.  This
proposed rule is expected to be published for public comment in the Federal Register in the fall of
2009."

       ARP concurs with all of the OIG recommendations.  However, we suggest a slight
rewording for recommendation #2-1 as follows:

2-1    Implement a better oversight program to assure the quality of EPA Protocol Gases used
       to ensure the accuracy of continuous emissions monitoring systems for EPA's Acid Rain
       Program, and other stationary source air programs.

Ambient Air Programs

[OIG NOTE: The comments initially submitted by the Ambient Air Program on August 5, 2009
were superseded by the amended comments provided September 2, 2009, in the following
memorandum.]
       Thank you again for the opportunity to comment on the draft final report.  If you have
questions regarding our comments, please contact Dawn Roddy, OAR Audit Follow-up
Coordinator, at 202-564-1228.
                                          21

-------
                                                                              09-P-0235
                                  September 2, 200915
MEMORANDUM

SUBJECT:   Amended Comments on Draft IG Report: "EPA Needs an Oversight Program for
             Protocol Gases" Assignment Number 2007-000877

FROM:      Gina McCarthy  /s/
             Assistant Administrator for Air and Radiation

TO:         Wade T. Najjum
             Assistant Inspector General for Program Evaluation

   The Office of Air and Radiation (OAR) appreciates the Office of Inspector General's (OIG)
effort to work with our staff on this unique testing program to verify the quality assurance of
protocol gases as published in the subject Draft Final Report "EPA Needs an Oversight Program
for Protocol Gases". In a follow-up meeting between our staffs, your office explained that
recommendation 2-2 was directed toward the program under development and outlined by the
Office of Air Quality Planning and Standards (OAQPS) during the audit. Consequently, we
agree with recommendation 2-2 and provide the following amended paragraph titled "Ambient
Air Program", along with new recommendations to amend our August 5, 2009 response
concerning the ambient air program's protocol gases.

Ambient Air Program

    The report demonstrates that the gases tested met the expectations of the EPA Traceability
Protocol with 95 percent confidence limit. We point out that the gases used for the ambient air
program rely mainly on the traceability protocol. Thus, there are no findings demonstrating a
concern with using protocol gases for that program.  Determination of compliance with an ambient
air standard is completed with consideration of many factors including the quality of the data. We
believe updating the traceability protocol is the most efficient and effective corrective action to
improve the quality of gases and subsequently the data collected for the ambient air program.
Revising the protocol will also benefit stationary source emissions measurements.  We, therefore,
agree with recommendation 2-2 and have already started creation of an appropriate oversight
program to reasonably assure the quality of protocol gases for the ambient air monitoring program.
We amend our August 5, 2009 response:

    1.  Replace the sentence in the "At A Glance" section that says, "However, as of April 2009,
       EPA had not implemented a verification program, primarily due to indecision over how to
       fund the program." with "However, as of April 2009, OAQPS, with commitments by EPA
       Regions 2 and 7 for analytical support, had begun development of a program to provide
       independent verification of EPA Protocol Gas cylinders used for continuous ambient air
       monitors."
 ' The OAR memorandum was undated but was received by OIG on September 2, 2009.
                                          22

-------
                                                                             09-P-0235
   2.  Add the following paragraph to the "Noteworthy Achievements" section: "OAQPS has
       planned and begun development of the Protocol Gas Verification Program to independently
       verify "EPA Protocol Gases" supplied to state, local and Tribal agencies for calibration of
       continuous, ambient gaseous pollutant monitors. EPA Regions 2 and 7 have committed to
       providing analytical support. The program is expected to be launched in March 2010.

   Finally, we suggest the following rewording for recommendation 2-2:  "Implement an oversight
program, which reasonably assures the quality of EPA Protocol Gases used to calibrate continuous
ambient air monitors for the National Ambient Air Quality Standards monitoring program as
specified in 40 CFR part 58 Appendix A."

       Thank you again for the opportunity to comment on the draft final report. If you have
questions regarding our comments, please contact Dawn Roddy, OAR Audit Follow-up
Coordinator, at 202-564-1228.
                                          23

-------
                                                                        09-P-0235


                                                                     Appendix D

             ORD's Response to OIG Draft Report

                                   July 30, 2009

MEMORANDUM

SUBJECT:  ORD Response to OIG's Draft Evaluation Report: EPA Needs an Oversight
            Program for Protocol Gases (Assignment No. 2007-00877)

FROM:     Lek G. Kadeli  /s/
            Acting Assistant Administrator (8101R)

TO:         Wade T. Najjum
            Assistant Inspector General for Program Evaluation (2460T)
      Thank you for the opportunity to provide comments on the draft Office of the Inspector
General (OIG) report entitled "EPA Needs an Oversight Program for Protocol Gases."

      Attached are the Office of Research and Development's (ORD) specific comments on the
draft report. In general, ORD concurs with the recommendation that ORD should update and
maintain the EPA Traceability Protocol for Gaseous Calibration Standards.  We appreciate your
consideration of our comments. If there are any questions please do not hesitate to call me at 202-
564-6620 or Jorge Rangel at 202-564-1606.
Attachment
                                       24

-------
                                                               09-P-0235
      Office of Research and Development Comments on

    Office of Inspector General Draft Audit Report, "EPA

      Needs an Oversight Program for Protocol Gases",

           Assignment No. 2007- 000877, July 8, 2009

This document is comprised of four sections:

1.  General comments regarding the OIG recommendations
2.  Table of ORD Response to OIG recommendations
3.  Proposed detailed schedule of corrective actions
4.  Specific comments by page number

1. General Comments regarding the OIG Recommendations

Thank you for the opportunity to comment on the draft Office of the Inspector General
(OIG) report entitled "EPA Needs an Oversight Program for Protocol Gases." EPA's Office of
Research and Development (ORD) concurs with the recommendation that ORD should update and
maintain the EPA Traceability Protocol for Gaseous Calibration Standards. We believe that the
scientific expertise within ORD puts it in a unique position to understand the gas metrological issues
that need to be addressed in the update and to prepare the update.

2. Table of ORD Response to OIG Recommendations
Rec.
No.
2-3









OIG Recommendation
Update and maintain the
EPA Traceability Protocol
to meet the defined
objectives of the Acid
Rain, NAAQS, and other
stationary source air
programs.



Action
Official
Assistant
Administrator
for Research
and
Development





ORD
Corrective
Actions
Update and
publish (as
guidance) a
revised protocol
to reduce the
percentage of
EPA Protocol
Gases that do not
attain the CAMD
accuracy criteria
Planned
Completion
Date
10/01/2010









3. Proposed Detailed Schedule of Corrective Actions
Corrective Action Taken
Final audit report disseminated to public
List of potential protocol changes* sent to
NIST and specialty gas producers for
Planned Completion Date
October 1, 2009
October 15, 2009
                                  25

-------
                                                                                          09-P-0235
Corrective Action Taken
review with request for their comments,
suggested revised text, and any
concentration stability data supporting
changes to allowable gas composition,
concentration ranges, minimum cylinder
pressures, and certification periods
Receive all comments and stability data
Complete internal review draft of protocol
Receive EPA comments on internal draft
Complete external review draft of protocol
by NIST and specialty gas producers
Receive external comments on revisions
Revised protocol published as guidance
Planned Completion Date

November 15, 2009
May 15,2010
June 1,2010
July 1, 2010
August 15,2010
October 1, 2010
* In 2005, EPA sent the following list of informal change suggestions to NIST and specialty gas producers:
    - include a procedure specifically for the use of Fourier Transform infrared (FTIR) spectroscopy;
    - improve statistical procedures for uncertainty calculations and revise the Excel spreadsheet;
    - include mercury calibration standards in the protocol;
    - eliminate (or retain) gas manufacturers intermediate standards (GMISs);
    - lengthen the certification periods for EPA Protocol Gases;
    - change the cylinder pressure limitations for EPA Protocol Gases;
    - include a procedure for analyzing and certifying zero gases;
    - include gas dilution systems in the protocol;
    - tighten documentation requirements;
    - develop audit participant identification number for regulatory data reporting purposes;
    - require ISO 17025 accreditation for producers of environmental calibration gases;
    - include provision for preparation and analysis of batches of EPA Protocol Gases;
    - require on-site visits of specialty gas producers by EPA representatives; and
    - provide for technical assistance/outreach to specialty gas producers by NIST representatives;


4.      Specific comments  by page number

[OIG NOTE:  ORD also provided several technical clarifications and comments to the draft
report.  We made changes to the final report based on these comments, as appropriate.]
                                                 26

-------
                                                                            09-P-0235
                                                                         Appendix E

                                 Distribution
Office of the Administrator
Assistant Administrator for Air and Radiation
Acting Assistant Administrator for Research and Development
Deputy Assistant Administrator for Air and Radiation
Deputy Assistant Administrator for Research and Development
Director, Office of Air Quality Planning and Standards, Office of Air and Radiation
Director, Office of Atmospheric Programs, Office of Air and Radiation
Director, National Risk Management Research Laboratory, Office of Research and Development
Agency Follow-up Official (the CFO)
Agency Follow-up Coordinator
General  Counsel
Associate Administrator for Congressional and Intergovernmental Relations
Associate Administrator for Public Affairs
Audit Follow-up Coordinator, Office of Air and Radiation
Audit Follow-up Coordinator, Office of Research and Development
Acting Inspector General
                                         27

-------