U.S. ENVIRONMENTAL PROTECTION AGENCY OFFICE OF INSPECTOR GENERAL Catalyst for Improving the Environment Evaluation Report EPA Needs an Oversight Program for Protocol Gases Report No. 09-P-0235 September 16, 2009 ------- Report Contributors: Jim Hatfield Geoff Pierce Rick Beusse Abbreviations CAA Clean Air Act CAIR Clean Air Interstate Rule CEMS Continuous Emissions Monitoring System CFR Code of Federal Regulations CO Carbon monoxide CC>2 Carbon dioxide EPA U.S. Environmental Protection Agency GMIS Gas Manufacturer's Intermediate Standard NAAQS National Ambient Air Quality Standards NIST National Institute of Standards and Technology NO Nitric oxide NOX Nitrogen oxides NO2 Nitrogen dioxide NTRM NIST Traceable Reference Material OAP Office of Atmospheric Programs OAQPS Office of Air Quality Planning and Standards OAR Office of Air and Radiation OIG Office of Inspector General ORD Office of Research and Development PGVP Protocol Gas Verification Program ppm parts per million SO2 Sulfur dioxide SRM Standard Reference Material Cover photo: Gas cylinders stored at the National Institute of Standards and Technology laboratory in Gaithersburg, Maryland. (Photo courtesy National Institute of Standards and Technology) ------- U.S. Environmental Protection Agency Office of Inspector General At a Glance 09-P-0235 September 16, 2009 Catalyst for Improving the Environment why we Did This Review EPA /Veec/s a/7 Oversight Program for Protocol Gases The accuracy of continuous emissions monitors is critical to the U.S. Environmental Protection Agency's (EPA's) Acid Rain Program because data from these monitors determine the number of allowances a utility can bank, sell, or trade. Specialized gases known as "EPA Protocol Gases" are used to calibrate and assure the quality of these monitors. We sought to determine whether certified concentrations of these gases are accurate. Background Vendors produce and certify EPA Protocol Gases in high- pressure cylinders according to EPA procedures. EPA regulations require the use of these gases, or National Institute of Standards and Technology (NIST)-certified reference materials, when conducting quality assurance for continuous emissions and ambient air monitoring systems. EPA's Acid Rain Program requires that the certified concentration of the I gases be within ± 2 percent of the true concentration. For further information, contact our Office of Congressional, Public Affairs and Management at (202) 566-2391. To view the full report, click on the following link: www.epa.qov/oiq/reports/2009/ 20090916-09-P-0235.pdf What We Found We purchased 87 cylinders of EPA Protocol Gases and had NIST analyze each cylinder to determine whether the three gaseous mixtures contained in each cylinder (carbon dioxide, nitric oxide, and sulfur dioxide) met the Acid Rain Program's accuracy criterion. We found that 89 percent (233 components) met the Acid Rain Program's accuracy criterion and 11 percent (28 components) did not. Of the 28 components that did not meet the criterion, 17 were within 3.0 percent of the NIST-determined true concentration; 7 were within 3.0 to 5.0 percent; and 4 exceeded the true concentration by more than 5.0 percent. Our sample was not designed to estimate the impact of the test results on the Acid Rain Program. However inaccurately certified concentrations could cause system operators to unknowingly calibrate their monitoring systems to record inaccurate measurements. For example, if a utility overstates its emissions, it could lose the opportunity to sell allowances to other utilities. If a utility understates its emissions, the utility and regulators may incorrectly conclude that the source is complying with emissions standards. With respect to ambient air monitoring, the accuracy of these monitors is important because the data are used to determine whether areas are in compliance with the Nation's ambient air quality standards. EPA has conducted only two tests of the accuracy of EPA Protocol Gases since 1997, when EPA's Office of Research and Development discontinued its annual testing program. Thus, EPA does not have reasonable assurance that the gases used to calibrate emissions monitors for the Acid Rain Program and continuous ambient monitors for the Nation's air monitoring network are accurate. What We Recommend We recommend that the Office of Air and Radiation (OAR) implement oversight programs to assure the quality of EPA Protocol Gases used to calibrate continuous emissions monitoring systems and ambient air monitors. We also recommend that the Office of Research and Development (ORD) update and maintain the protocol gas procedures to ensure that the protocol meets the objectives of the Acid Rain, ambient air, and stationary source air programs. OAR and ORD concurred with our recommendations. OAR has initiated efforts to conduct another gas audit later this year. OAR also plans to propose a rule later in 2009 to establish a largely self-supported, annual gas audit program of protocol gases used for the Acid Rain Program. Further, OAR plans to implement a separate verification program to address the lower concentration protocol gases used to calibrate continuous ambient air monitors. ORD will update and maintain the protocol gas procedures. EPA's planned actions meet the intent of our recommendations. ------- UNITED STATES ENVIRONMENTAL PROTECTION AGENCY WASHINGTON, D.C. 20460 OFFICE OF INSPECTOR GENERAL MEMORANDUM SUBJECT: FROM: TO: September 16, 2009 EPA Needs an Oversight Program for Protocol Gases Report No. 09-P-0235 tt^ ^J^- Wade T. Najjum A-i Assistant Inspector General for Program Evaluation Gina McCarthy Assistant Administrator for Air and Radiation Lek G. Kadeli Acting Assistant Administrator for Research and Development This is our report on the subject evaluation conducted by the Office of Inspector General (OIG) of the U.S. Environmental Protection Agency (EPA). This report contains findings that describe the problems the OIG has identified and corrective actions the OIG recommends. This report represents the opinion of the OIG and does not necessarily represent the final EPA position. Final determinations on matters in this report will be made by EPA managers in accordance with established audit resolution procedures. The estimated cost of this report - calculated by multiplying the project's staff days by the applicable daily full cost billing rates in effect at the time, including the costs of purchasing EPA Protocol Gases and having them analyzed by the National Institute of Standards and Technology -is $665,846. Action Required In accordance with EPA Manual 2750, EPA 's Audit Management Process, the Office of Air and Radiation should provide a written response within 90 calendar days. The Office of Air and Radiation's response should include a corrective action plan and planned completion dates for Recommendations 2-1 and 2-2. The Office of Research and Development submitted a corrective action plan that sufficiently addresses Recommendation 2-3. As such, we are "closing" Recommendation 2-3 in our tracking system upon issuance of this report. These recommendations will be tracked to completion in the Agency's tracking system. No further ------- response is required for Recommendation 2-3. As outlined in EPA Manual 2750, the Agency is responsible for tracking the implementation of these actions in its Management Audit Tracking System. We have no objections to the further release of this report to the public. This report will be available at http://www.epa.gov/oig. If you or your staff have any questions regarding this report, please contact me at (202) 566-0832 or najjum.wade@epa.gov, or Rick Beusse at (919) 541-5747 or beusse.rick@epa.gov. ------- EPA Needs an Oversight Program for Protocol Gases 09-P-0235 Table of Contents Chapters 1 Introduction 1 Purpose 1 Background 1 Noteworthy Achievements 5 Scope and Methodology 5 2 EPA Needs an Oversight Program to Assure the Accuracy of Protocol Gases 7 EPA Does Not Have an Oversight Program to Assure the Accuracy of Protocol Gases 7 Low Concentration NO and SO2 Gases and Lower-Cost Cylinders Performed Worst in OurTests 8 Calibration Gases Play an Important Role in Assuring Data Quality for EPA Air Programs 10 Conclusions 13 Recommendations 13 Agency Comments and OIG Evaluation 13 Status of Recommendations and Potential Monetary Benefits 15 Appendices A Details on Scope and Methodology 16 B Test Results with Uncertainty Factored into Analyses 18 C OAR's Response to OIG Draft Report 20 D ORD's Response to OIG Draft Report 24 E Distribution 27 ------- 09-P-0235 Chapter 1 Introduction Purpose The U.S. Environmental Protection Agency (EPA) Protocol Gases are used to calibrate and test continuous emissions monitoring systems (CEMS) and ambient air quality monitoring systems. EPA requires the use of CEMS for large utilities covered by the Acid Rain Program and other clean air programs. Consequently, the accuracy of the EPA Protocol Gases is critical to ensuring the integrity of emissions trading and other EPA programs. We conducted this evaluation to determine whether the certified concentrations of EPA Protocol Gases were within acceptable limits for accuracy. Background The integrity and effectiveness of EPA's air programs depend upon collecting accurate pollutant emissions and air quality data. EPA requires that the systems collecting these data periodically undergo certain quality assurance procedures to assure their accuracy. These procedures include daily calibration and accuracy tests of the monitoring systems. Calibration tests check the accuracy of these monitoring systems by running a gas mixture of a known concentration through the systems and comparing the systems' readings with the known concentration. The systems are adjusted accordingly to eliminate measurement inaccuracies. The certified concentrations of gaseous mixtures used to calibrate and test monitoring systems should accurately reflect the true gaseous concentration. EPA regulations require that the gases used for calibrating and testing the accuracy of ambient air quality analyzers and continuous emissions monitors1 be traceable to either a National Institute of Standards and Technology (NIST) Standard Reference Material (SRM) or a NIST Traceable Reference Material (NTRM). EPA has established an EPA Traceability Protocol that specialty gas producers must follow when preparing calibration gases to meet the NIST traceability requirement. The gases produced in accordance with these standards are referred to as EPA Protocol Gases. This protocol was designed by EPA, NIST, and specialty gas producers so that a laboratory analyst can analyze and certify compressed gas calibration standards with low uncertainty and with traceability to NIST. The protocol specifies a general analytical procedure for determining the concentration of the standards, but it does not specify the analytical instrumentation or the gas-handling 1 Continuous emissions monitors continuously measure pollutants emitted into the atmosphere in exhaust gases from combustion or industrial processes. ------- 09-P-0235 apparatus that the analyst must use during the analysis. In summary, the protocol requires that (1) a NIST-certified gaseous mixture be used as the analytical reference standard, (2) a monthly calibration curve for the instrumentation be established, and (3) both the NIST-certified gaseous mixture and the candidate EPA Protocol Gas be measured at least three times, with reactive gas mixtures measured at least three additional times at least 7 days after the first set of measurements. The protocol has detailed statistical procedures for estimating the total uncertainty of these standards. Although it includes procedures for estimating uncertainty, the protocol does not specify that EPA Protocol Gases meet any given level of uncertainty and does not establish specific accuracy criteria for the standards' uncertainty. However, the Acid Rain Program has prescribed an uncertainty standard for EPA Protocol Gases used to calibrate CEMS as described in the following section. Acid Rain Program Requirements for EPA Protocol Gases The Acid Rain Program implements Title IV of the Clean Air Act (CAA) to reduce emissions of sulfur dioxide (SO2) and nitrogen oxides (NOX), the primary causes of acid rain. The program uses an allowance trading system to reduce 862 emissions. Under this system, affected utility units are allocated allowances (one allowance = one ton of 802) that permit a unit to emit SC>2 during or after a specified year. Utilities can buy, sell, or bank allowances. The program began in 1995. For each ton of 862 emitted in a given year, one allowance is retired and can no longer be used. The program is phased in, with the final SC>2 cap set at 8.95 million tons in 2010. The Acid Rain Program does not use a trading system2 to reduce emissions for NOX, but instead establishes emissions limits for covered units that are designed to achieve the program's goal of reducing NOX emissions by 2 million tons from 1980 levels. Continuous emissions monitoring is instrumental in ensuring that the mandated reductions of 862 and NOX under the Acid Rain Program are achieved. Accordingly, EPA has established requirements for continuously monitoring SO2 and NOX (and other parameters) for units regulated under the Acid Rain Program. Appendices A and B to 40 Code of Federal Regulations (CFR) Part 75 describe specification and test procedures for these monitors, which include daily calibration tests and periodic accuracy tests. The regulations require that the calibration gases used to conduct these tests must be NIST-certified reference standards or that their concentrations must be traceable to NIST-certified reference standards according to the EPA Traceability Protocol for Assay and Certification of Gaseous Calibration Standards. In January 1993, EPA's Office of Atmospheric Programs (OAP) included a 2.0 percent accuracy criterion in its final Acid Rain Program emissions monitoring rule (58 Federal Register 3731). 2 NOX emissions are traded under other air programs, including the NOX Budget Trading Program, which is designed to reduce NOX emissions in order to reduce ambient ozone levels. ------- 09-P-0235 Appendix A to 40 CFR Part 753 specifies that the EPA Protocol Gases must have a certified uncertainty (95 percent confidence interval) that must not be greater than plus or minus (±) 2.0 percent of the certified concentration (tag value) of the gas mixture. In January 2008, EPA revised the Acid Rain Cap and Trade Program regulations to require that producers of EPA Protocol Gases participate in EPA's Protocol Gas Verification Program in order to certify their gases as EPA Protocol Gases. National Ambient Monitoring Program Requirements for EPA Protocol Gases Appendix A to 40 CFR Part 584 describes quality assurance procedures for national ambient air monitors. States and EPA use these monitors to determine whether an area's air quality meets or exceeds the National Ambient Air Quality Standards (NAAQS). The quality control procedures include biweekly one-point checks of carbon monoxide (CO), 862, and nitrogen dioxide (NC^), and annual performance evaluations that test the monitors against gases of known concentrations for at least three different ranges of concentrations. Appendix A to this regulation requires that cylinders of compressed gas used to obtain test concentrations must be traceable to either a NIST Traceable Reference Material or a Gas Manufacturer's Intermediate Standard, certified in accordance with EPA's Traceability Protocol for Assay and Certification of Gaseous Calibration Standards. Further, vendors advertising certification with the procedures and distributing gases as "EPA Protocol Gas" must participate in the EPA Protocol Gas Verification Program or not use "EPA" in any form of advertising. Unlike the Acid Rain Program, the ambient monitoring regulations do not specify accuracy criteria for the EPA Protocol Gases used to calibrate ambient air monitors. Stationary Source Performance Standards Requirements for EPA Protocol Gases The CAA required EPA to establish emissions standards5 for certain stationary sources that cause or contribute significantly to air pollution. These standards are intended to promote use of the best air pollution control technologies and apply to sources that have been constructed or modified since the proposal of the standard. These standards are published in 40 CFR Part 60, and may require the source to install CEMS to demonstrate compliance. 3 Appendix A to Part 75—Specifications and Test Procedures. 4 Appendix A to Part 58—Quality Assurance Requirements for State and Local Air Monitoring Stations, Special Purpose Monitors, and Prevention of Significant Deterioration Air Monitoring. 5 Referred to as New Source Performance Standards. ------- 09-P-0235 Part 60 contains varied requirements for using calibration gases depending upon the EPA method used to conduct emissions testing. Specifically, the methods for determining SC>2 (Method 6C) and NOX (Method 7E) emissions from stationary sources require that the calibration gases be prepared according to the EPA protocol or certified by the tester using a specified wet chemical test method. The electric utility, petroleum, and municipal waste incineration industries are the largest industries (in terms of emissions and the number of sources) subject to continuous emissions monitoring requirements and the use of EPA Protocol Gases for Part 60 emissions standards. EPA's Protocol Gas Verification Program From 1985 to 1997, EPA's Office of Research and Development (ORD) conducted a series of analytical tests6 of EPA Protocol Gases sold by specialty gas producers. In 1996, ORD issued a final draft exit strategy that transferred the administration of this and other quality assurance programs to the Office of Air Quality Planning and Standards (OAQPS) of the Office of Air and Radiation (OAR). ORD provided OAR with resource support during the transition, but provided no further resources after the transfer was completed in 1998. The annual tests were discontinued after 1997, and no additional tests occurred until 2003. Due to a lack of analytical tests and a concern that EPA Protocol Gas quality may have declined, the OAP conducted a test in 2003. OAP's contractor purchased 42 EPA Protocol Gas cylinders containing tri-blend mixtures of carbon dioxide (CO2). nitric oxide (NO), and SO2 from a total of 14 specialty gas vendors nationwide. A third party purchased the cylinders for EPA so that the gas vendors would not know that EPA was analyzing the cylinders. A gas passed the test if the "true" concentration of the gas differed from the certified concentration by ± 2 percent or less. Overall, 89 percent of the gases passed on a gas component basis; 11 percent did not pass the test. OAP conducted another test in 2006 but decided not to publish the results because of concerns about the representativeness of some of the sample gases. Instead of purchasing gases directly from vendors, OAP arranged for end users (i.e., utilities) to provide EPA with the Protocol Gases for testing. However, EPA encountered several problems with this approach, including over- and under-representation of manufacturers and production sites, as well as old and used samples. Approximately 6 years ago, EPA started developing an ongoing EPA Protocol Gas Verification Program (PGVP). After a development process involving U.S. specialty gas companies, NIST, the Institute of Clean Air Companies, and EPA, tentative agreement was reached on a specialty gas company-funded, ongoing draft PGVP. EPA promulgated a requirement for producers to participate in this 6 EPA calls these tests EPA Protocol Audits. We refer to these audits as tests in this report to avoid confusion with the term "audit" as defined by the Government Accountability Office's Government Auditing Standards, and applicable to work conducted by the Office of Inspector General (OIG). ------- 09-P-0235 program in a January 24, 2008, final rule. However, in March 2008, one specialty gas vendor petitioned EPA for reconsideration of the final rule. The vendor cited several concerns including the nature of the funding for the PGVP. Since that time, EPA has been working to resolve these concerns, but still had not funded the program as of May 2009. EPA had planned to have EPA Protocol Gas vendors fund the verification program through payments of fees to a third party. However, EPA has determined such a process could violate the Miscellaneous Receipts Act.7 Noteworthy Achievements Recognizing the importance of EPA Protocol Gases, the OAP conducted two protocol gas assessments, one in 2003 and another in 2006. Although the sample size in each of these assessments was not statistically significant and market share information was not available, both assessments indicated that some specialty gas vendors were not meeting the EPA Protocol Gas accuracy criterion of ± 2.0 percent. OAP posted the results, including the names of the specialty gas companies, on an Agency Website. Based on comments by specialty gas companies, the 2003 accuracy check resulted in some vendors making improvements in personnel, training, quality, and equipment. When we originally approached EPA with our proposal to assess the accuracy of EPA Protocol Gases, OAR supported our plans and cooperated fully with our evaluation team to develop a useful assessment of the EPA Protocol Gas program. Scope and Methodology We conducted this evaluation to test the accuracy of EPA Protocol Gases used to calibrate continuous emissions monitoring equipment. We designed our sampling methodology to test at least one set of three EPA Protocol Gas cylinders from every specialty gas manufacturing location in the country and to maintain the confidentiality of the tests so that the gases we purchased would represent those sold to utilities and other customers.8 In order to maintain the confidentiality of the tests, we contracted with a third party to purchase the gases and deliver them to NIST for analysis. Our contractor purchased tri-blend mixtures of NO, CO2, and SO2, since tri-blend gases are often used by utilities. We identified 11 gas producers nationwide using 18 manufacturing locations to produce EPA Protocol Gases. Our contractor purchased at least one set of EPA Protocol Gas cylinders produced from each of the 18 manufacturing locations. In all, we purchased 87 EPA Protocol Gas cylinders from 14 specialty gas vendors (i.e., 11 producers and 3 distributors), and 7 The Miscellaneous Receipts Act was passed in order to ensure that government agencies did not bypass the appropriations authority of Congress by augmenting their budgets via other means, such as user fees, fees for training courses, parking fees, contract and lease fees and revenues, monetary awards in court cases involving the agencies, court costs and fees, or civil penalties. 8 A set consisted of three cylinders in low-, medium-, and high-range concentrations. ------- 09-P-0235 18 manufacturing locations. Our contractor delivered the sample gases to NIST for analysis. We selected NIST because of its extensive experience in gas metrology and its reputation for analytical accuracy. NIST analyzed the gaseous concentrations of the 87 tri-blend cylinders (i.e., 261 separate component analyses) to determine whether the "true" concentrations were within ± 2 percent of the concentration certified by the vendor.9 We reviewed documentation related to EPA's past assessments of EPA Protocol Gases, including past results, program funding and implementation, and current plans for implementing a revised EPA Protocol Gas Verification Program. We reviewed federal regulations to identify requirements for the use of EPA Protocol Gases. We interviewed staff and managers from ORD, OAQPS, and OAP. We conducted this evaluation in accordance with generally accepted government auditing standards. Those standards require that we obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our evaluation objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our objectives. We conducted our work from June 2007 to July 2009. Appendix A provides more detailed information on our scope and methodology. 9 Vendors typically certify gaseous concentrations using either a NIST Traceable Reference Material (NTRM), a Standard Reference Material (SRM), or a Gas Manufacturer's Intermediate Standard (GMIS). NTRMs are reference material produced by a commercial supplier with a well-defined traceability linkage to NIST. These reference materials are allowed to bear the NIST NTRM certification mark and were developed to allow NIST to respond to increasing needs for high-quality reference materials. SRMs are reference materials produced by NIST and are certified to have specific chemical or physical properties. A GMIS is a NIST traceable standard produced by the gas vendor. A NIST traceable standard is certified with an instrument calibrated through direct traceability with a NIST standard such as an SRM or NTRM. ------- 09-P-0235 Chapter 2 EPA Needs an Oversight Program to Assure the Accuracy of Protocol Gases EPA does not have reasonable assurance that EPA Protocol Gases used to calibrate CEMS for the Acid Rain Program and continuous ambient monitors for the Nation's air monitoring network are accurate. EPA has conducted only two assessments since 1997, when ORD discontinued its program to test the accuracy of EPA Protocol Gases. We analyzed 261 gaseous components and found that 89 percent (233 components) met the Acid Rain Program's accuracy criterion for calibration gases, while 11 percent (28 components) did not meet the accuracy criterion. Calibration gases must be accurate because inaccurate calibration gases can cause a system operator to calibrate a measurement system to produce inaccurate measurements. In turn, inaccurate CEMS measurements can affect the integrity of the computed allowances available for trade on the open market. EPA Does Not Have an Oversight Program to Assure the Accuracy of Protocol Gases EPA does not have a viable oversight program in place to assure the quality of EPA Protocol Gases. Since 1997, EPA oversight to assure the accuracy of these gases has consisted of two analytical tests conducted by OAP in 2003 and 2006. However, EPA did not publish the 2006 test results because of concerns over the sampling methodology. Without an oversight program, EPA does not have reasonable assurance that EPA Protocol Gases meet the accuracy criterion certified to by the vendors and required for the Acid Rain Program. In January 2008, EPA promulgated a final rule requiring EPA Protocol Gas producers to participate in an EPA PGVP. EPA anticipated implementing a verification program to assess the accuracy of Protocol Gases on an ongoing basis. However, implementation was delayed when a specialty gas vendor petitioned EPA to reconsider the rule. As of May 2009, EPA's OAP was still working to resolve legal issues raised in the petition and deciding how to fund the program. As of May 2009, OAR's OAQPS was developing a similar but separate verification program to test the accuracy of the EPA Protocol Gases used to calibrate continuous ambient air monitors. Even though the Acid Rain Program and the ambient air monitoring program generally use different concentrations of EPA Protocol Gases in their programs, the gases must be produced and certified under the same EPA Protocol. Although ORD originally developed the Protocol, a co-author of the Protocol informed us that it was not clear who currently has responsibility for maintaining the Protocol. Although the 2003 EPA Protocol Gas assessment report and specialty gas ------- 09-P-0235 producers have identified several analytical issues not addressed by the current Protocol, the Protocol has not been revised since 1997. Low Concentration NO and SO2 Gases and Lower-Cost Cylinders Performed Worst in Our Tests We purchased and analyzed 87 cylinders of tri-blend EPA Protocol Gases to determine whether each of the three gaseous concentrations (per cylinder) met the Acid Rain Program's accuracy criterion. Of the 261 gaseous components analyzed, 89 percent met the Acid Rain Program's accuracy criterion for EPA Protocol Gases, while 28 (11 percent) did not meet the accuracy criterion. Of the 28 components that did not meet the criterion, 17 were within 3.0 percent of the NIST-determined true concentration; 7 were within 3.0 to 5.0 percent; and 4 exceeded the true concentration by more than 5.0 percent. All exceptions were for gaseous components from lower-cost cylinders (i.e., under $380). Twelve of the gas components' "true" concentrations were higher than the certified concentration, and 16 were lower than the certified concentration. The gaseous components with the most exceptions were low concentration NO and SO2. The certified concentrations for low range NO and SO2 gases exceeded the NIST analyzed or "true" concentration by more than ± 2 percent for 24 and 17 percent of the samples, respectively. The following table summarizes the test results for each range of gaseous components analyzed. Table 2-1: Results by Gas and Concentration Gas NO CO2 SO2 Range Low Med High Low Med High Low Med High Total No. of Samples 29 29 29 29 29 29 29 29 29 261 Met the Accuracy Criterion No. 22 29 25 26 28 29 24 25 25 233 Percentage 76 100 86 90 97 100 83 86 86 89 a Did Not Meet the Accuracy Criterion No. 7 0 4 3 1 0 5 4 4 28 Percentage 24 0 14 10 3 0 17 14 14 11a Source: OIG, developed from data in NIST Report of Analysis, December 4, 2008 a Represents the overall percentage, not the column total. We also assessed the test results on a cost-per-cylinder and production-facility basis. This analysis revealed that all gaseous components not meeting the accuracy criterion were from cylinders costing under $380. All of the gaseous components not meeting the accuracy criterion came from 7 of the 18 (39 percent) facilities. These seven facilities accounted for 108 of the 261 components we tested, and 28 of these 108 components did not meet the Acid Rain Program's ------- 09-P-0235 accuracy criterion. All 153 gaseous components associated with cylinders costing over $380 met the accuracy criterion. The following table shows the total tests conducted, the cost per cylinder, and the test results for each production facility. Table 2-2: Test Results by Cost and Production Facility Production Facility A B N D K Q R Subtota C D E F G H 1 J L M O P Cost Per Cylinder <380 <380 <380 <380a <380 <380 <380 <380 >380 >380a >380 >380 >380 >380 >380 >380 >380 >380 >380 >380 Subtotal >380 Totals Total Tests 18 18 18 18 9 18 9 108 9 9 9 9 18 18 9 9 9 18 18 18 153 261 No. of Tests Meeting the Accuracy Criterion 15 17 9 15 7 10 6 80 9 9 9 9 18 18 9 9 9 18 18 18 153 233 No. of Tests Not Meeting the Accuracy Criterion 3 1 9 2 2 8 3 28 0 0 0 0 0 0 0 0 0 0 0 0 0 28 Source: OIG table developed from data in NIST Report of Analysis, December 4, 2008. 3 Three sets of cylinders (a set consists of a low-, a medium-, and a high-range cylinder) were produced at this location but were purchased from two different vendors. Two sets of cylinders cost under $380 per cylinder and one set cost over $380 per cylinder. We further analyzed the test results, taking into account the vendor's analytical technique and the reference standard used by the vendor to certify its gaseous concentrations. This analysis showed that low-cost cylinders certified with a Gas Manufacturer's Intermediate Standard (GMIS) performed the worst in our tests. Specifically, 24 of the 28 (86 percent) gaseous components that did not meet the Acid Rain Program's accuracy criterion were from lower-cost cylinders that were certified using GMIS. Although most of the gaseous components not meeting the accuracy criterion were certified using GMIS, many GMIS-certified components did meet the accuracy standard. However, all of these gaseous components were from cylinders costing over $380. The following table shows the test results by certification standard and cylinder cost. ------- 09-P-0235 Table 2-3: Pass/Fail Rates By Analytical Reference Standard Reference Standard Used and Cost of Gas Cylinder NTRM/SRM < $380 NTRM/SRM > $380 GMIS < $380 GMIS > $380 Unknown3 < $380 Totals No. of Gaseous Components Certified with this Standard 8 103 96 50 4 261 Components Not Meeting the Acid Rain Program's Accuracy Criterion No. 2 0 24 0 2 28 Percentage 25 0 25 0 50 11" Source: OIG table developed from data in NIST Report of Analysis, December 4, 2008. a Vendor documentation did not specify type of reference standard used to certify the gaseous concentration. b Represents the overall percentage, not the column total. Calibration Gases Play an Important Role in Assuring Data Quality for EPA Air Programs Calibration gases play an important role in helping to assure the quality of data used in EPA programs. Inaccurate calibration gases can negatively affect EPA programs that rely upon accurate emissions or ambient air quality measurements by causing system operators to calibrate their measurements systems inaccurately. For example, if a calibration gas used by a utility was certified to contain 100 parts per million (ppm) of 862, but only contained 96 ppm, the system operator would unknowingly calibrate the CEMS to read 96 ppm as 100 ppm. This would result in the CEMS overestimating emissions. Conversely, if the true concentration of the calibration gas were more than the certified concentration, the system operator would unknowingly calibrate the CEMS to underestimate emissions. The number of monitoring systems potentially using EPA Protocol Gases is significant. For example, over 3,500 electric generating units were covered by the SC>2 allowance trading program in 2007. A subset of almost 1,000 of these 3,500 units was also covered by the NOx provisions of the Acid Rain Program. Further, approximately 1,300 ambient air monitors collect data on CO, SO2, and NOX concentrations. An EPA contractor surveyed Protocol Gas use in preparation for EPA's analytical tests of Protocol Gases in 2003. The contractor estimated that utilities purchased from 37,500 to 75,000 Protocol Gas cylinders annually and missions testing companies purchased from 16,500 to 55,000 cylinders annually. The survey did not estimate annual purchases by State and local agencies, but one large State reported using about 75 Protocol Gases a year for its ambient monitoring network. Due to the nature of our sample selection and a lack of data on vendor market shares, we were unable to develop a sampling methodology to ensure that the number of samples selected from each producer represented the number of cylinders produced and sold to consumers. Consequently, we did not project our test results to the universe of EPA Protocol Gases, or estimate the impact that 10 ------- 09-P-0235 these results could have on the Acid Rain Program. Despite this limitation, our tests represent the largest one-time assessment of the Protocol Gas industry, and revealed some potential areas of concern with respect to the quality of less- expensive gases certified using GMIS. The following sections describe the importance of accurate monitoring data to the Acid Rain and Air Monitoring programs. These data are collected by measurement systems whose accuracy is assured by quality assurance activities such as calibrations using EPA Protocol Gases. Calibration Gases Help Assure Accuracy of Emissions Data for Trading Programs For sources participating in emissions trading programs, overestimating emissions can result in lost opportunities to sell or bank allowances. Underestimating emissions could result in banking or selling allowances that did not really exist, and provide an inaccurate picture of a facility's progress in reducing emissions. The volume of SC>2 allowances traded under the Acid Rain Cap and Trade Program is significant. According to EPA's Acid Rain and Related Programs: 2007 Progress Report, the total value of the SC>2 allowance market was over $5.1 billion in 2007.1U In that same year, 4,700 private transactions involved 16.9 million allowances. In addition to the Acid Rain Cap and Trade Program, utilities and other large combustion sources participate in EPA's NOx Budget Trading Program. This program is designed to reduce emissions of NOx to improve air quality in the eastern part of the United States. Under this program, facilities in the eastern States can buy and sell NOx emissions allowances. As noted in the NOxBudget Trading Program Compliance and Environmental Results report for 2007, 99 percent of the NOx emissions measured under the NOx Budget Program were measured by CEMS. In 2007, approximately 700,000 NOx allowances were traded in private transactions.11 The price of NOx allowances varied from about $500 to $1,000 per ton during 2007, with the year-end closing price at $825 per ton. The total value of NOx allowances traded in 2007 exceeded $350 million. Importance of Quality Assurance Tests for CEMS "Sources are required to conduct stringent quality assurance tests of their monitoring systems, such as daily and quarterly calibration tests and a semiannual or annual relative accuracy test audit. These tests ensure that sources report accurate data and provide assurance to market participants that a ton of emissions measured at one facility is equivalent to a ton measured at a different facility." Source: NOx Budget Trading Program Compliance and Environmental Results 2007, EPA-430-R-08-008, December 2008 10 10 Based on total annual volume of 15,776,130 tons at an average nominal price of $325 per ton. 11 Private transactions include all transfers initiated by authorized account representatives for any compliance or general account purposes. 11 ------- 09-P-0235 The importance of calibration gases could increase as EPA expands the use of emissions trading programs. On March 10, 2005, EPA announced the Clean Air Interstate Rule (CAIR).12 CAIR caps emissions for NOx and SO2 for 28 eastern States and the District of Columbia. States must achieve required emissions reductions by requiring power plants to participate in an interstate cap and trade program or meet individual State emissions budget by implementing its own measures. Under CAIR, States may choose to participate in an EPA-administered regional trading program. In order to address climate change, Congress is considering a cap and trade program to control greenhouse gas emissions. The primary greenhouse gases include CO2, methane, nitrous oxide, and synthetically produced fluorinated gases.13 Implementing a greenhouse gas emissions cap and trade program could greatly expand the use and importance of EPA Protocol Gases if such a program included monitoring and calibration requirements similar to the Acid Rain Program. Calibration Gases Help Assure the Accuracy of Ambient Monitoring Measurements EPA Protocol Gases also help assure the accuracy of ambient air quality monitors used to collect ambient air quality data. EPA Protocol Gases are needed to calibrate the monitoring systems for three National Ambient Air Quality Standards - CO, NO2, and SO2. Based on the ambient monitoring data, EPA determines whether the air quality in an area meets or exceeds the standard. If the air quality exceeds the standard, the economic cost to the community for control measures needed to reduce emissions and comply with the standard can be in the hundreds of millions of dollars. Conversely, if EPA were to determine that an area met the standard because of monitoring data that produced inaccurately low data, control measures may not be implemented to protect the public health. Consequently, it is extremely important that the monitoring data are accurate so that EPA can make the proper attainment decision. The EPA Protocol Gases we tested were tri-blend cylinders with gaseous concentration levels generally used by utilities to calibrate CEMS. Ambient air monitoring systems operators would generally use EPA Protocol Gases of lower concentrations to calibrate their monitors, and perform other quality assurance tests. For example, low concentration tri-blend EPA Protocol Gas cylinders are used by OAQPS when they conduct the National Performance Audit Program to assess the accuracy of ambient air quality monitors. Since we did not test gases of the lower concentration range used by OAQPS for the National Performance Audit Program, we cannot comment on their quality. On December 23, 2008, the DC Circuit Court of Appeals reversed an earlier decision and granted EPA's petition to remand the rule to EPA without vacatur. This ruling means that the regulation remains in effect but EPA must remedy CAIR's flaws in accordance with the Court's July 11, 2008, opinion in the case. 13 Hydrofluorocarbons, perfluorocarbons, and sulfur hexafluoride. 12 ------- 09-P-0235 Conclusions Acid Rain regulations specify accuracy criterion for EPA Protocol Gases and require vendors to participate in an EPA verification program. Ambient monitoring regulations also require the use of EPA Protocol Gases and gas vendor participation in a verification program. However, EPA has not implemented a program to provide reasonable assurance that EPA Protocol Gases are accurate. In addition, the Protocol for preparing EPA Protocol Gases has not been revised since 1997 and programmatic responsibilities for maintaining the Protocol are unclear. Recommendations We recommend that the EPA Assistant Administrator for Air and Radiation: 2-1 Implement an oversight program to provide reasonable assurance of the quality of EPA Protocol Gases used to calibrate continuous emissions monitoring systems for EPA's Acid Rain Cap and Trade Program, and other stationary source air programs. 2-2 Implement an oversight program to provide reasonable assurance of the quality of EPA Protocol Gases used to calibrate continuous ambient air monitors for the NAAQS monitoring program. We recommend that the EPA Assistant Administrator for Research and Development: 2-3 Update and maintain the EPA Traceability Protocol to meet the defined objectives of the Acid Rain, NAAQS, and other stationary source air programs. Agency Comments and OIG Evaluation OAR and ORD agreed with the report's recommendations. • For Recommendation 2-1, OAR plans to propose a rule for public comment in fall 2009 to establish a largely self-supported, annual gas audit program of protocol gases used for the Acid Rain program. In the interim, OAR has already initiated efforts to conduct another OAR-funded test of EPA Protocol Gases gas later in 2009. • For Recommendation 2-2, OAR plans to implement a separate verification program to address the lower concentration protocol gases used to calibrate continuous ambient air monitors. Using analytical support from EPA Regions 2 and 7, OAR plans to verify EPA Protocol Gases supplied to State, 13 ------- 09-P-0235 local, and tribal agencies for calibration of continuous, ambient gaseous pollutant monitors. OAR plans to launch this program by March 2010. • For Recommendation 2-3, ORD will update and publish, as guidance, a revised protocol to reduce the percentage of EPA Protocol Gases that do not attain the Acid Rain Program's accuracy criterion. ORD plans to complete and issue the revised protocol by October 1, 2010. ORD also provided a corrective action plan in response to Recommendation 2-3 (see Appendix D). The general actions outlined in OAR's and ORD's responses meet the intent of our recommendations. Recommendations 2-1 and 2-2 will remain open in our tracking system pending our receipt and approval of OAR's final corrective action plan. ORD's proposed corrective action plan sufficiently addresses Recommendation 2-3. As such, we are closing Recommendation 2-3 in our tracking system upon issuance of this report. These recommendations will be tracked to completion in the Agency's tracking system. With respect to the ambient monitoring program, OAR commented that "the report demonstrates that the gases tested met the expectations of the EPA Traceability Protocol with 95 percent confidence limit. We point out that the gases used for the ambient air program rely mainly on the traceability protocol. Thus, there are no findings demonstrating a concern with using protocol gases for that program." We disagree with this characterization. Our tests found that the low-range concentration gases performed the worst in meeting the 2 percent accuracy criterion for the Acid Rain Program. The ambient monitoring program uses even lower concentration gases than the ones we tested during our evaluation. The past two Protocol Gas assessments conducted by EPA have not included these lower concentration gases. We believe the lack of test data to assure the accuracy of the lower concentration gases used for the ambient monitoring program is a concern that should be addressed by EPA. After receiving OAR's response to the draft report, the OIG met with OAR to discuss its response. OAR requested that we revise Recommendation 2-2 to state that OAR should implement an oversight program to provide reasonable assurance of the quality of EPA Protocol Gases. We made this change to Recommendation 2-2. OAR and ORD also provided several technical clarifications and comments to the report. We made changes to the final report based on these comments, as appropriate. OAR and ORD's responses to the draft report are contained in Appendices C and D. 14 ------- 09-P-0235 Status of Recommendations and Potential Monetary Benefits RECOMMENDATIONS Rec. Page No. No. Subject Status1 Action Official Planned Completion Date POTENTIAL MONETARY BENEFITS (in $OOOs)2 Claimed Agreed To Amount Amount 2-1 13 Implement an oversight program to provide reasonable assurance of the quality of EPA Protocol Gases used to calibrate continuous emissions monitoring systems for EPA's Acid Rain Cap and Trade Program, and other stationary source air programs. 2-2 13 Implement an oversight program to provide reasonable assurance of the quality of EPA Protocol Gases used to calibrate continuous ambient air monitors for the NAAQS program. 2-3 13 Update and maintain the EPA Traceability Protocol to meet the defined objectives of the Acid Rain, NAAQS, and other stationary source air prog rams. Assistant Administrator for Air and Radiation Assistant Administrator for Air and Radiation Assistant Administrator for 10/01/2010 Research and Development O = recommendation is open with agreed-to corrective actions pending; C = recommendation is closed with all agreed-to actions completed; U = recommendation is undecided with resolution efforts in progress. Identification of potential monetary benefits was not an objective of this evaluation. 15 ------- 09-P-0235 Appendix A Details on Scope and Methodology To test the accuracy of EPA Protocol Gases, we contracted with a third party to purchase EPA Protocol Gases directly from the specialty gas producers. Our intent was to purchase at least one set of three EPA Protocol Gas cylinders from each gas producer and manufacturing location in the country. After purchasing at least one cylinder from each identified producer14 or manufacturing location, we purchased additional cylinders from each producer to maximize our sample size and obtain a better representation of the facilities' production capability. Our contractor purchased 87 EPA Protocol Gas tri-blend cylinders from 11 different producers. NIST analyzed these cylinders and compared the results with each cylinder's certified tag values. Chain of custody procedures were employed to account for each sample cylinder throughout the process. Both our contractor and NIST conducted their work in accordance with the OIG- approved quality assurance project plans. Procedures for Purchasing Sample Gases We tasked our contractor with surveying the industry to identify EPA Protocol Gas vendors and manufacturing locations. To accomplish this survey, the contractor developed a set of keywords to electronically search the Internet and the Thomas Register® for potential vendors of EPA Protocol Gases. Our contractor also obtained industry information through conversations with gas vendor personnel during the process of obtaining vendor quotes. The survey identified a total of 11 gas producers and 18 manufacturing locations. We authorized our contractor to purchase gas cylinders from each of the identified vendors with an initial goal of purchasing at least three tri-blend cylinders of a low-, medium-, and high- concentration range from each vendor manufacturing location. After accomplishing this goal, we purchased additional cylinders from vendors with a goal of obtaining additional cylinders from a cross section of vendor types taking into account such factors as cylinder cost and potential market share. An initial set of three cylinders was purchased from all 18 manufacturing locations, an additional set of cylinders was purchased from 9 of the 18 locations, and a third set was purchased from one of the 18 locations. In all, 87 cylinders (29 sets of 3) were purchased. Our contractor purchased the gases from January 30, 2008, to April 29, 2008. The following table shows the number of cylinders by pollutant and concentration range. Table A-1: Summary of Sample Gases Purchased No. of Cylinders 29 29 29 Range Type High Mid Low C02 (%) 18.0 12.0 5.00 NO (ppm) 900 400 50.0 SO2 (ppm) 1000 500 50.0 Source: EPA Protocol Gas Industry Survey and Blind Audit, Final Report, February 2009 14 To ensure cylinders were purchased from every producer, sets of cylinders were purchased from three distributors. 16 ------- 09-P-0235 NIST Analysis Under an interagency agreement with the OIG, NIST was tasked with analyzing the concentrations of the sample cylinders and comparing the test results to the manufacturers' certified value. NIST's primary objective in conducting its analysis was to achieve a calculated uncertainty of ± 0.5 percent or better for its analysis to achieve a 4:1 or better ratio between the acceptance criterion (i.e., the Acid Rain Program's ± 2 percent accuracy criterion) being tested and the uncertainty of NIST's analysis. A general standard of practice in metrology is for this ratio - referred to as TAR or test to accuracy ratio - to be greater than or equal to 4:1. NIST's analytical uncertainty ranged from ± 0.42 percent to ± 0.68 percent at the 95 percent confidence level, depending upon the gas analyzed and the concentration. To be consistent with the reporting of past EPA Protocol Gas tests conducted by EPA, Chapter 2 of this report presents the analytical results without adjustment for the uncertainty of the analyses. Appendix B presents NIST's results with uncertainty. Limitations We instructed all parties involved in this evaluation not to discuss our plans with specialty gas vendors or persons not needing such information to implement the evaluation. However, several months prior to purchasing the gas cylinders, we were informed that at least one specialty gas producer had become aware our intended evaluation. We do not know whether additional vendors also became aware of our planned evaluation, or whether they would have been able to identify the orders intended for the tests. If a specialty gas vendor was aware of our evaluation, the results may not necessarily represent the quality of gases such a vendor routinely produces and sells to users, but only indicates its ability to produce quality EPA Protocol Gases. Because of a lack of data on the specialty gas vendor market share, we were unable to design a sample that would allow us to project our results to the entire universe of EPA Protocol Gases with an acceptable level of confidence and precision. Since we were unable to select gas cylinders for testing in a manner that would ensure a statistically representative sample of the EPA Protocol Gas market, we did not perform any inferential statistical analysis to project the sample test results to the overall EPA Protocol Gas market. 17 ------- 09-P-0235 Appendix B Test Results With Uncertainty Factored into Analyses NIST calculated the total uncertainty of its analysis at the 95 percent confidence level for each of the component ranges it tested. NIST's calculated analytical uncertainty ranged from ± 0.42 percent to ± 0.68 percent. To assess the results taking into account the uncertainty of NIST's analysis, we adjusted the test limits to account for uncertainty using the following formula: Test limit = Specification (Acid Rain Program ± 2 percent criterion) - ± Uncertainty (NIST's analytical uncertainty) The following table shows how the test limits were adjusted using a NIST analytical uncertainty of ± 0.42 percent as an example. Table B-1: Example of Test Limits Adjusted for Uncertainty Test Limit (Expressed as Percentage Difference) < - 2.42 -2.42 to -1.58 -1.57 to 1.57 1.58 to 2.42 >2.42 Conclusion if Sample Falls in This Range Failed with 95 percent confidence Within the range of uncertainty Passed with 95 percent confidence Within the range of uncertainty Failed with 95 percent confidence Source: OIG table developed from data in NIST Report of Analysis, December 4, 2008. When the test results are adjusted for uncertainty, the number of components meeting or exceeding the Acid Rain Program's accuracy criterion is smaller than the unadjusted test results presented in Chapter 2. For example, when we factor in uncertainty, we can only say with 95 percent confidence that 82 percent (214 of 261) of the components met the criterion, whereas the unadjusted test results show that 89 percent of the components met the accuracy criterion. Similarly, when we factor in uncertainty, we can only state with 95 percent confidence that 5 percent (12 of 261) of the components did not meet criterion. The remaining 13 percent (35 of 261) of the components were within the range of NIST's analytical uncertainty. Table B-2 on the next page summarizes the test results adjusted for uncertainty. 18 ------- 09-P-0235 Table B-2: Summary of Test Results Category Met the Acid Did Not Meet Test Results Rain Accuracy Criterion the Acid Rain Accuracy Criterion Within the Range of Analytical Uncertainty Total Test Results No. 214 12 35 261 Percentage 82 5 13 100 Source: Developed from data in NIST Report of Analysis, December 4, 2008. Both the unadjusted and adjusted test results showed that the lower-cost cylinders in our sample performed worse than higher-cost cylinders. The adjusted results show that 11 of the 12 gaseous mixtures not meeting the Acid Rain Program's accuracy criterion were lower-cost cylinders (< $380) certified with GMIS. The following table summarizes the tests meeting the accuracy criterion by cost and reference standard used to certify the concentration. Table B-3: Results by Vendor's Analytical Reference Standard and Cost Reference Standard Used and Cost of Gas NTRM/SRM < $380 NTRM/SRM > $380 GMIS < $380 GMIS > $380 Unknowna<$380 Totals No. of Components Certified with this Standard 8 103 96 50 4 261 Components Meeting the Acid Rain Accuracy Criterion No. 6 100 59 47 2 214 Percentage 75 97 61 94 50 82" Source: Developed from data in NIST Report of Analysis, December 4, 2008 a Vendor documentation did not specify type of reference standard used to certify the gaseous concentration. b Represents the overall percentage, not the column total. 19 ------- 09-P-0235 Appendix C OAR's Response to OIG Draft Report August 5, 2009 MEMORANDUM SUBJECT: Comments on Draft IG Report: "EPA Needs an Oversight Program for Protocol Gases" Assignment number 2007-000877 FROM: Gina McCarthy /s/ Assistant Administrator for Air and Radiation TO: Wade T. Najjum Assistant Inspector General for Program Evaluation The Office of Air and Radiation (OAR) appreciates the Office of Inspector General's (OIG) effort to work with our staff on this unique testing program to produce the subject Draft Final Report "EPA Needs an Oversight Program for Protocol Gases" to improve the quality assurance of protocol gases. We generally agree with the data described in the draft final report. However, the issues raised apply differently to the Acid Rain Program's (ARP's) source emissions monitoring systems and National Ambient Air Quality measurement network systems. Therefore, this memorandum is structured in two parts - an Acid Rain Program section and an Ambient Air Program section. Acid Rain Program Our major comments are presented below. We include both major and detailed comments in the attached Word document in Track Changes. After you've reviewed our comments, we'd like to have another meeting with you as soon as possible. • At the end of the first paragraph in "At a Glance", please add: "However, 96 percent were within 3 percent of the true value; and there was no bias associated with the 28 gas components that did not meet the ARP performance specification. Based on the analysis results, there are no anticipated nationwide environmental consequences of the calibration gases used in the ARP. However, the ARP is addressing potential source-level problems in two ways: (1) by performing an EPA-funded gas audit in 2009; and (2) by proposing a rule establishing a largely self-supported, ongoing, annual gas audit program." • Add the following paragraph to the "At a Glance" section: "Due to the nature of our sample selection and a lack of data on vendor market shares, we were unable to develop a sampling methodology to ensure that the number of samples selected from each producer represented the number of cylinders produced and sold to consumers. Consequently, 20 ------- 09-P-0235 we could not project our test results to the universe of EPA Protocol Gases, or estimate the potential impact that gases not meeting the Acid Rain Program's accuracy criteria could have on that program. Despite this limitation, our tests represent the largest one-time assessment of the Protocol Gas industry. The final results of this study identified some potential areas of concern with respect to the quality of less expensive gases certified using GMIS." • Add the following paragraph to the "Noteworthy Achievements" section: "The ARP is currently conducting a third gas audit, covering all known producers of EPA Protocol gas. As in past ARP audits, the results will be released to the public. We expect results by early 2010. To provide a cost-effective, long term audit capability, OAR is also developing a proposed rule package to implement a largely self-supporting, annual gas audit program. This proposed rule is expected to be published for public comment in the Federal Register in the fall of 2009." ARP concurs with all of the OIG recommendations. However, we suggest a slight rewording for recommendation #2-1 as follows: 2-1 Implement a better oversight program to assure the quality of EPA Protocol Gases used to ensure the accuracy of continuous emissions monitoring systems for EPA's Acid Rain Program, and other stationary source air programs. Ambient Air Programs [OIG NOTE: The comments initially submitted by the Ambient Air Program on August 5, 2009 were superseded by the amended comments provided September 2, 2009, in the following memorandum.] Thank you again for the opportunity to comment on the draft final report. If you have questions regarding our comments, please contact Dawn Roddy, OAR Audit Follow-up Coordinator, at 202-564-1228. 21 ------- 09-P-0235 September 2, 200915 MEMORANDUM SUBJECT: Amended Comments on Draft IG Report: "EPA Needs an Oversight Program for Protocol Gases" Assignment Number 2007-000877 FROM: Gina McCarthy /s/ Assistant Administrator for Air and Radiation TO: Wade T. Najjum Assistant Inspector General for Program Evaluation The Office of Air and Radiation (OAR) appreciates the Office of Inspector General's (OIG) effort to work with our staff on this unique testing program to verify the quality assurance of protocol gases as published in the subject Draft Final Report "EPA Needs an Oversight Program for Protocol Gases". In a follow-up meeting between our staffs, your office explained that recommendation 2-2 was directed toward the program under development and outlined by the Office of Air Quality Planning and Standards (OAQPS) during the audit. Consequently, we agree with recommendation 2-2 and provide the following amended paragraph titled "Ambient Air Program", along with new recommendations to amend our August 5, 2009 response concerning the ambient air program's protocol gases. Ambient Air Program The report demonstrates that the gases tested met the expectations of the EPA Traceability Protocol with 95 percent confidence limit. We point out that the gases used for the ambient air program rely mainly on the traceability protocol. Thus, there are no findings demonstrating a concern with using protocol gases for that program. Determination of compliance with an ambient air standard is completed with consideration of many factors including the quality of the data. We believe updating the traceability protocol is the most efficient and effective corrective action to improve the quality of gases and subsequently the data collected for the ambient air program. Revising the protocol will also benefit stationary source emissions measurements. We, therefore, agree with recommendation 2-2 and have already started creation of an appropriate oversight program to reasonably assure the quality of protocol gases for the ambient air monitoring program. We amend our August 5, 2009 response: 1. Replace the sentence in the "At A Glance" section that says, "However, as of April 2009, EPA had not implemented a verification program, primarily due to indecision over how to fund the program." with "However, as of April 2009, OAQPS, with commitments by EPA Regions 2 and 7 for analytical support, had begun development of a program to provide independent verification of EPA Protocol Gas cylinders used for continuous ambient air monitors." ' The OAR memorandum was undated but was received by OIG on September 2, 2009. 22 ------- 09-P-0235 2. Add the following paragraph to the "Noteworthy Achievements" section: "OAQPS has planned and begun development of the Protocol Gas Verification Program to independently verify "EPA Protocol Gases" supplied to state, local and Tribal agencies for calibration of continuous, ambient gaseous pollutant monitors. EPA Regions 2 and 7 have committed to providing analytical support. The program is expected to be launched in March 2010. Finally, we suggest the following rewording for recommendation 2-2: "Implement an oversight program, which reasonably assures the quality of EPA Protocol Gases used to calibrate continuous ambient air monitors for the National Ambient Air Quality Standards monitoring program as specified in 40 CFR part 58 Appendix A." Thank you again for the opportunity to comment on the draft final report. If you have questions regarding our comments, please contact Dawn Roddy, OAR Audit Follow-up Coordinator, at 202-564-1228. 23 ------- 09-P-0235 Appendix D ORD's Response to OIG Draft Report July 30, 2009 MEMORANDUM SUBJECT: ORD Response to OIG's Draft Evaluation Report: EPA Needs an Oversight Program for Protocol Gases (Assignment No. 2007-00877) FROM: Lek G. Kadeli /s/ Acting Assistant Administrator (8101R) TO: Wade T. Najjum Assistant Inspector General for Program Evaluation (2460T) Thank you for the opportunity to provide comments on the draft Office of the Inspector General (OIG) report entitled "EPA Needs an Oversight Program for Protocol Gases." Attached are the Office of Research and Development's (ORD) specific comments on the draft report. In general, ORD concurs with the recommendation that ORD should update and maintain the EPA Traceability Protocol for Gaseous Calibration Standards. We appreciate your consideration of our comments. If there are any questions please do not hesitate to call me at 202- 564-6620 or Jorge Rangel at 202-564-1606. Attachment 24 ------- 09-P-0235 Office of Research and Development Comments on Office of Inspector General Draft Audit Report, "EPA Needs an Oversight Program for Protocol Gases", Assignment No. 2007- 000877, July 8, 2009 This document is comprised of four sections: 1. General comments regarding the OIG recommendations 2. Table of ORD Response to OIG recommendations 3. Proposed detailed schedule of corrective actions 4. Specific comments by page number 1. General Comments regarding the OIG Recommendations Thank you for the opportunity to comment on the draft Office of the Inspector General (OIG) report entitled "EPA Needs an Oversight Program for Protocol Gases." EPA's Office of Research and Development (ORD) concurs with the recommendation that ORD should update and maintain the EPA Traceability Protocol for Gaseous Calibration Standards. We believe that the scientific expertise within ORD puts it in a unique position to understand the gas metrological issues that need to be addressed in the update and to prepare the update. 2. Table of ORD Response to OIG Recommendations Rec. No. 2-3 OIG Recommendation Update and maintain the EPA Traceability Protocol to meet the defined objectives of the Acid Rain, NAAQS, and other stationary source air programs. Action Official Assistant Administrator for Research and Development ORD Corrective Actions Update and publish (as guidance) a revised protocol to reduce the percentage of EPA Protocol Gases that do not attain the CAMD accuracy criteria Planned Completion Date 10/01/2010 3. Proposed Detailed Schedule of Corrective Actions Corrective Action Taken Final audit report disseminated to public List of potential protocol changes* sent to NIST and specialty gas producers for Planned Completion Date October 1, 2009 October 15, 2009 25 ------- 09-P-0235 Corrective Action Taken review with request for their comments, suggested revised text, and any concentration stability data supporting changes to allowable gas composition, concentration ranges, minimum cylinder pressures, and certification periods Receive all comments and stability data Complete internal review draft of protocol Receive EPA comments on internal draft Complete external review draft of protocol by NIST and specialty gas producers Receive external comments on revisions Revised protocol published as guidance Planned Completion Date November 15, 2009 May 15,2010 June 1,2010 July 1, 2010 August 15,2010 October 1, 2010 * In 2005, EPA sent the following list of informal change suggestions to NIST and specialty gas producers: - include a procedure specifically for the use of Fourier Transform infrared (FTIR) spectroscopy; - improve statistical procedures for uncertainty calculations and revise the Excel spreadsheet; - include mercury calibration standards in the protocol; - eliminate (or retain) gas manufacturers intermediate standards (GMISs); - lengthen the certification periods for EPA Protocol Gases; - change the cylinder pressure limitations for EPA Protocol Gases; - include a procedure for analyzing and certifying zero gases; - include gas dilution systems in the protocol; - tighten documentation requirements; - develop audit participant identification number for regulatory data reporting purposes; - require ISO 17025 accreditation for producers of environmental calibration gases; - include provision for preparation and analysis of batches of EPA Protocol Gases; - require on-site visits of specialty gas producers by EPA representatives; and - provide for technical assistance/outreach to specialty gas producers by NIST representatives; 4. Specific comments by page number [OIG NOTE: ORD also provided several technical clarifications and comments to the draft report. We made changes to the final report based on these comments, as appropriate.] 26 ------- 09-P-0235 Appendix E Distribution Office of the Administrator Assistant Administrator for Air and Radiation Acting Assistant Administrator for Research and Development Deputy Assistant Administrator for Air and Radiation Deputy Assistant Administrator for Research and Development Director, Office of Air Quality Planning and Standards, Office of Air and Radiation Director, Office of Atmospheric Programs, Office of Air and Radiation Director, National Risk Management Research Laboratory, Office of Research and Development Agency Follow-up Official (the CFO) Agency Follow-up Coordinator General Counsel Associate Administrator for Congressional and Intergovernmental Relations Associate Administrator for Public Affairs Audit Follow-up Coordinator, Office of Air and Radiation Audit Follow-up Coordinator, Office of Research and Development Acting Inspector General 27 ------- |