OFFICE OF INSPECTOR GENERAL Catalyst for Improving the Environment Evaluation Report ENERGY STAR Program Can Strengthen Controls Protecting the Integrity of the Label Report No. 2007-P-00028 August 1, 2007 ------- Report Contributors: Laurie Adams Jerri Dorsey Jeffrey Fencil Elizabeth Grossman Jeffrey Harris Abbreviations DOE EPA HVAC OAR OIG PEARL RSL Department of Energy U.S. Environmental Protection Agency Heating, Ventilating, and Air Conditioning Office of Air and Radiation Office of the Inspector General Program for the Evaluation and Analysis of Residential Lighting Retail Store Level Assessment Cover image: A compact fluorescent light bulb from the ENERGY STAR qualified list (photo courtesy EPA) ------- £ < V""" *" ? U.S. Environmental Protection Agency Office of Inspector General At a Glance 2007-P-00028 August 1, 2007 Catalyst for Improving the Environment Why We Did This Review We initiated this review to evaluate how effectively the U.S. Environmental Protection Agency (EPA) is managing the ENERGY STAR ® Product Labeling Program. We specifically sought to determine whether EPA ensures that consumer product specifications are sufficient, the extent EPA verifies that products adhere to specifications, and whether EPA adequately ensures that the ENERGY STAR label is properly used. Background The ENERGY STAR Product Labeling Program identifies and promotes energy-efficient products. EPA reported in 2006 that using ENERGY STAR products prevented estimated greenhouse gas emissions equivalent to those from 23 million vehicles, and saving Americans an estimated $12 billion in their utility bills. For further information, contact our Office of Congressional and Public Liaison at (202) 566-2391. To view the full report, click on the following link: www.epa.aov/oia/reports/2007/ 20070801 -2007-P-00028.pdf ENERGY STAR Program Can Strengthen Controls Protecting the Integrity of the Label What We Found To ensure the efficiency and effectiveness of the ENERGY STAR program and the integrity of its label, EPA established several processes. These processes include product specification setting and revision, product self-certification, product verification testing, and label utilization monitoring. We reviewed these processes and found improvements could be made that could better assure the integrity of the ENERGY STAR label for the consumer of home and office products. The criteria for revising specifications were unclear and not documented. It was not evident when or what factors would trigger a specification revision. Furthermore, EPA does not have reasonable assurance that the self-certification process is effective. EPA relies on some alternative verification mechanisms, but lacks any quality assurance or review of these reported results. The Agency's verification testing also lacks a clear documented methodology governing products selected for verification tests and does not test for statistically valid results. Consequently, product efficiency and energy savings reported by manufacturers are, for the most part, unverified by EPA review. We found little oversight in using the ENERGY STAR label in retail stores, which is commonly the purchase point for the consumer. EPA could not provide documentation related to followup actions taken, final results for all retail store assessments, or the resolution status of label inconsistencies. We also found that manufacturers may label and sell products as ENERGY STAR qualified prior to submitting test results to the Agency. Using the label on products that do not meet ENERGY STAR requirements may weaken the value of the label and negatively impact the ENERGY STAR program. What We Recommend EPA should strengthen management controls to protect the integrity of the ENERGY STAR label. EPA should clarify and document the criteria for product specification revisions. EPA should establish a quality assurance program for its verification program. Also, EPA should improve its oversight in using the ENERGY STAR label by establishing a systematic methodology and procedures for monitoring, resolving, and following up on label misuse. EPA disagreed with many of our conclusions, but stated it had implemented many of the recommendations. However, EPA's planned actions do not address the intent of our recommendations, and we consider the issues unresolved. ------- jriCOSS UNITED STATES ENVIRONMENTAL PROTECTION AGENCY WASHINGTON, D.C. 20460 OFFICE OF INSPECTOR GENERAL August 1, 2007 MEMORANDUM SUBJECT: FROM: ENERGY STAR Program Can Strengthen Controls Protecting the Integrity of the Label Report No. 2007-P-00028 UM Wade T. Najjum a4 Assistant Inspector General for Program Evaluation TO: Robert J. Meyers Principal Deputy Assistant Administrator for Air and Radiation This is our report on the subject evaluation conducted by the Office of Inspector General (OIG) of the U.S. Environmental Protection Agency (EPA). This report contains findings that describe the problems the OIG has identified and corrective actions the OIG recommends. This report represents the opinion of the OIG and does not necessarily represent the final EPA position. Final determinations on matters in this report will be made by EPA managers in accordance with established resolution procedures. The estimated cost of this report - calculated by multiplying the project's staff days by the applicable daily full cost billing rates in effect at the time - is $338,079. Action Required In accordance with EPA Manual 2750, you are required to provide a written response to the report within 90 calendar days. You should include a corrective action plan for agreed-upon actions, including milestone dates. We have no objection to the further release of this report to the public. This report will be available at http://www.epa.gov/oig. If you or your staff have any questions regarding this report, please contact me at (202) 566-0827 or naiium.wade@epa.gov; or Jeffrey Harris, Director of Special Studies, at (202) 566-0831 or harris.ieffrev@epa.gov. ------- ENERGY STAR Program Can Strengthen Controls Protecting the Integrity of the Label Table of Contents Chapters 1 Introduction 1 Purpose 1 Background 1 Noteworthy Program Accomplishments 2 Scope and Methodology 3 Prior Audit Coverage 4 2 EPA Processes to Establish and Protect the Integrity of the ENERGY STAR Label 5 Product Specification Process 5 Product Self-Certification Process 6 Product Verification Testing Process 7 Product Label Utilization Monitoring Process 8 3 Transparency Needed in Specification Revision Decisions 9 Product Specifications Set to Identify Top Performers 9 Product Specifications Revision Criteria Not Clear 10 Conclusion 11 OIG Recommendation 11 Agency Comments and OIG Evaluation 11 4 Improving Product Verification 12 ENERGY STAR Verification Testing Conducted on a Minimal Basis 12 Selection Method Inconsistently Applied 14 Cost of ENERGY STAR Verification Testing 15 Quality Assurance Plan Needs Improvement 16 Conclusion 17 OIG Recommendations 17 Agency Comments and OIG Evaluation 17 5 Tracking and Monitoring the Use of the ENERGY STAR Label 19 Oversight for Label Use in Retail Stores 19 Reports Not Finalized and Results Not Documented 20 Results Not Monitored or Tracked 21 Products Marketed as ENERGY STAR Prior to EPA Notification 22 Conclusion 22 OIG Recommendations 23 Agency Comments and OIG Evaluation 23 Status of Recommendations and Potential Monetary Benefits 24 ------- ENERGY STAR Program Can Strengthen Controls Protecting the Integrity of the Label Appendices A ENERGY STAR Qualified Product Categories 25 B Detailed Scope and Methodology 26 C OIG's Detailed Analysis of Agency's Response 28 D Agency Response 37 E Distribution 44 ------- Chapter 1 Introduction Purpose As part of an EPA Office of Inspector General (OIG) evaluation agenda to assess the Agency's new approach to environmental protection, we initiated this review to evaluate how effectively EPA is managing the ENERGY STAR Product Labeling Program. Specifically we sought to determine to what extent does EPA: 1. Ensure that consumer product specifications to obtain and maintain the ENERGY STAR logo are sufficient and up to date? 2. Verify that products adhere to the ENERGY STAR specifications? 3. Ensure that the ENERGY STAR label/logo is properly used? Background Historically, EPA has relied upon regulations to achieve environmental results and risk reduction. However, in the early 1990s, Section 103(g) of the Clean Air Act directed the Administrator to "conduct a basic engineering research and technology program to develop, evaluate, and demonstrate non-regulatory strategies and technologies for reducing air pollution." In 1992, EPA's Office of Air and Radiation (OAR) established the ENERGY STAR Product Labeling Program (the ENERGY STAR Program) as an innovative, effective, and efficient approach to environmental protection. The ENERGY STAR Program was subsequently authorized in the Energy Policy Act of 2005. ENERGY STAR is a voluntary program to help businesses and individuals protect the environment through superior energy efficiency. The ENERGY STAR Program was designed to overcome selected market barriers towards adopting cost-effective energy efficient products and services. The program was first introduced to recognize and promote energy-efficient computers. It has since grown to cover many additional consumer products and services within both the residential and commercial settings.1 In 1996, EPA partnered with the Department of Energy (DOE) to promote the ENERGY STAR label and broaden the range of activities covered.2 1 ENERGY STAR is composed of three major components: Products, Residential, and Commercial. 2 A Memorandum of Cooperation was signed jointly on May 29, 1996. The Memorandum described each Agency's responsibilities as it relates to using and overseeing the ENERGY STAR logo. 1 ------- EPA's ENERGY STAR budget for Fiscal Year 2006 was approximately $50 million, of which $38 million was allocated for ENERGY STAR contractor support. The remaining $12 million was allocated for total staff equal to 77.4 full time equivalents. DOE's 2006 ENERGY STAR budget was approximately $6 million. Noteworthy Program Accomplishments EPA launched the ENERGY STAR Program to "realize significant reductions in emissions and energy consumption by permanently transforming markets for energy consuming products." Since ENERGY STAR began, the program has grown steadily in terms of the energy efficient solutions it offers, the variety of partners, and the benefits it delivers. As of October 2006, the ENERGY STAR Program had 48 products qualified (see Appendix A for a detailed listing). The ENERGY STAR Program enables consumers to identify the products, practices, services, homes, and buildings that offer potential energy savings. ENERGY STAR has overcome informational, institutional, and practical obstacles to greater investment in energy efficient technologies and practices. EPA reported3 that in 2005 ENERGY STAR prevented an estimated 34 million metric tons of greenhouse gas emissions, or the equivalent of annual emissions from 23 million vehicles. Furthermore, EPA reported more than an estimated $12 billion in savings were shown in utility bills of the American public. EPA has been successful in marketing the ENERGY STAR brand/label. In 2005, EPA reported that 60 percent of households nationwide recognize the ENERGY STAR label and 70 percent correctly interpret the meaning of the ENERGY STAR label. ENERGY STAR The ENERGY STAR Program has also grown internationally. EPA has engaged with government agencies in a number of countries to promote certain ENERGY STAR products. International partners include Australia, Canada, the European Union, Japan, New Zealand, and Taiwan. 3 ENERGY STAR and Other Climate Protection Partnerships 2005 Annual Report, EPA 430-R-06-014, October 2006. 2 ------- Scope and Methodology This review did not attempt to evaluate the overall effectiveness and impact of the ENERGY STAR program. This evaluation focused on the ENERGY STAR Product Labeling Program procedures designed to protect the logo and lead to benefits. The Product Labeling Program is responsible for overseeing an estimated two billion ENERGY STAR products sold since 1992.4 Figure 1.1 summarizes the products sold. Figure 1.1: Types of ENERGY STAR Products by Sales Since 19925 Source: ENERGY STAR and Other Climate Protection Partnerships 2005 Annual Report In contrast, other ENERGY STAR divisions are responsible for overseeing 520,000 ENERGY STAR-qualified homes and 2,500 ENERGY STAR-labeled commercial buildings.6 To evaluate how effectively EPA is managing the ENERGY STAR Product Labeling Program, we reviewed Agency guidance documents, and met with external partners and ENERGY STAR staff. To address the evaluation objectives we specifically analyzed: • The product specification setting and revision process, including reviewing all applicable guidance documents. • The self-certification and verification processes, including reviewing all applicable guidance documents. • EPA's efforts to monitor how well the ENERGY STAR label was used. 4 This total includes both EPA and DOE ENERGY STAR products sold. 5 The Office Equipment portion of products sold excludes computers and monitors. 6 ENERGY STAR and Other Climate Protection Partnerships 2005 Annual Report, EPA 430-R-06-014, October 2006. COMPUTERS AND MONIT~°° 31% OTHER 6% LIGHTING t 22% OFFICE EQUIPMENT 18% CONSUMER ELECTRONICS 23% 3 ------- For a more detailed discussion of our analysis, see Appendix B. Furthermore, we examined management and internal controls as they related to our objectives. In conducting our review, we applied criteria from the Government Performance and Results Act and EPA program management guidance. We performed our evaluation in accordance with Government Auditing Standards, issued by the Comptroller General of the United States. We performed our field work from September 2006 through December 2006. Prior Audit Coverage According to the 1997 OIG report entitled, Risk Reduction Through Voluntary Programs, 1 the ENERGY STAR program used good management practices and developed ways to estimate environmental results. The report noted that while the ENERGY STAR program had a good planning process, it could improve. For example, according to the report, EPA's practice of allowing manufacturers to self-certify their products for ENERGY STAR compliance might not preserve the integrity of the label. As a result, as consumer demand for ENERGY STAR product increased, manufacturers would be pressured to provide these products. One result would be increased potential for misusing the ENERGY STAR label. The OIG made several recommendations to the ENERGY STAR program including one specific to the scope of this evaluation: that the Principal Deputy Assistant Administrator for Air and Radiation "consider the need for additional efforts to maintain ENERGY STAR logo integrity, as the program applies the logo to new products." The Principal Deputy Assistant Administrator for Air and Radiation agreed with all the findings in the report and proposed corrective actions to address the recommendations. The Agency agreed to evaluate whether spot-testing products with the ENERGY STAR label was needed to maintain label integrity. Since 1997, the Agency began considering monitoring using the ENERGY STAR label. In 2002, EPA started a compliance testing initiative. In conducting our evaluation, we reviewed these efforts and report further on the importance of EPA taking appropriate efforts to maintain the integrity of the ENERGY STAR label. 7 EPA Office of Inspector General; Northern Audit Division, Chicago, Illinois. Report #7100130, Risk Reduction Through Voluntary Programs. 1997, http://www.epa.gov/oigeartli/reports/1997/voltable.htm. 4 ------- Chapter 2 EPA Processes to Establish and Protect the Integrity of the ENERGY STAR Label To ensure the efficiency and effectiveness of the ENERGY STAR program and the integrity of its label, EPA has established several processes, including product specification, product self-certification, product verification testing, and product label utilization monitoring. Specifically, • EPA identifies consumer products that have the potential for improvement and increased energy savings. • EPA develops individual product specifications with projected energy savings. • Manufacturers self-test their models and label them as ENERGY STAR if they meet established Agency guidelines. • EPA conducts compliance tests to ensure select products meet specifications. • EPA monitors ENERGY STAR label usage by advertisers and retailers. Product Specification Process The Agency follows six key principles when establishing consumer product energy efficient specifications. They are as follows. 1) Significant energy savings can be realized on a national basis. 2) Product performance can be maintained or enhanced with increased energy efficiency. 3) Purchasers will recover their investment in increased energy efficiency within a reasonable time. 4) Energy efficiency can be achieved with several technology options, at least one of which is non-proprietary. 5) Product energy consumption and performance can be measured and verified with testing. 6) Labeling would effectively differentiate products and be visible for purchasers. Typically, the specification is set to recognize the top 25 percent of energy performing models on the market. The process of setting a product specification is illustrated in Figure 2.1. Prior to making the decision to develop a new product specification, EPA gathers market information to determine if a specification is warranted and viable. 5 ------- Figure 2.1: Product Specification Development Cycle Specification Development Cycle Test Methodology Development (it necMsaryl Market, Industry & Design Research Release Draft Specification Energy & Environmental Analysis Stakeholde Meetings Rpst DuflK and Stukeholdt ColWWiWitS t:l Wei) Site Monitor Market Penetration Officially Launch Specification with Industry artel ftteliolders Finalize? Specification MannVicturersP^B Join Program as Paftiw ri Specification and Begin Takes labeling Effect Products Final Decision Memorandum Stockholder Notification Open Specification for Revisions with Intel Decision Men Source: EPA ENERGY STAR Website According to the Agency, EPA sets an initial specification for a product category and then monitors the product in the marketplace to determine when it is appropriate to begin revising each specification. Annually, the ENERGY STAR staff meet to determine what specification revisions are needed. The decision to revise a specification depends on a number of factors. These include the percentage of ENERGY STAR products sold and available technologies for each ENERGY STAR product. Before EPA sets or revises a product specification, it requests input from manufacturers and stakeholders. This input is used to determine availability of new or advanced technologies, interest in producing products under a new revised specification level, and provide notice to the manufacturer of a pending revision. According to ENERGY STAR staff, this collaboration is an essential part of the process. We analyze this process in Chapter 3. Product Self-Certification Process For manufacturers to participate in the ENERGY STAR program, they must enter into an ENERGY STAR Partnership Agreement. After entering into this agreement the manufacturers consent to: 6 ------- • Test their products according to established specifications and guidelines. • Provide a list (or notification) to EPA of the qualified products, certifying that these products are ENERGY STAR qualified and meet the ENERGY STAR performance specifications. • Provide certification summary data as required per individual product specifications. • Provide EPA with annual shipping data for all ENERGY STAR qualified products. • Agree to put ENERGY STAR labels on qualified products only and to include ENERGY STAR literature. • Agree to follow all rules pertaining to the application and use of ENERGY STAR labels/logos. Once a company signs this agreement it is allowed to self-certify that its products meet the ENERGY STAR specifications, and it may immediately begin to utilize the EPA ENERGY STAR label. Self-certification involves the manufacturer testing its product models per ENERGY STAR specifications and reporting the results. The requirements for the amount and specifics of the reported data results vary by product and are according to specification requirements. For example, for some products a simple one page qualified product information form, with limited testing information, is the only required submittal. For other products, the amount of testing information required to be submitted is greater in both detail and volume. We analyze this process in Chapters 4. Product Verification Testing Process In 2002, EPA started the ENERGY STAR Compliance Testing Initiative to support efforts to ensure that the program delivers on all of its energy-efficiency promises. According to the Agency, the objective of compliance testing is to test for the presence of issues within various product categories. If issues are identified, further action would be warranted. Key components of this testing initiative include the following: • Testing is completed by an independent lab(s) which obtain units directly from the marketplace for testing. • The goal of testing is to "identify potential compliance problems and set in motion a review process to ensure manufacturers take corrective measures as appropriate." • EPA generally targets the most popular product models on the market for testing. These are identified based on unit sales, with consideration also given to a range of product features, prices, and manufacturers. 7 ------- • Partners (manufacturers) whose units are selected for testing are notified prior to testing in writing. Also, they are notified of completed testing results. • For failed testing results, the partner has 30 days to submit additional information in support of the original certification. • If a partner fails to adequately respond with 30 days, the test data still do not meet ENERGY STAR requirements, or if the partner acknowledges that the product is mislabeled, only then will EPA remove the model in question from the ENERGY STAR qualified product list. EPA allows 6 months for the existing labeled ENERGY STAR products to be sold. At the end of the 6 months, retailers and wholesalers are required to cover the ENERGY STAR label on any unqualified product they still wish to sell. We analyze this process in Chapter 4. Product Label Utilization Monitoring Process According to the Agency, once the companies certify their product, they can begin marketing the product using the ENERGY STAR label. ENERGY STAR is a registered trademark of the EPA and thus is protected by Federal trademark law. In order to maintain the full protection of the trademark and other applicable laws, EPA works to monitor and stop unauthorized or confusing use of the ENERGY STAR label. To ensure the ENERGY STAR label is used appropriately in the marketplace, EPA undertakes the following activities to track the use of the ENERGY STAR name, label, and to help maintain the value of the ENERGY STAR program: • Monthly print advertisement and article monitoring, • Quarterly Internet monitoring, and • Retail store level assessments (RSL). According to Agency guidance, once a possible label infringement is identified, EPA is to take appropriate follow up action. The type of action taken depends on the nature of the infringement and the type of company involved. Examples of routine follow up actions include contacting the company by email, letter, or phone call; and in extremely rare cases terminating the partnership. We analyze this process in Chapter 5. 8 ------- Chapter 3 Transparency Needed in Specification Revision Decisions We found that the criteria for initiating and revising specifications are unclear and are not documented. Agency documents state that specifications are initially designed to capture approximately 25 percent of a product market. However, we found the Agency's guidance documents were not clear as to what factors would trigger a specification revision. Consumer confidence that ENERGY STAR products represent the most cost-effective technologies available may be undermined if this process is not transparent. Product Specifications Set to Identify Top Performers According to Agency documents, the ENERGY STAR label is intended to identify the top performers in energy efficiency. The Agency's 2001 Product Labeling Business Plan stated: ENERGY STAR specifications are designed to capture approximately 25% of a product market. When significantly more of the market qualifies for the ENERGY STAR label, it ceases to uphold its purpose of differentiating more efficient products. Consequently, the integrity of the label is diminished and credibility with consumers may decline. At this point, it is necessary to "ratchet down " or design a more stringent specification. EPA's 25 percent goal results in an "exclusive" program, meaning most products available in the market place are not ENERGY STAR certified. According to EPA, by recognizing the top 25 percent, ENERGY STAR products are distinguished from other products, thereby adding to their intrinsic value. However, EPA still wanted to act as a catalyst and motivate the bottom 75 percent to increase energy performance. According to the Agency, the record of program participation over the last 12 years has demonstrated that industry has risen to the original challenge and improved the efficiency of their products. This results in the majority of product models on the market labeled as ENERGY STAR. However, if the majority of certain products are labeled as ENERGY STAR, hence becoming a more "inclusive program," the integrity of the label is diminished and credibility with consumers may decline. Further, with a goal of 25 percent of the available products being ENERGY STAR certified, three-quarters of the products available for purchase are not resulting in maximum energy efficiencies that could contribute to EPA 9 ------- greenhouse gas reduction goals. Therefore one option for achieving EPA's ENERGY STAR goal is to reconsider the desirable level of exclusivity. Additionally, a trade-off exists between a program that seeks to maximize the purchase (or at least availability) of energy efficient products, versus one that seeks to continually advance the standards for energy efficiency and promotes the advancement of energy-saving technology. Product Specifications Revision Criteria Not Clear As part of the ENERGY STAR specification setting process, EPA identifies the top performers in a given product category. Once an ENERGY STAR specification is set, manufacturers are then motivated by consumer demand for ENERGY STAR qualified products. According to Agency documents, once an ENERGY STAR specification is in place for some amount of time, the probability that market conditions and the available model mix have changed increases. Once the number of ENERGY STAR models reaches a high percentage of availability, the label begins to lose its intended purpose of identifying the top energy efficient products. According to Energy Star staff, the specification revision process begins with the ENERGY STAR staff meeting annually and deciding which specifications will be revised based on the guiding principles. However, we found the Agency does not document the results of this process, making it unclear how individual decisions were made. To ensure participants are fully aware of what the program delivers, all stakeholders must understand what triggers a specification revision. Also, the results of the annual process need to be clear and transparent. We reviewed EPA ENERGY STAR product specification revisions and market shares and found that specifications appear to be revised on a case-by-case basis. As discussed previously, according to EPA staff, a number of factors influence the decision to subsequently revise a specification. However, we found that the Agency's current draft guidance document8 is not clear regarding what triggers a specification revision. According to the Agency, while a market share in excess of 50 percent may trigger a review of the product specification for possible revision, EPA may consider several other factors as well. It uses the six guiding principles (discussed in Chapter 2) as guidelines in determining when a specification revision is necessary. However, these principles are not clearly defined and the draft guidance document does not describe how the principles will be used in the revision. For example, one of the six principles is to ensure that ENERGY STAR labeling would effectively differentiate products. Typically, the guidance states that the specification is set to recognize the top 25 percent of energy performing models 8 During our fieldwork, the Agency provided a draft document entitled Maintaining the Value of ENERGY STAR. This document summarizes the components and processes in place to protect the integrity of the ENERGY STAR label. This document has not been formally published or issued in final. 10 ------- on the market. However, we found that the Agency does not track the ratio of ENERGY STAR models to non-ENERGY STAR models available on the market, which would allow for determining this percentage. Furthermore, EPA considers market share defined as the number of ENERGY STAR products sold. While this interpretation of market share is not clearly identified as a guiding principle, tracking such information may be relevant as an indicator of success for the ENERGY STAR outreach component. However, in order to ensure the ENERGY STAR label identifies the top performers in a product category, EPA needs to also monitor information related to the percentage of ENERGY STAR products available. Conclusion One of the key principles of the ENERGY STAR Product Labeling Program is that the ENERGY STAR label would enable the consumer to easily identify the most energy-efficient products. However, the timing and inconsistent criteria of past revisions means that the ENERGY STAR label may not deliver what is promised by the program, to market the most energy efficient products. In an effort to increase transparency in the revision process, EPA should ensure clear and consistent criteria are used and documented. OIG Recommendation To ensure the integrity of the ENERGY STAR label for qualified EPA products, we recommend that the Principal Deputy Assistant Administrator for the Office of Air and Radiation: 3-1. Clarify the decision criteria and document the process for revising an ENERGY STAR specification, including identifying circumstances when a specification revision would not be revised, despite a high market share of qualified products. Agency Comments and OIG Evaluation EPA agreed that clarification and documentation was needed for the specification revision process. EPA provided a document that describes the guiding principles for product specification revision. However, EPA's actions do not meet the intent of the recommendation. The document simply restates the original principles (though deleting reference to market share as a guiding principle). Therefore, we consider the recommendation open and unresolved. For the OIG's detailed analysis and EPA's full response, see Appendices C and D. The OIG has incorporated technical corrections and clarifications requested by EPA as appropriate. 11 ------- Chapter 4 Improving Product Verification Although the self-certification process is a fundamental component of the ENERGY STAR program, we found that EPA is not reasonably assured that this process is effective. The Agency in part relies on both Federal standards and third-party testing, but lacks any quality assurance or review of these results. The Agency's verification testing also lacks a clear documented methodology governing products selected for verification tests and does not strive for statistically valid results. Consequently, product efficiency and energy savings are, for the most part, unverified by EPA review. ENERGY STAR Verification Testing Conducted on a Minimal Basis We found that the initial verification test was not conducted until 2002, or 10 years after the ENERGY STAR program began. From 1992 until 2002, EPA did not validate self-certifications because the ENERGY STAR staff considered that effort to be too costly and time-consuming. Beginning in 2002, EPA began limited verification testing on select products and models. Presently, EPA still relies overwhelmingly on the validity of the manufacturers' self-certifications. To date, EPA has completed verification testing on nine ENERGY STAR products. Ten tests have been completed, with one product being tested twice. Even though computers were the initial product qualified as ENERGY STAR in 1992, they were not tested until 2006. Furthermore, of the 48 ENERGY STAR product categories managed by EPA, they exclude 14 categories from the scope of their verification testing. According to EPA, due to alternative verification mechanisms such as the Federal energy efficiency standards, third-party certification programs, and the Program for the Evaluation and Analysis of Residential Lighting (PEARL,) the Agency does not believe independent verification testing is needed for certain qualified products. See Table 4.1 for EPA's determination of which product categories are excluded from the scope of its verification testing universe. 12 ------- Table 4.1: Exclusions from EPA ENERGY STAR Testing Universe EPA Qualified ENERGY STAR Product Third-party Certification Federal Standards PEARL Air-Source Heat Pumps X X Boilers X X Central Air Condition X X Dehumidifiers X Exit Signs X Furnaces X X Geothermal Heat Pumps X Light Commercial Heat Pumps X Light Commercial HVAC X Residential Light Products X Roof Products X Room Air Cleaners X Traffic Signals X Ventilating Fans X Source: OIG Analysis of EPA data EPA excludes 11 ENERGY STAR qualified products from the scope of its testing universe because these products are eligible for testing under various third-party certification programs. These programs typically are developed and maintained by industry associations as a service to their members. Participation with these programs is voluntary on the part of the manufacturer. In addition, EPA excludes six ENERGY STAR qualified products from the verification testing universe scope because they fall under product categories subject to Federal energy efficiency standards. According to EPA, these standards are subject to Federal oversight and noncompliance penalties. Finally, EPA excludes residential lighting products from the verification testing universe scope because they fall under the PEARL program. PEARL was created in 1999 in response to complaints received about the performance of certain ENERGY STAR qualified lighting products. PEARL purchases and tests lighting products that are available to the consumer in the market place. While EPA relies on these alternative verification mechanisms, it did not provide us with documentation of receipt or review of test results from products excluded due to these mechanisms. The information EPA provided does not indicate whether EPA receives or monitors results of these activities. Therefore, EPA lacks both assurance of the results of these other testing mechanisms and data related to which manufacturers or actual qualified product models have voluntarily participated. In addition to EPA's lack of review of these test results, all of the third-party certification programs are voluntary, thus manufacturer 13 ------- participation is not a requirement. Therefore, without an independent quality assurance plan in place, EPA is narrowing the scope of its manufacturer data verification testing without any assurance that it is justified in doing so. This practice further threatens the integrity of and consumer confidence in the ENERGY STAR label. Selection Method Inconsistently Applied EPA's verification testing lacks a clear documented methodology governing products selected for verification tests and a clear sampling protocol. After analyzing the ENERGY STAR selection methodology used for verification testing, we found that EPA uses different criteria when selecting which product models to test. According to Agency guidance and ENERGY STAR staff, products are targeted and selected for testing based on the "most popular products on the market." However, upon analyzing the selection method for products selected for testing, we found EPA has been inconsistent with its selection method. (See Table 4.2 for details on products tested.) Table 4.2: ENERGY STAR Verification Testing Year Tested Product Tested Model Selection Method Mfgs Tested Models Tested Units Tested 2002 Televisions Top Market sales 5 15 45 2002 DVDs Top Market sales 8 15 45 2003 Monitors Top Market sales 7 15 45 2003 Telephony9 Top Market sales 6 20 50 2004 Scanners Top Market sales 7 15 45 2004 Multifunction devices & Upgradeable digital copier EPA's Product Development Team input 11 11 11 2004 Printers & Faxes Top Market sales 10 14 14 2005 Dehumidifiers EPA's Product Development Team input 12 20 20 2006 Computers EPA's Product Development Team input 10 16 16 2006 Monitors Top Market sales 11 20 20 Totals 87 161 311 Source: OIG Analysis of EPA data 9 Analog and digital cordless telephones, multi-handset cordless telephones, answering machines, combination cordless telephones/answering machines, multi-handset combination cordless telephones/answering machines, and additional handsets using a variety of frequency ranges. The product round included the testing of three each of cordless models sampled and one each of handset models. 14 ------- For example, we found that model selection for three of the nine product categories tested were chosen based on input from ENERGY STAR staff rather than using the stated criteria of top market sales used for the other seven products tested. We spoke with the ENERGY STAR staff about these inconsistencies and were told that sometimes they make selection decisions based on other factors. For example, if they were testing computers, more laptops might need to be tested than desktops due to an upcoming specification revision. The inconsistent or subjective application of the selection criteria decreases transparency. While we found no evidence of such, this practice may lend itself to the perception of bias and pre-selection. The lack of a documented process and criteria and inconsistency in implementation leaves the program vulnerable to perceptions of bias or pre-selection. Furthermore, it may devalue the confidence in the overall results which is important since testing is already done on such a limited basis. The ENERGY STAR verification testing does not attempt to produce statistically significant results in its testing program. Moreover, EPA can not conclude that its test results are representative of the performance of the product population it has sampled. In 5 of the 10 product categories subjected to verification testing to date, only one unit of each model was tested. (See Figure 4.2 for details on units and models tested.) When such a small number is tested within a sampled model, the results cannot account for inter-item variability. Manufacturers also agree to test as little as one unit to certify that their products meet ENERGY STAR qualifications. These testing practices can limit consumer confidence that expected energy savings will be realized. Cost of ENERGY STAR Verification Testing The integrity of the ENERGY STAR label is vital to the success of the program. The ENERGY STAR staff said that testing is done to "protect the ENERGY STAR brand." Additional testing may increase the integrity of the ENERGY STAR brand and reinforce consumer confidence in the program. These elements would allow for continued program success and further safeguard the public's confidence in the validity of the program. According to ENERGY STAR staff, costs associated with verification testing include the cost of testing, product purchase and handling costs, and the cost associated with using a contractor. We reviewed the costs for two products, computers and computer monitors, which were selected by ENERGY STAR staff as representative examples. The cost to complete the two verification tests was approximately $24,000 each. This represented less than half of one percent of the total ENERGY STAR budget. Furthermore, in Fiscal Year 2006, the ENERGY STAR Product Labeling Program had an estimated $18.2 million of which $70,000 was allocated for verification testing. (See Figure 4.1.) 15 ------- Figure 4.1: ENERGY STAR Product Labeling Program Budget Energy Star Labeling Program (intramural) 18.13% Product and Labeling n A-,~inistration Program Evaluation Labeling 4.01 % Technical Assistani Utilities/Prograrr Sponsors 12.64% Verificati 0.3 Outreach/ Outreach Training 35.16% New Product Assessments/Existing Specifications Review 10.99% Source: OIG Analysis of EPA Budget Information Since the beginning of verification testing, EPA has averaged annually two sets of product verification tests. Now, according to EPA, 44,000 qualified product models exist within the qualified ENERGY STAR product categories. At the end of 2006, EPA had only conducted verification testing on 160 product models in 9 product categories over 4 years. We found no official documented quality assurance plan was in place to ensure the integrity of the verification testing and the ENERGY STAR label. A Quality Assurance Plan describes the detailed quality assurance procedures and other technical activities that must be implemented to ensure that the results of work performed satisfies stated performance criteria. During our fieldwork, EPA provided a draft document enti tl ed Maintaining the Value of ENERGY STAR. This document summarizes the components and processes in place to protect the integrity of the ENERGY STAR label, including the scope of verification testing. While this is a good first step, this draft document does not constitute a formal quality assurance plan. However, as of July 2007, this document had not been issued in final or formally published. Quality Assurance Plan Needs Improvement 16 ------- ENERGY STAR staff assert that the quality and integrity of the ENERGY STAR program is further ensured through "self-policing." They believe that in a competitive marketplace it is in each manufacturer's interest to police or test its competitor's products. The ENERGY STAR staff believe that ENERGY STAR products not meeting qualification standards for the program will be reported to EPA by rivals. ENERGY STAR program officials did not produce any evidence the asserted self-policing is occurring. Conclusion EPA has limited the scope and amount of Agency-conducted verification testing even though it does not have reasonable assurance that the self-certification process is effective. The Agency relies on both Federal standards and third-party testing in limiting the scope of their testing, however it lacks any documented quality assurance of these results. EPA's verification testing also lacks a clear documented methodology governing products selected for verification tests and a clear sampling protocol. Consequently, the validity of the manufacturer's self- certification of products is, for the most part, unverified by EPA review. These practices weaken the integrity of the ENERGY STAR label and threaten the validity of self-certification. Furthermore, since the Agency has been successful in its marketing of the ENERGY STAR brand, we believe that some resources could now be more effectively used in quality assurance. OIG Recommendations To ensure the integrity of the ENERGY STAR label for qualified EPA products, we recommend that the Principal Deputy Assistant Administrator for the Office of Air and Radiation: 4-1. Establish a formal Quality Assurance Program for product and verification testing to provide a reasonable assurance results are representative of the products available and the certification of others (self- and third party) may be relied upon. 4-2. Coordinate verification testing with product specification setting and revision process to ensure products are selected in a timely and relevant basis. Agency Comments and OIG Evaluation EPA generally concurred with our recommendations. However, EPA's response to both recommendations did not meet the intent of either recommendation. For Recommendation 4-1 it is insufficient for the Agency to simply formally document its existing program. The OIG found that the Agency lacked a formal program which is needed to provide reasonable assurance that the self- certification, verification testing, and third-party testing programs are effective. 17 ------- To complete this recommendation EPA needs a well-documented formal program as prescribed in this chapter and discussed in additional detail in Appendix C. For Recommendation 4-2, the OIG agrees with the Agency's initial effort, but to be responsive to the recommendation, further details are needed. EPA's detailed response and the OIG's evaluation are in Appendices C and D. The OIG has incorporated technical corrections and clarifications from EPA's response in the final report as appropriate. 18 ------- Chapter 5 Tracking and Monitoring the Use of the ENERGY STAR Label To help protect the integrity of the ENERGY STAR program, EPA reviews the use of the ENERGY STAR label by retailers. We found that EPA does not document the follow-up actions taken on label inconsistencies identified during these reviews. ENERGY STAR staff do not receive final reports detailing the review results, conclusions, and followup actions taken. Followup detailing any corrective actions taken is essential to provide assurance that label inconsistencies are corrected and do not become repetitive. Manufacturers may label and sell products as ENERGY STAR prior to submitting test results to the Agency for up to a year. The use of the label on products that do not meet ENERGY STAR requirements may decrease the value of the label and negatively impact the ENERGY STAR program. Oversight for Label Use in Retail Stores According to EPA the Agency needs to ensure qualified products are presented properly in the retail setting in order to maintain the value of the ENERGY STAR program. To help protect the integrity of the ENERGY STAR program, EPA initiated semiannual quality assurance reviews to track and monitor using the ENERGY STAR label. EPA conducts these reviews at selected retailers within 10 geographically diverse metropolitan areas. According to EPA, due to the cost associated with such studies, this assessment does not use a statistically significant sample nor is it comprehensive. Specifically, the goal of the Retail Store Level Assessment (RSL) is to evaluate how effective is marketing and communications for the ENERGY STAR Consumer Campaign and Product Labeling Program. The RSL seeks to address four goals: • to assess retailer knowledge and how salespeople use ENERGY STAR in the retail sales process; • to check the visibility and overall presence of the ENERGY STAR label; • to assess the availability and visibility of ENERGY STAR products; and, • to assess the accuracy of ENERGY STAR product labeling. The RSL involves monitoring the accuracy of the use of the ENERGY STAR label on a sample of select ENERGY STAR qualified products. Field personnel examine the product models with and without ENERGY STAR labels within select retail store settings. This information is used to determine whether products are labeled or mislabeled as ENERGY STAR. Mislabeled products found during the RSL are identified as label inconsistencies. 19 ------- While the assessment serves multiple purposes, a key aspect includes reviewing ENERGY STAR product label usage for accuracy. EPA can use this tool to identify strengths and weaknesses of the Agency's processes. However, we found that the emphasis of the RSL appears to be on assessing the impact of the Agency's marketing and outreach efforts. Furthermore, it is important in assuring the accuracy of label use and the overall integrity of the ENERGY STAR label. Reports Not Finalized and Results Not Documented Since 2001, EPA initiated eight rounds of RSLs. These reviews were conducted in 11 unique retailers, covering 16 ENERGY STAR product categories. The assessments in part examined how accurately the ENERGY STAR label was used. However, we found that EPA has not consistently required the contractor to provide final reports documenting the RSL results. Specifically, of the seven assessments completed by the end of our fieldwork, only two reports were finalized. The remaining six reports are in draft and present preliminary results of the review. We were advised by ENERGY STAR staff that the results would need to be reviewed and updated prior to issuing a final report. The draft reports lacked an Executive Summary, Background/Methodology sections, and in many cases, Conclusions. EPA officials said they did not require that the draft reports be updated to final in many instances due to limited resources. Furthermore, EPA explained it was neither a good investment nor necessary to finalize the fieldwork draft report to reflect the final outcomes of the reconciliations and followup process. We also found these reports did not document the needed followup actions taken when the apparent inappropriate use of the ENERGY STAR label was found. According to guidance documents, improperly labeled ENERGY STAR products or label inconsistencies are referred to the appropriate program mangers for followup action. When we requested documentation of followup actions taken, EPA staff could not provide any and were unaware as to whether the contractor followed up on inconsistencies found during the RSLs. According to ENERGY STAR staff, a second contractor conducted follow up on label inconsistencies. However, the Agency did not have any documentation to support the followup actions taken by the contractor on behalf of the Agency. Furthermore, we found that EPA lacks specific guidance for the contractor regarding what followup actions should be taken for various mislabeling circumstances. After we had discussions with ENERGY STAR staff, they acknowledged the lack of guidance in this area and drafted a flowchart as a guidance document. This draft chart is a good starting point and could be used in developing potential guidance for the contractor and/or EPA staff in addressing ENERGY STAR label inconsistencies. 20 ------- Results Not Monitored or Tracked We analyzed all completed RSL rounds and found inconsistencies in the monitoring and tracking label inconsistencies. For example, the RSL conducted in April 2003 found that 43 percent of nonqualified television models appear to either always or sometimes be labeled as ENERGY STAR. In the assessment conducted in October 2003, the percentage of nonqualified television models mislabeled as ENERGY STAR grew to 44 percent. The primary explanations given for products mislabeled as ENERGY STAR was that they previously qualified but had been removed from the current ENERGY STAR qualified list due to changes in standards. However, of the inconsistencies found, 18 percent of the televisions never met the ENERGY STAR standards and were caused by the manufacturer mislabeling the products. At our request, the contractor provided updated tables for the RSL conducted in May 2006. The final results of this assessment indicated that the percentage of mislabeling had decreased for televisions. The percentage of nonqualified television models mislabeled dropped to 21 percent. Of the models reviewed, 48 models were always or sometimes mislabeled as ENERGY STAR. We requested all information related to followup actions taken on the inconsistencies identified during this RSL. According to the ENERGY STAR staff, they only have the contractor follow up on products that are considered always labeled incorrectly. We were provided a narrative that described the nature of the 21 products that were considered always mislabeled as ENERGY STAR. (See Table 5.1.) The remaining 27 products were sometimes labeled as ENERGY STAR and were considered to be label errors by the retailer or by regional program labeling practices. Table 5.1: Explanation for Televisions Found to Always Be Mislabeled Television Models Always Labeled Incorrectly as ENERGY STAR Are currently being reviewed 2 Meet ENERGY STAR standards but had not completed the qualification process 17 Never qualified 2 Source: OIG Analysis of RSL data Furthermore, we found EPA does not track results and conduct analysis to identify program strengths and weaknesses. For example, the Agency does not identify and track repeat offenders in successive assessments. Based on assessments conducted to date, some product categories such as lighting fixtures seem to consistently have a high percentage of mislabeling problems. In contrast some product models that are certified as ENERGY STAR are found to not be consistently labeled as ENERGY STAR. Since the retail store is generally the purchase point for the consumer, EPA should monitor and track such results to ensure that the consumer is making an informed decision. EPA can use results from the assessment to monitor the impact of management decisions, identify weaknesses and strengths in their processes, and ultimately ensure the integrity of 21 ------- the label. However, we found the emphasis of the oversight is skewed to the actions of ENERGY STAR partners, not the accuracy of information available to individual retail consumers. Products Marketed as ENERGY STAR Prior to EPA Notification According to EPA, even though the RSL may find instances where a product was mislabeled as ENERGY STAR, EPA may determine that subsequent to the assessment the product does qualify for ENERGY STAR. The Agency explained that this situation is caused by the manufacturer labeling the products prior to officially reporting the products to EPA as ENERGY STAR qualified. According to EPA, 17 of the 21 television models found to be always labeled incorrectly in the May 2006 RSL were caused by the manufacturers labeling the products prior to completing the qualification process. According to ENERGY STAR staff, once the manufacturer tests the product and it qualifies under the ENERGY STAR standards the manufacturer can begin labeling the product as ENERGY STAR. Therefore labeling can occur prior to EPA ever receiving any notification that this particular product model qualifies under the self-certification guidelines of the ENERGY STAR program. However, according to the ENERGY STAR product certification process, as discussed in Chapter 2, the manufacturer consents to testing its products according to established specifications and guidelines. The manufacturer then must notify EPA of the qualified products, certifying that these products are ENERGY STAR qualified and meet the ENERGY STAR performance specifications. Further the manufacturer is required to provide certification summary data per individual product specification requirements. Therefore the process of manufacturer self-certification of its products includes notifying EPA and submitting appropriate summary data to the Agency. In contrast, the Agency interprets this criterion to mean that the manufacturer can begin using the ENERGY STAR label prior to sending in certification and supporting documentation. ENERGY STAR staff said it would be too time consuming and labor intensive to review the summary data provided by the manufacturer prior to granting it access to the label. Rather EPA requires the manufacturer to provide annual (or in few cases semi-annual) updates identifying which new product models should be added to the ENERGY STAR product qualification list. ENERGY STAR staff said that as they move to a more automated system the information may be provided to EPA on a more timely basis. Conclusion While monitoring the accuracy of the use of the ENERGY STAR label is a component of the Retail Store Level Assessment, we found little oversight over the use of the ENERGY STAR label in retail stores, which is generally the 22 ------- purchase point for the consumer. ENERGY STAR staff could not provide documentation related to followup actions taken, final results for all retail store assessments, or status of label inconsistencies. Additionally, when planning for upcoming assessments ENERGY STAR staff stated that no consideration is given to past "bad performers." Such consideration might allow EPA to determine if certain models of products reviewed are consistently being mislabeled. Also, the results of these assessments can show both strengths and weaknesses of the ENERGY STAR program and allow the program to focus on and budget for particular areas of concern. Furthermore, we found that EPA lacks specific guidance for the contractor regarding what follow up actions should be taken for various circumstances. After we had discussions with ENERGY STAR staff, they acknowledged the lack of guidance in this area and drafted a chart. This draft chart could be used to develop potential guidance for the contractor or EPA staff in addressing label inconsistencies. OIG Recommendations To ensure the integrity of the ENERGY STAR label for qualified EPA products, we recommend that the Principal Deputy Assistant Administrator for the Office of Air and Radiation: 5-1. Establish standards to ensure label use inconsistencies found during the retail store level assessments are systematically recorded, appropriate actions taken, and inconsistencies are tracked until resolved or otherwise complete. 5-2. Establish standard operating procedures for contract oversight to assure that all contractually required work is complete and meets the contract requirements. Agency Comments and OIG Evaluation EPA generally concurred with our recommendations. However, EPA's response to both recommendations is not sufficient to complete the intent of the recommendations. In regard to recommendation 5-1, the Agency needs to establish standards to ensure documentation of resolutions to identified labeling inconsistencies. Furthermore all followup actions should be documented and tracked systematically. In regard to recommendation 5-2, the Agency has revised the format of retail store level assessment. However, as stated in the recommendation, the Agency needs to establish a standard operating procedure to ensure proper oversight of all contractually required work. EPA's detailed response and the OIG's evaluation are in Appendices C and D. The OIG has incorporated technical corrections and clarifications from EPA's response into the final report as appropriate. 23 ------- Status of Recommendations and Potential Monetary Benefits POTENTIAL MONETARY RECOMMENDATIONS BENEFITS (in $000s)2 Planned Rec. Page Completion Claimed Agreed To No. No. Subject Status1 Action Official Date3 Amount Amount 3-1 11 Clarify the decision criteria and document the 0 Principal Deputy process for revising an ENERGY STAR Assistant Administrator, specification, including identifying circumstances Office of Air and Radiation when a specification revision would not be revised, despite a high market share of qualified products. 4-1 17 Establish a formal Quality Assurance Plan for product and verification testing to provide a reasonable assurance results represent the products available and the certification of others may be relied upon. 4-2 17 Coordinate verification testing with product 0 Principal Deputy specification setting and revision process to ensure Assistant Administrator, products are selected in a timely and relevant Office of Air and Radiation basis. 5-1 23 Establish standards to record label use 0 Principal Deputy inconsistencies found during the retail store level Assistant Administrator, assessments are systematically recorded, Office of Air and Radiation appropriate actions are taken, and infraction are tracked until resolved or otherwise completed. 5-2 23 Establish standard operating procedures for 0 Principal Deputy contract oversight to assure that all contractually Assistant Administrator, required work is complete and meets the contract Office of Air and Radiation requirements. 0 Principal Deputy Assistant Administrator, Office of Air and Radiation 1 0 = recommendation is open with agreed-to corrective actions pending; C = recommendation is closed with all agreed-to actions completed; U = recommendation is undecided with resolution efforts in progress 2 Identification of potential monetary benefits was not an objective of this evaluation. 3 In accordance with EPA Manual 2750, the Agency is required to provide a written response to this report within 90 calendar days that will include a corrective actions plan for agreed upon actions, including milestone dates. 24 ------- Appendix A ENERGY STAR Qualified Product Categories EPA ENERGY STAR PRODUCTS (48) Air Source Heat Pumps Battery Charging Systems Boilers Cassette Decks CD Players Ceiling Fans Central Air Conditioners Clock Radios Combo Phone Units Commercial Freezers Commercial Fryers Commercial Hot Food Holding Cabinets Commercial Refrigerators Commercial Steam Cookers Computers Copiers Cordless Phones Dehumidifiers DVD Products Exit Signs External Power Adaptors Fax Machines Furnaces Geothermal Heat Pumps Indoor Residential Light Fixtures Insulation & Air Sealing Light Commercial Air Conditioners Light Commercial Heat Pumps Mailing Machines Mini Home Audio Systems Monitors Outdoor Residential Light Fixtures Powered Speakers Printers Programmable Thermostats Rack Home Audio Systems Roof Products Room Air Cleaners Scanners Stereo Amplifiers Stereo Receivers Televisions Traffic Signs1 Transformers 2 VCRs Vending Machines Ventilating Fans Water Coolers DOE ENERGY STAR PRODUCTS (9) Clothes Washers Compact Fluorescent Light Bulbs Dishwashers Doors Freezers Refrigerators Room Air Conditioners Skylights Windows TOTAL ENERGY STAR PRODUCTS AS OF OCTOBER 2006: 57 Source: Internal EPA ENERGY STAR data 1 EPA suspended the ENERGY STAR Traffic Signals specification effective May 2007. 2 EPA suspended the ENERGY STAR Transformers specification effective May 2007. 25 ------- Appendix B Detailed Scope and Methodology To describe how effectively EPA managed the ENERGY STAR Product Labeling Program, we reviewed internal ENERGY STAR policy and guidance documents and recent ENERGY STAR reports, and held detailed discussions with both internal and external stakeholders. We reviewed EPA's 2003-2008 Strategic Plan: Direction for the Future, EPA's Fiscal Year 2005 Annual Performance Plan, and budget allocations for the ENERGY STAR program. We also reviewed relevant reports by the U. S. Government Accountability Office, Lawrence Berkley National Laboratory, and the Consortium for Energy Efficiency. To determine the extent EPA ensures that ENERGY STAR product specifications are sufficient and revised timely, we analyzed the product specification setting and revision processes. We reviewed applicable guidance documents such as the 2001 Product Labeling Business Plan and The ENERGY STAR Label: A Summary of Product Labeling Objectives and Guiding Principles to determine EPA's procedures. We also conducted a detailed analysis of relevant data for all product revisions made to date or scheduled for the immediate future, and compiled a detailed schedule documenting the history of all product specifications and revisions. Prior to and subsequent to completing our analysis and schedule, we held detailed discussions with ENERGY STAR staff. To determine the extent EPA verifies that certified products adhere to the ENERGY STAR specifications, we analyzed both the self-certification and verification processes. We reviewed applicable guidance or compliance documents such as ENERGY STAR Partnership Agreement, The ENERGY STAR Label: A Summary of Product Labeling Objectives and Guiding Principles, and ENERGY STAR Compliance Testing Initiative. Our review included a detailed review of all certification requirements and guidance documents on verification testing. We further analyzed EPA's efforts to date with verification testing, including analyzing applicable costing data to determine average estimated cost of a completed test. We analyzed whether EPA's use of their selection methodology was consistent with available guidance. Finally, we analyzed the amount of completed verification tests versus overall budget expenditures to demonstrate the relative commitment of resources that have been allocated to this quality assurance process. To determine the extent EPA ensures the proper use of the ENERGY STAR label/logo, we analyzed efforts to monitor the proper use of the ENERGY STAR label. We reviewed all applicable guidance and criteria documents such as the ENERGY STAR Program Identity Guidelines, EPA's Product Labeling Business Plan and The ENERGY STAR Label: A Summary of Product Labeling Objectives and Guiding Principles. We reviewed EPA's efforts in monitoring newspapers, magazines, and trade journals for use of the ENERGY STAR label. However, we did not validate these efforts. We performed a detailed analysis of the Retail Level Store Assessments (RSL), including reviewing the results of the assessments and applicable followup actions. To determine contractor requirements and 26 ------- deliverables, we reviewed the applicable statement of work for Contract EP-WO-6001. Finally, we reviewed the contractor reports submitted to EPA of completed RSL rounds to determine the rate and nature of inconsistencies and followup actions taken. 27 ------- Appendix C OIG's Detailed Analysis of Agency Response The OIG's evaluation of the Agency's Comments is documented within the table below, by key bulleted point and recommendation number. For full text of the Agency's comments see Appendix D. # Section/Page OAR Comments OIG Response 1 Chapter 3, pages 10-11 EPA has clearly articulated principles, formally published in 2003 and available on our web site, for ENERGY STAR specifications. These principles are also employed as part of revising ENERGY STAR specifications. The principles are well-documented, distributed broadly with industry and used consistently in the specification setting and revision processes. See Appendix D, Page 40, first bullet The OIG did not conclude that the original guiding principles lacked documentation. The OIG was aware of the guiding principles as evidenced by the fact they were cited within the draft report. However, we found that the original document defining the guiding principles did not state that they were to be used as the principles for specification revision. We were informed by Agency staff during our evaluation that the same principles were used for both the specification setting and revision processes. In addition as discussed in more detail in OIG's response #2, the Agency has not documented the actual decision making process. Therefore there is no evidence that the principles have been used consistently. 2 Recommendation 3-1, page 11 EPA uses the same principles for initial specification of a product and for revising an ENERGY STAR specification. However, we believe it is a useful and valuable addition to program documentation to explicitly describe the conditions under which the EPA undertakes a revision of an existing ENERGY STAR specification and, in particular, to explain why high market share in and of itself is not necessarily sufficient to warrant a specification revision. Status: Complete See attached "Factors Considered: Initiating an ENERGY STAR Specification Revision," completed June 26, 2007 See Appendix D, Pages 37-38 The OIG does not consider the Agency's response adequately addresses the intent of the recommendation because it lacks details as to how the revision process is to be carried out annually. While EPA "revised" the guidance to include language for specification revision, the document did not clearly address how the program staff make decisions. Rather, the revised document is a restatement of the previously existing six principles referred to in Chapter 2, with one change: EPA deleted the statement, "Typically, the specification is set to recognize the top 25 percent of energy performing models on the market." In addition, the OIG finds that the guiding principles, whether they are used for initial specification setting or revision, do not constitute criteria to determine when specifications need revision. These principles do not provide direction as to how they should be interpreted or how they will be applied. For example, EPA has not specified the conditions that trigger the revision process or what constitutes the decision making process. 28 ------- In addition, while the Agency states that the guiding principles are factors to be considered in the specification revision process, it does not explain how these principles are to be applied to the decision making process. For example, it does not explain what constitutes "significant energy savings" and how they can be realized on a national basis. The document also states that: Each time an ENERGY STAR efficiency level is established, whether for the first time or as part of a revision, these principles are balanced to ensure that the specified level will deliver significant aggregate energy savings while differentiating products that are cost-effective to the consumer and do not compromise functionality or performance. However, the document does not explain how the principles are "balanced" nor if there is a system in place that prioritizes and/or weights some principles over others in the decision making process. Furthermore, the OIG believes that the initial specification setting and revision processes are two separate processes with different intended purposes, thus EPA should have specific guidance and criteria relative to each process. Finally, while the Agency states they make decisions annually on a case by case basis, we see no evidence in their response of agreement to maintain documentation of the results of their annual decision making process. Until this is done, there is no evidence to support how EPA makes any of the specification revision decisions. Therefore, it remains the OIG's position that the Agency's response is non responsive and EPA still lacks a formal methodology for revising specifications. 3 Draft Recommendation 3-2 We do not find this recommendation to be a useful addition to evaluating and improving the operation of the ENERGY STAR program. Instead, we believe that the investment we have made in routinely tracking market share as a percent of product sales more closely tracks with program results. This provides information that directly translates to energy and greenhouse gas savings and plays an important role in our program evaluation process. Furthermore, EPA already collects information on the percent of models in the specification setting process which is when it is most programmatically useful. One of the guiding principles previously had a component that related to the percentage of Energy Star products available on the market as compared to non-ENERGY STAR products. During our fieldwork we were informed that while this was listed as a part of the guiding principle, they did not track data regarding what percentage of available products were ENERGY STAR. However as indicated in their response to recommendation 3-1, the Agency has deleted that reference to market share from the guiding principles that govern specification revision. Based on the new specification revision guiding principles the OIG agrees to remove this recommendation. 29 ------- Status: Suggest removing this recommendation given that ENERGY STAR qualified models are already tracked as part of the specification revision process. See Appendix D. Page 38 Chapter 4, pages 12-14 and 17 Self-certification is a broadly used practice in the United States and other countries. It is generally viewed as effective and has been employed for decades across many types of programs that touch U.S. consumers. Self-certification programs are viewed as particularly appropriate in countries such as the United States that have competitive market places that allow manufacturers to question their competitors and that also have robust consumer protection activities. Importantly, there is no evidence that these self-certification programs do not work. The ENERGY STAR program was designed using self- certification in the early 1990's. Over the last fifteen years there has been no new information to suggest that self-certification is not a reasonable approach in the U.S. Two other examples of well-known programs that rely upon self- certification include the Energy Guide label and the food nutrition label. A review of these two programs relative to the ENERGY STAR program would show that EPA has instituted a comprehensive program in to assure the proper use of the ENERGY STAR label that goes well beyond those of other programs. This includes product verification testing, as well as the development of product-specific post-manufacturing testing programs in product areas requiring additional attention. EPA should be recognized for instituting these additional practices as opposed to faulted for relying upon the broadly used and accepted practice of self- certification. The OIG did not evaluate the merits of the self- certification process and thereby did not conclude that the self-certification process was unacceptable as the Agency's response implies. Rather we evaluated the effectiveness of the management controls in place over the process in ensuring the integrity of the ENERGY STAR label. The OIG also never "faulted" EPA for using the practice of self-certification. However, the OIG found that the Agency has not done enough to ensure that the self- certification process is effective. We found that the amount of testing and/or controls in place by EPA is neither effective to act as a deterrent or to identify system issues. The OIG's position and conclusions are unchanged on this matter and are explained in detail within Chapter 4 under the subsection entitled, "ENERGY STAR Verification Testing Conducted on a Minimal Basis." The OIG reemphasizes the fact that EPA is conducting verification testing on a limited basis and has no reasonable assurance that the self-certification process is effective. Limited testing is not cost efficient if it does not provide enough confidence that the system is operating effectively. The OIG did not review either the EnergyGuide or the Food Nutrition Labels, as they were not within the scope of the review on EPA's ENERGY STAR program. Therefore, we take no position as to the effectiveness or controls in place of these labels. 30 ------- EnergvGuide Label Since 1980, manufacturers of appliances have been required to label their products with energy use information in the form of the Federal Trade Commission's yellow Energy Guide label. Similar to the ENERGY STAR program, manufacturers agree to test their products, submit data and attach a label. No prior approval is needed. This label is used on thousands of products that consumers purchase each year. Food Nutrition Label Since 1990, producers of most processed food products have been required to label those items with nutrition information. Producers are required to measure required nutrients and report them on food packaging. The FDA does not require prior approval to verily manufacturer labels nor does it track labeling by food producers. These labels are used to guide billions of food- related purchasing decisions each year. See Appendix D, Pages 40-41, second bullet 5 Recommendation 4-1, page 17 EPA has a comprehensive program in place for assuring the proper use of the ENERGY STAR label, including that the products displaying the label meet the performance requirements. Verification testing is one of many checks on the certification process that EPA has established to cost- effectively protect the integrity of the ENERGY STAR program. Other important elements include formal partnership agreements with manufacturers and an initial certification process that uses standardized, formal test procedures and the review of submitted data. EPA also leverages a number of third-party test programs, where they exist. These include programs established by trade associations and The OIG accepts the time frame established for completion of the formal Quality Assurance Plan However, for the OIG to accept this action as responsive to the recommendation, the Agency cannot simply document their existing program. The Agency's quality assurance plan should at a minimum describe in detail the necessary quality assurance, procedures, and other technical activities that must be implemented to ensure that the results of work performed satisfy stated performance criteria. Simply documenting the processes currently in place will not address the intent of the recommendation. The Agency claims to have a comprehensive program to ensure proper use of the label. However, we found the program lacks key components such as a quality assurance plan and adequate oversight and testing. The OIG has no objection to the reliance on self- certifications, verification testing or third-party testing. However, as discussed within Chapter 4 of 31 ------- relied upon by the federal government to ensure compliance with federal efficiency standards and other federal programs. EPA will establish formal documentation - a Quality Assurance Program - on the various elements of its compliance monitoring system for ENERGY STAR qualified products, how these elements fit together, and on how this program provides reasonable assurance that the results are representative of the products available Status: Initiated. Expected to be Complete in 9 months. See Appendix D, Page 38 this report, we found EPA's current program to be insufficient to provide reasonable assurance that the self-certification program, verification testing and third-party testing is effective. The OIG emphasizes the fact that EPA is conducting verification testing on a limited basis and excludes eleven qualified products from testing because these products are eligible for testing under various third- party programs. The OIG disagrees with the Agency's position that the use of third-party testing services as a check of the certification's accuracy is sufficient, since EPA has no process for reviewing and accepting third-party process. EPA needs to validate both what third-party testing is occurring and their procedures in order to exclude products from consideration from the Agency's verification testing. The OIG restates its position that EPA lacks both assurance of this testing mechanism and data related to which manufacturers or actual qualified product models have voluntarily participated. In order for the Agency to have a proper Quality Assurance Plan in place and to continue to exclude these products from testing, it should include data and measures to validate these results. Finally, the OIG does not agree that the formal partnership agreements serve as a check on the certification process given that the review of the submitted data by the Agency's own omission is very limited. Furthermore, self-certification data submittal is sometimes not received by EPA for up to one year after the manufacturer has been labeling their products as ENERGY STAR. This practice leaves the Agency more vulnerable and demonstrates the lack of oversight and reasonable assurance that the self-certification process is effective. 6 Chapter 4, pages 14-15 According to the Agency the statement, the Agency's verification testing lacks a clear documented methodology governing products selected for verification tests and does not test for statistically valid results needs further context. EPA lias a clear methodology for its verification testing program which it lias used for more than five years and which it has further documented in a technical addendum to the work order for this testing. Further, this testing is designed to identify systemic issues and to act as a deterrent. If initial issues are found, EPA places the burden on the manufacturer to do additional product testing. EPA also undertakes As reported in the draft report Table 4.2, EPA lias not adopted a consistent approach to selecting products for verification testing. For example, when televisions were tested in 2002, fifteen models from five manufacturers were selected. In all, 45 units were tested which averages to three televisions per model. In contrast, in 2003 EPA tested an average of 2.5 telephony units per model (i.e., 50 units selected among 20 models from six manufacturers), while in 2006 EPA tested an average of only one monitor unit (i.e., 20 units selected among 20 models from 11 manufacturers.) The inconsistent number of units per model/manufacturer and the subjective application of the selection criteria (i.e., market sales vs. Product Development Team input) decreases transparency. Furthermore, the small number of units tested per model/manufacturer provides little confidence in the testing results. 32 ------- additional work to address any systemic issues such as establishing broader testing programs, where warranted. This approach is substantially more cost-effective than pursuing statistically valid testing across the 40,000 or more products currently represented in the ENERGY STAR program. See Appendix D, Page 41, first full bullet The OIG disagrees with EPA's statement, "Further, this testing is designed to identify systemic issues and to act as a deterrent." As discussed within this section in Chapter 4, presently EPA is not testing enough to project any systematic issues or to demonstrate that their limited testing acts as a deterrent. Despite the fact that the program has grown, EPA did not begin to conduct verification tests until ten years after its creation. Furthermore, once they began it was and has continued to be done on a very limited basis. The costs incurred for the verification testing represented less than half of one percent of the total ENERGY STAR budget. Finally, in their response EPA overstated the amount of products available in the overall universe for consideration in its verification testing. In respect to the testing program, the Agency cited a universe of "40,000 or more" products, however, as reported in chapter 4, 14 of the 48 Energy Star product categories are excluded from the Agency's verification testing program. 7 Recommendation 4-2, page 17 EPA currently selects product categories that are to undergo verification testing in a given year based on a number of factors including whether or not the specification has been recently revised. EPA has included this factor in written documentation for product selection for verification testing; this was completed in May 2007. As an example, EPA is including imaging products in the next round of verification testing. The ENERGY STAR specification for this product category was effective as of April 2007 and this testing will provide important information on the transition to this new specification. Status: Complete See Appendix D, Pages 38-39 The OIG agrees with the Agency's effort to coordinate verification testing and the specification revision processes. This is a start toward the intent of the recommendation, but is insufficient to complete it. One example does not demonstrate how this process is working. The OIG reiterates the need for more specificity regarding implementation of this recommendation to be included in the corrective action plan. 8 Chapter 5, pages 21-23 According the Agency the statement, "little oversight of the ENERGY STAR label in retail stores, " is not true and overstates the role of the retailer relative to product labeling. Labeling of products is controlled by the product manufacturer, and EPA lias a number of mechanisms in place to ensure appropriate use of the The Retail Store Level assessment (RSL) is a vital part of monitoring the usage of the ENERGY STAR label in the retail store, which is generally the purchase point for the consumer. The OIG stands by its conclusion because previous RSL's have identified consistent problems with products being labeled as ENERGY STAR that were not on the Agency's qualified list. Furthermore, the RSL's also found problems with products that are qualified as 33 ------- ENERGY STAR label. The in-store labeling check is used primarily to update the ENERGY STAR qualifying product lists and check for broad systemic issues. The vast majority of issues found in the retail setting are ones where the product is qualified for the ENERGY STAR but the manufacturer is behind in submitting the information to EPA. If products are found that do not meet the ENERGY STAR specifications, EPA pursues these logo violations using standard protocols. EPA maintains information on the status of all such violations. See Appendix D, Page 41, second Ml bullet ENERGY STAR, but are not labeled consistently as ENERGY STAR. The OIG does not draw any conclusions regarding the role of the retailer in the labeling or mislabeling of ENERGY STAR products. We highlight issues raised by the RSLs and EPA's lack of oversight over the follow up process. As indicated in the draft report and in several RSLs, the issues found can mostly be attributed to the fact that manufacturers are able to label their products prior to submitting certification test results to EPA. The Agency's statement with regard to the primary purpose of the RSL's contrasts their stated goals of the program. As the OIG cited within Chapter 5 of the draft report, the RSL seeks to address four goals. The assessment of the accuracy of product labeling is one of EPA's stated goals of the RSL. 9 Chapter 5, pages 22 According to the Agency the IG presented the statement. Manufacturers may label and sell products as ENERGY STAR-qualified up to a year before submitting test results to the Agency, out of context. This practice is consistent with the product certification process the IG cites and the self-certification process overall. EPA has not found there to be significant issues with this process. Manufacturers are providing testing information that shows that their products meet the requirements of the ENERGY STAR program. A more involved effort at this point of the ENERGY STAR product certification process would present unnecessary costs to the federal government and undue delay to the private sector. See Appendix D, Page 41, third full bullet The OIG does not present this statement out of context. The draft report states, "Manufacturers may label and sell products as ENERGY STAR prior to submitting test results to the Agency for up to a year." As highlighted in Chapter 5, under "Products Marketed As ENERGY STAR Prior to EPA Notification", the Agency does not require manufactures to submit certification and supporting documentation prior to using the ENERGY STAR label. This is a control weakness for ensuring the integrity of the label is not compromised. While the Agency states in its response that the OIG has used this fact out of context, they further state that this practice is consistent with EPA's application of the self-certification process. 10 Recommendation 5-1, page 23 EPA experience is that misuse of the ENERGY STAR label is extremely low. For example, trademark violations as a percent of companies using ENERGY STAR in their advertising is measured at 0.6 percent from 2003 to 2006. EPA has well established standards to ensure that identified labeling misuse is systematically pursued and resolved, including when misuse is identified The OIG does not consider the Agency's response to this recommendation adequate to meet the intent of the recommendation. The scope of the OIG's review was on the Agency's oversight of the use of the ENERGY STAR label. Specifically, the focus of the OIG's review was on the retail store level assessments conducted by the Agency and not on the use of the ENERGY STAR label in general advertising. The statistics provided by the Agency in their response, while they may be accurate, are not relevant to the review of the retail store level 34 ------- as part of the retail store level assessments (RSL). These standards which have been in place for many years are documented in the report. Maintaining the Value of ENERGY STAR — 2006, finalized in spring 2007. This document includes a new flowchart documenting the protocol for responding to labeling violations discovered as part of RSL. EPA has enhanced its management review of the status of the labeling violations. Status: Complete See Appendix D, Page 39 assessments. In addition, while EPA states they have "well established standards" in place to ensure label misuse is systematically resolved; the OIG was not provided any documentation regarding specific follow up actions taken related to inconsistencies identified the retail store level assessments. Furthermore, the document entitled, "Maintaining the Value of ENERGY STAR - 2006\ stated by EPA in their response as "finalized in spring of 2007" still has not been officially issued. As of July 2007, it still did not appear in any form on their official web site. EPA's statement that these standards have been in place for years directly contradicts what the OIG was told during the review. Furthermore, the "new flowchart" referenced was developed at the end of our fieldwork and later incorporated into the Agency's draft guidance document. Prior to that, the officials were unclear as to what follow up actions were being taken by the contractor when issues where identified during the assessment. Furthermore, EPA has not routinely required the contractor to provide final reports on the RSL reviews. While the Agency states in the response that they are enhancing the RSL reports, there is no specificity as to what enhancements EPA has made. Reports should still provide information related to issues found and document the status of follow up actions. Therefore, the Agency's response does not meet the intent of this recommendation. In order for the RSL program to be cost effective, EPA must report what was found and how follow up actions were resolved. In addition, without such detailed reporting EPA has no basis to conclude that the misuse of the label is "extremely low." The use of detailed reporting also will assist the Agency in identifying systematic problems and enable them to reduce problems in the future. 11 Recommendation 5-2, page 23 The issue of concern to the IG did not stem from a lack of standard operating procedures for contract oversight (i.e. finalization of the reports was not contractually required), but concerns by the IG that EPA was not finalizing field data that needed to go through an important reconciliation process before it could be actionable. Accordingly, we recommend that the IG reword this recommendation to state "Establish standard operating procedures so that The OIG's recommendation relates to the lack of oversight by Agency staff regarding the RSL. The fact that many of the assessments were never issued in final is evidence that the Agency was not providing proper oversight. Many of the reports lacked conclusions and according to the Agency the numbers were not reflective of the final results. Furthermore the contract for the RSL does require a final report as a deliverable and should be obtained as a standard practice. In addition, one of the benefits of the RSL is that it can be used as a mechanism to identify systematic 35 ------- RSL field reports are routinely finalized." EPA has concluded that the standard report format can be adjusted so that report can be finalized in a cost efficient manner from the results of the initial screening work; this change in practice has been instituted. Status: Complete As of June 2007, the standard report format for the field studies lias been revised so that the reports can be finalized in a cost efficient manner, and the change in practice has been instituted See Appendix D, Page 39 problems or strengths of the program. However, we found the Agency was not tracking issues found during the RSL in an effort to identify "bad performers" or systematic strengths or weaknesses. The Agency's proposed corrective action does not meet the intent of the recommendation. Simply revising the format of the report but not requiring the report to be finalized is not acceptable. Furthermore, in order for the recommendation to be considered complete the OIG needs to see documentation of what will be included within the final reports and how the process was modified. It continues to be the OIG's position that a final report is necessary. At a minimum, the report should document the results of the RSL reviews, how any inconsistencies or mislabeling instances were found, the nature of the inconsistencies, any follow up actions taken and how issues were resolved. 36 ------- Appendix D Agency Response June 28, 2007 MEMORANDUM SUBJECT: Comments on the Draft Evaluation Report: ENERGY STAR Program Can Strengthen Controls Protecting the Integrity of the Label FROM: Elizabeth Craig Deputy Assistant Administrator TO: Jeffrey Harris, Director Cross-Media Issues, Office of Program Evaluation Thank you for the opportunity to comment on the Draft Evaluation Report: ENERGY STAR Program Can Strengthen Controls Protecting the Integrity of the Label. We are pleased that the Inspector General's Office found no significant issues relating to the quality or performance of ENERGY STAR qualified products or the overall success of the ENERGY STAR labeling program. At the same time, we appreciate and share your interest in making the ENERGY STAR Program as effective as possible. Accordingly, we have implemented most of the recommendations provided in your draft report and would like to report on the status of this implementation. We are particularly pleased to report that we have implemented four of the six recommendations and initiated the implementation of a fifth. However, there is one recommendation that we do not find to be a useful addition to improving the operation of the ENERGY STAR program and ask for that recommendation to be deleted. In addition, we have identified that the Report contains some important factual errors as well as other misstatements. We are providing comments on these issues so that the report can better represent how the ENERGY STAR program works and contribute to the continuing success of the ENERGY STAR program. OAR Response to IG Recommendations in Draft Report and Status of Implementation Recommendation 3.1: Clarify and document the decision criteria for revising an ENERGY STAR specification, including identifying circumstances when a specification would not be revised, despite a high market share of qualified products. OAR Response: EPA uses the same principles for initial specification of a product and for revising an ENERGY STAR specification. However, we believe it is a useful and valuable addition to program documentation to explicitly describe the conditions under which the EPA undertakes a revision of an existing ENERGY STAR specification and, in 37 ------- particular, to explain why high market share in and of itself is not necessarily sufficient to warrant a specification revision. Status: Complete See attached "Factors Considered: Initiating an ENERGY STAR Specification Revision," completed June 26, 2007 Recommendation 3.2: Monitor the percentage of Energy Star qualified products as compared to total available products in a product category. OAR Response: We do not find this recommendation to be a useful addition to evaluating and improving the operation of the ENERGY STAR program. Instead, we believe that the investment we have made in routinely tracking market share as a percent of product sales more closely tracks with program results. This provides information that directly translates to energy and greenhouse gas savings and plays an important role in our program evaluation process. Furthermore, EPA already collects information on the percent of models in the specification setting process which is when it is most programmatically useful Status: Suggest removing this recommendation given that ENERGY STAR qualified models are already tracked as part of the specification revision process. Recommendation 4.1: Establish a formal Quality Assurance Program for products and verification testing to provide a reasonable assurance results are representative of the products available and the certification of others (self- and third party) may be relied upon. OAR Response: EPA has a comprehensive program in place for assuring the proper use of the ENERGY STAR label, including that the products displaying the label meet the performance requirements. Verification testing is one of many checks on the certification process that EPA has established to cost-effectively protect the integrity of the ENERGY STAR program. Other important elements include formal partnership agreements with manufacturers and an initial certification process that uses standardized, formal test procedures and the review of submitted data. EPA also leverages a number of third-party test programs, where they exist. These include programs established by trade associations and relied upon by the federal government to ensure compliance with federal efficiency standards and other federal programs. EPA will establish formal documentation - a Quality Assurance Program - on the various elements of its compliance monitoring system for ENERGY STAR qualified products, how these elements fit together, and on how this program provides reasonable assurance that the results are representative of the products available. Status: Initiated Expected to be complete in 9 months Recommendation 4.2: Coordinate verification testing with product specification setting and revision process to ensure products are selected in a timely and relevant basis. 38 ------- OAR Response: EPA currently selects product categories that are to undergo verification testing in a given year based on a number of factors including whether or not the specification has been recently revised. EPA has included this factor in written documentation for product selection for verification testing; this was completed in May 2007. As an example, EPA is including imaging products in the next round of verification testing. The ENERGY STAR specification for this product category was effective as of April 2007 and this testing will provide important information on the transition to this new specification. Status: Complete Recommendation 5.1: Establish standards to ensure label inconsistencies found during the retail store level assessments are systematically recorded, appropriate actions taken and inconsistencies are tracked until resolved or otherwise complete. OAR Response: EPA experience is that misuse of the ENERGY STAR label is extremely low. For example, trademark violations as a percent of companies using ENERGY STAR in their advertising is measured at 0.6 percent from 2003 to 2006. EPA has well established standards to ensure that identified labeling misuse is systematically pursued and resolved, including when misuse is identified as part of the retail store level assessments (RSL). These standards which have been in place for many years are documented in the report, Maintaining the Value of ENERGY STAR — 2006, finalized in spring 2007. This document includes a new flowchart documenting the protocol for responding to labeling violations discovered as part of RSL. EPA has enhanced its management review of the status of the labeling violations. Status: Complete Recommendation 5.2: Establish standing operating procedures for contract oversight to assure that all contractually required work is complete and meets the contract requirements. OAR Response: The issue of concern to the IG did not stem from a lack of standard operating procedures for contract oversight (i.e. finalization of the reports was not contractually required), but concerns by the IG that EPA was not finalizing field data that needed to go through an important reconciliation process before it could be actionable. Accordingly, we recommend that the IG reword this recommendation to state "Establish standard operating procedures so that RSL field reports are routinely finalized." EPA has concluded that the standard report format can be adjusted so that report can be finalized in a cost efficient manner from the results of the initial screening work; this change in practice has been instituted. Status: Complete As of June 2007, the standard report format for the field studies has been revised so that the reports can be finalized in a cost efficient manner, and the change in practice has been instituted. 39 ------- OAR Recommendations to Address Factual Errors and Other Misstatements We remain concerned that the draft Report contains important factual errors as well as other misstatements about the program that suggest there are problems where no problems have been found. We seek to address these so that this report can better represent how the ENERGY STAR program works and contribute to the continuing success of the ENERGY STAR program. We encourage you to reexamine and modify the report as it relates to the issues below. • Criteria for initiating and revising ENERGY STAR specifications were unclear and not documented. EPA has clearly articulated principles, formally published in 2003 and available on our web site, for ENERGY STAR specifications. These principles are also employed as part of revising ENERGY STAR specifications. The principles are well-documented, distributed broadly with industry and used consistently in the specification setting and revision processes. • No reasonable assurance that the self-certification process is effective. Self- certification is a broadly used practice in the United States and other countries. It is generally viewed as effective and has been employed for decades across many types of programs that touch U.S. consumers. Self-certification programs are viewed as particularly appropriate in countries such as the United States that have competitive market places that allow manufacturers to question their competitors and that also have robust consumer protection activities. Importantly, there is no evidence that these self- certification programs do not work. The ENERGY STAR program was designed using self-certification in the early 1990's. Over the last fifteen years, there has been no new information to suggest that self-certification is not a reasonable approach in the U.S. Two other examples of well-known programs that rely upon self-certification include the EnergyGuide label and the food nutrition label. A review of these two programs relative to the ENERGY STAR program would show that EPA has instituted a comprehensive program in to assure the proper use of the ENERGY STAR label that goes well beyond those of other programs. This includes product verification testing, as well as the development of product-specific post-manufacturing testing programs in product areas requiring additional attention. EPA should be recognized for instituting these additional practices as opposed to faulted for relying upon the broadly used and accepted practice of self-certification. EnergyGuide Label Since 1980, manufacturers of appliances have been required to label their products with energy use information in the form of the Federal Trade Commission's yellow Energy Guide label. Similar to the ENERGY STAR Program, manufacturers agree to test their products, submit data and attach a label. No prior approval is needed. This label is used on thousands of products that consumers purchase each year. 40 ------- Food Nutrition Label Since 1990, producers of most processed food products have been required to label those items with nutrition information. Producers are required to measure required nutrients and report them on food packaging. The FDA does not require prior approval to verify manufacturer labels nor does it track labeling by food producers. These labels are used to guide billions of food-related purchasing decisions each year. • The Agency's verification testing lacks a clear documented methodology governing products selected for verification tests and does not test for statistically valid results. This statement needs further context. EPA has a clear methodology for its verification testing program which it has used for more than five years and which it has further documented in a technical addendum to the work order for this testing. Further, this testing is designed to identify systemic issues and to act as a deterrent. If initial issues are found, EPA places the burden on the manufacturer to do additional product testing. EPA also undertakes additional work to address any systemic issues such as establishing broader testing programs, where warranted. This approach is substantially more cost- effective than pursuing statistically valid testing across the 40,000 or more products currently represented in the ENERGY STAR program. • Little oversight of the ENERGY STAR label in retail stores. This statement is not true and overstates the role of the retailer relative to product labeling. Labeling of products is controlled by the product manufacturer, and EPA has a number of mechanisms in place to ensure appropriate use of the ENERGY STAR label. The in-store labeling check is used primarily to update the ENERGY STAR qualifying product lists and check for broad systemic issues. The vast majority of issues found in the retail setting are ones where the product is qualified for the ENERGY STAR but the manufacturer is behind in submitting the information to EPA. If products are found that do not meet the ENERGY STAR specifications, EPA pursues these logo violations using standard protocols. EPA maintains information on the status of all such violations. • Manufacturers may label and sell products as ENERGY STAR-qualified up to a year before submitting test results to the Agency. The IG presents this statement out of context. This practice is consistent with the product certification process the IG cites and the self-certification process overall. EPA has not found there to be significant issues with this process. Manufacturers are providing testing information that shows that their products meet the requirements of the ENERGY STAR program. A more involved effort at this point of the ENERGY STAR product certification process would present unnecessary costs to the federal government and undue delay to the private sector. For your consideration, we have also provided more detailed feedback below on several sections of the Draft Report to address other errors and/or misstatements with regard to the ENERGY STAR program. Attachments 41 ------- Factors Considered: Initiating an ENERGY STAR Specification Revision The ENERGY STAR product labeling program was designed to ensure that key consumer preferences are met - economic savings, product performance and the reward of protecting the environment. In order to consistently deliver on these expectations, the ENERGY STAR program has six established principles that guide decisions in terms of whether and at what level to establish an ENERGY STAR specification and when to revise it. 1) Significant energy savings can be realized on a national basis 2) Product performance can be maintained or enhanced with increased energy efficiency. 3) Purchasers will recover their investment in increased energy efficiency within a reasonable time. 4) Energy efficiency can be achieved with several technology options, at least one of which is non-proprietary. 5) Product energy consumption and energy performance can be measured and verified with testing. 6) Labeling would effectively differentiate products and be visible for purchasers. Each time an ENERGY STAR efficiency level is established, whether for the first time or as part of a revision, these principles are balanced to ensure that the specified level will deliver significant aggregate energy savings while differentiating products that are cost-effective to the consumer and do not compromise functionality or performance. In order to effectively differentiate products, the specified efficiency level must allow for a reasonable, nationally available selection of products from a range of manufacturers. In many circumstances, setting an ENERGY STAR efficiency level so that the top 25% of models in terms of efficiency can meet it, offers the desired amount of selection and availability while also promising significant energy savings, cost-effective options and no compromise in performance. The ENERGY STAR product labeling program delivers energy savings and greenhouse gas reductions by facilitating the purchase of more efficient product alternatives. Accordingly, an increase in qualified product market share after an ENERGY STAR specification goes into effect is an important measure of program success. Further, the value of the program is not diminished when there is ultimately broad availability of qualified product in the market place. No matter what the market share of ENERGY STAR qualified products, a consumer who purchases a labeled product gets a product that will contribute to a cleaner environment and save them money without sacrifice in performance. At the same time, when ENERGY STAR qualified products represent a high percentage of the market for a given product category, it suggests there may be an opportunity for additional savings. In general, once market share for a given product category exceeds 50%, that product category is identified for ongoing evaluation in terms of a possible revision. Whether and when a specification revision goes forward is determined by an assessment of market conditions against the key principles. 42 ------- For example, although the market share of ENERGY STAR qualified computers under version 3.0 of the specification exceeded 90% for some time, the specification was not immediately revised because of the challenges associated with addressing "active power." To capture additional meaningful energy savings (principle 1), the existing computer specification needed to extend beyond "stand-by power" to address "active power." Significant time and coordination were required to establish a measurement approach (principle 5) and to overcome the challenge associated with establishing an "active power" specification that did not compromise functionality (principle 2). It was not until these issues were addressed that a specification change, consistent with ENERGY STAR program principles, could be justified. 43 ------- Distribution Appendix E Office of the Administrator Principal Deputy Assistant Administrator for Air and Radiation Agency Followup Official (the CFO) Agency Followup Coordinator Office of General Counsel Associate Administrator for Congressional and Intergovernmental Relations Associate Administrator for Public Affairs Audit Followup Coordinator, Office of Air and Radiation Acting Inspector General 44 ------- |