<
3J
V
r'
"U P R
T>
2
LU
O
T
J
U.S. ENVIRONMENTAL PROTECTION AGENCY
OFFICE OF INSPECTOR GENERAL
Catalyst for Improving the Environment
Evaluation Report
ENERGY STAR Program Integrity
Can Be Enhanced Through
Expanded Product Testing
Report No. 10-P-0040
November 30, 2009
-------
Report Contributors: Daniel Carroll
Jerri Dorsey
Jill Ferguson
Jeffrey Harris
Abbreviations
DVD Digital Versatile Disc
EPA U.S. Environmental Protection Agency
NIST National Institute of Standards and Technology
OIG Office of Inspector General
Cover photos: ENERGY STAR product categories, from left: monitors, DVD products,
and printers. (Photos courtesy EPA)
-------
$
<
73
\
O
1 c>s°
pRQl^
U.S. Environmental Protection Agency
Office of Inspector General
At a Glance
10-P-0040
November 30, 2009
Catalyst for Improving the Environment
Why We Did This Review
We initiated this evaluation to
independently test ENERGY
STAR products to determine
whether their energy-efficient
performance complied with the
ENERGY STAR program's
required specifications.
Background
The U.S. Environmental
Protection Agency (EPA)
established the ENERGY
STAR Labeling Program as an
innovative approach to
environmental protection.
More than 2,400 manufacturers
and over 40,000 individual
product models across
60 product categories are
ENERGY STAR qualified.
In 2007, EPA reported that the
ENERGY STAR program
helped Americans save
180 billion kilowatt-hours,
about 5 percent of U.S.
electricity demand, and
prevented the emission of
40 million metric tons of
carbon equivalents of
greenhouse gases.
ENERGY STAR Program Integrity Can Be
Enhanced Through Expanded Product Testing
What We Found
Almost all of the ENERGY STAR products in our test sample met, and in most
cases exceeded, the program's performance standards. However, selected non-
ENERGY STAR products performed comparably to, and in some cases better
than, ENERGY STAR products. That level of product performance affects the
ENERGY STAR label's image as a trusted national symbol for environmental
protection through superior energy efficiency.
In addition, the performance results of ENERGY STAR and non-ENERGY
STAR products call into question the assumptions used to calculate energy
savings and greenhouse gas reductions attributed to this program. Without an
enhanced testing program, including the testing of non-ENERGY STAR
products, EPA cannot be certain ENERGY STAR products are the more energy-
efficient and cost-effective choice for consumers.
What We Recommend
We recommend that EPA verify estimated energy savings and greenhouse gas
reduction calculations using a market-based performance-testing program that
includes testing non-ENERGY STAR products.
We also recommend that EPA revise the ENERGY STAR Website to include the
established standard alongside qualifying product performance data and to
provide a summary listing of the highest performers.
The Agency disagreed with our conclusions but concurred with both
recommendations.
For further information, contact
our Office of Congressional,
Public Affairs and Management
at (202) 566-2391.
To view the full report,
click on the following link:
www.epa.aov/oia/reports/2010/
20091130-10-P-0040.pdf
-------
^£0SX
? \ UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
| | WASHINGTON, D.C. 20460
pRQl^
OFFICE OF
INSPECTOR GENERAL
November 30, 2009
MEMORANDUM
SUBJECT:
FROM:
ENERGY STAR Program Integrity Can Be Enhanced Through
Expanded Product Testing
Report No. 10-P-0040 • ,
'{Mr
Wade T. Najjum
Assistant Inspector General, Office of Program Evaluation
TO:
Gina McCarthy
Assistant Administrator for Air and Radiation
This is our report on the subject evaluation conducted by the Office of Inspector General (OIG) of
the U.S. Environmental Protection Agency (EPA). This report contains findings that describe the
problems the OIG has identified and corrective actions the OIG recommends. This report represents
the opinion of the OIG and does not necessarily represent the final EPA position. Final
determinations on matters in this report will be made by EPA managers in accordance with
established audit resolution procedures.
The estimated cost of this report - calculated by multiplying the project's staff days by the applicable
daily full cost billing rates in effect at the time, including the costs of purchasing ENERGY STAR
and non-ENERGY STAR products and having them analyzed by the National Institute of Standards
and Technology - is $489,338.
Action Required
In accordance with EPA Manual 2750, you are required to provide a written response to this report
within 90 calendar days. You should include a corrective actions plan for agreed-upon actions,
including milestone dates. We have no objections to the further release of this report to the public.
This report will be available at http://www.epa.gov/oig.
If you or your staff have any questions regarding this report, please contact me at 202-566-0827 or
naiium.wade@,epa.gov. Jeffrey Harris at 202-566-0831 or harris.ieffrev@,epa.gov, or Jill Ferguson at
202-566-2718 or ferguson.iill@,epa.gov.
-------
ENERGY STAR Program Integrity Can Be Enhanced
Through Expanded Product Testing
10-P-0040
Table of C
Chapters
1 Introduction 1
Purpose 1
Background 1
Scope and Methodology 2
2 Enhanced ENERGY STAR Testing Needed to Ensure Program Integrity 4
Product Specification Process 4
Both ENERGY STAR and non-ENERGY STAR Products Met
Specifications 5
ENERGY STAR Product Performance Varied but Similar to
Non-ENERGY STAR Products 5
ENERGY STAR Benefit Claims May Not Be Valid 9
Impact on Consumer Cost Savings 9
Conclusions 10
Recommendations 10
Agency Comments and OIG Evaluation 10
Status of Recommendations and Potential Monetary Benefits 12
Appendices
A Agency Comments on Draft Report 13
B Distribution 15
-------
10-P-0040
Chapter 1
Introduction
Purpose
The purpose of this review was to test U.S. Environmental Protection Agency
(EPA) ENERGY STAR-qualified products for compliance with required
specifications. This independent testing was a check on the validity and
reliability of manufacturers' self-certification testing conducted by EPA's
ENERGY STAR partners. As the testing evolved and we received initial results
showing that a number of the ENERGY STAR products tested exceeded the
standard, we decided to expand our scope and test non-ENERGY STAR products.
Thus, our initial purpose was expanded to also determine how non-ENERGY
STAR products performed in comparison with ENERGY STAR products.
Background
In 1992, EPA's Office of Air and Radiation established the ENERGY STAR
Product Labeling program (ENERGY STAR program) as an innovative approach
to environmental protection. In 1996, EPA partnered with the U.S. Department of
Energy to promote the ENERGY STAR label and broaden the range of products
covered. In the ENERGY STAR 2008 overview of achievements, EPA stated
that the ENERGY STAR label is recognized by more than 75 percent of the
American public. According to EPA, the ENERGY STAR label is the trusted
national symbol for environmental protection through superior energy efficiency.
EPA also reports that ENERGY STAR has helped individuals save on their
energy bills by clearly identifying energy-efficient products with superior
performance in the marketplace.
ENERGY STAR is the most significant of EPA's greenhouse gas avoidance
programs. In 2006 and 2007, the program accounted for over 50 percent of
EPA's contribution. EPA reported that in 2007, the ENERGY STAR program
helped Americans save 180 billion kilowatt-hours, about 5 percent of U.S.
electricity demand, and prevented the emission of 40 million metric tons of
carbon equivalents of greenhouse gases.
The ENERGY STAR program is advertised as a credible, objective source of
information for Americans wanting to make well-informed decisions on how to
improve the energy efficiency of their homes and businesses. More than 2,400
manufacturers and over 40,000 individual product models across 60 product
categories are ENERGY STAR qualified. EPA does not bestow membership into
the ENERGY STAR program without a request from the manufacturer.
Membership is voluntary. Manufacturers are responsible for certifying product
1
-------
10-P-0040
performance and compliance with ENERGY STAR specifications when they
apply to use the label.
In EPA Office of Inspector General (OIG) Report No. 2007-P-00028, ENERGY
STAR Program Can Strengthen Controls Protecting the Integrity of the Label,
issued August 1, 2007, we identified the need to improve EPA's product testing
verification. Specifically, we found the following:
• Verification testing was conducted on a minimal basis.
• Test selection methods were inconsistently applied.
• Only a limited amount of the program budget was used for verification
testing.
• The quality assurance plan needed improvement.
Our report found that testing of ENERGY STAR products did not begin until
2002, 10 years after the program began. When verification testing began, it was
only a small component of EPA's activities. In 2006, the cost of testing
represented less than 0.5 percent of the total ENERGY STAR budget. In its first
5 years of verification testing, EPA averaged only two sets of product verification
tests per year. When our 2007 report was issued, 44,000 qualified product models
existed within the qualified ENERGY STAR product categories. At the end of
2006, EPA had only conducted verification testing on 160 product models in
9 product categories.
Scope and Methodology
Between December 2008 and February 2009, we selected 20 different ENERGY
STAR-qualified products for testing from each of 3 product categories. Products
were selected and purchased online from the list of qualified products on the
ENERGY STAR Website1 and based on availability at major retailers. Products
selected were chosen to represent those that would be available for purchase by
the general public. Two identical models of each product category were
purchased for a total of 40 ENERGY STAR-qualified products per category,
120 products in all.
1 http://www.energystar.gov
2
-------
10-P-0040
The product categories selected and the specification dates were as follows:
In addition to the ENERGY STAR products, we also tested the performance of
some non-ENERGY STAR products. We purchased 10 non-ENERGY STAR
products (2 each of 5 models) from each of the above categories for a total of 30
non-ENERGY STAR products. These products underwent the same testing as
our sample of ENERGY STAR products, and the results were compared with
ENERGY STAR specifications.
We performed our evaluation between April 2008 and September 2009 in
accordance with generally accepted government auditing standards. Those
standards require that we plan and perform the review to obtain sufficient,
appropriate evidence to provide a reasonable basis for our findings and
conclusions based on our review objectives. We believe that the evidence
obtained provides a reasonable basis for our findings and conclusions based on
our objectives.
We entered into an Interagency Agreement with the National Institute of
Standards and Technology (NIST) to perform the testing of all the products in our
sample. NIST tested these products for compliance with ENERGY STAR
program specifications between December 2008 and June 2009.
2 Most recent efficiency standard date listed in the product specification requirements that were used for testing
purposes.
3 DVD products included in our testing were CD players/changers, stereo amplifiers/pre-amplifiers, and stereo
receivers.
4 Forty ENERGY STAR printers were tested; 24 were ink jet and 16 were laser printers.
Products
Specification Date2
Digital Versatile Disc (DVD) products3
Computer monitors
Printers4
January 1, 2003
January 1, 2006
April 1, 2007
3
-------
10-P-0040
Chapter 2
Enhanced ENERGY STAR Testing Needed to Ensure
Program Integrity
Although almost all of the ENERGY STAR products in our test sample met, and
in most cases exceeded, the program's performance standards, many of the non-
ENERGY STAR products tested comparably to, and in some cases better than,
the ENERGY STAR products. Comparable performance affects the image of the
ENERGY STAR label as a trusted national symbol for environmental protection
through superior energy efficiency. In addition, the results call into question the
assumptions used to calculate energy savings and greenhouse gas reductions
attributed to this program. Based on our testing of non-ENERGY STAR
products, EPA cannot be certain ENERGY STAR products are the more energy-
efficient and cost-effective choice for consumers.
Product Specification Process
EPA follows six key principles when establishing consumer product
energy-efficiency specifications:
1. Significant energy savings can be realized on a national basis.
2. Product performance can be maintained or enhanced with increased
energy efficiency.
3. Purchasers will recover their investment in increased energy efficiency
within a reasonable time.
4. Energy efficiency can be achieved with several technology options, at
least one of which is nonproprietary.
5. Product energy consumption and performance can be measured and
verified with testing.
6. Labeling would effectively differentiate products and be visible for
purchasers. Typically, the specification is set to recognize the products
that rank in the top 25 percent in terms of energy efficiency.
4
-------
10-P-0040
Both ENERGY STAR and non-ENERGY STAR Products Met
Specifications
We tested 120 ENERGY STAR-qualified products and 118, or 98 percent, met,
and in most cases exceeded, program requirements for compliance. During our
product testing, we did not find any evidence that the self-certification process
EPA uses allows for products that do not meet the specifications to enter the
program. The only products that failed to meet specifications were two printers of
the same model. Both failed to meet the "Time to Sleep Mode" requirement.5
The categories tested and positive compliance rates are presented in Table 2-1.
Table 2-1: ENERGY STAR-qualified Product Compliance Rates
Product Category
Compliance Rate
DVD products
100%
Computer monitors
100%
Printers
95%
Source: NIST.
Non-ENERGY STAR Product Results
We tested 30 non-ENERGY STAR products comparing performance to program
requirements. A majority of these products were also in compliance with
ENERGY STAR requirements, as shown in Table 2-2 below.
Table 2-2: Non-ENERGY STAR Product Compliance Rates
Product Category
Compliance Rate
DVD products
60%
Computer monitors
80%
Printers
40%
Source: NIST.
ENERGY STAR Product Performance Varied but Similar to
Non-ENERGY STAR Products
Although we expected the ENERGY STAR designation to be a challenge for
products (qualified or nonqualified) to meet, we found the majority of products
tested, including non-ENERGY STAR products, met the ENERGY STAR
efficiency standard, with some exceeding the standard by a wide margin. We also
found that not all ENERGY STAR products performed comparably within the
same product categories; some models were up to several times more efficient
than others. Almost all of the models in our sample, including the non-ENERGY
STAR products, would still qualify under more stringent energy efficiency
standards.
5 The "Time to Sleep Mode" requirement is only one of three tests needed to be ENERGY STAR qualified. Further,
as of July 2009, this model was no longer listed as an ENERGY STAR-qualified product.
5
-------
10-P-0040
Manufacturer-supplied performance data for each ENERGY STAR-qualified
product is available on EPA's Website. However, these data are not readily
usable for performance comparison by consumers because the ENERGY STAR-
qualifying standard is not included.
DVD Testing Results
DVD products must consume less than 1 watt of electricity in standby mode to
qualify for the ENERGY STAR label. In our sample, we found the average
consumption by the ENERGY STAR-qualified products was 0.56 watt, slightly
more than half the allowable amount. The performance of these qualified
products ranged from a low of 0.12 watt to a high of 0.89 watt. Six of the 10 non-
ENERGY STAR products we tested also met the standard. Two of the 10 non-
ENERGY STAR products performed better than 38 of the 40 (95 percent)
ENERGY STAR products we tested.
Specific details on the DVD products tested are presented in Figure 2-1 and
described below. All items below the green bar met ENERGY STAR
requirements.
Figure 2-1: DVD Product Performance
O
~o
o
>
Si
~o
c
TO
1.40
1.20
1.00
% 0.80-
¦o
o
E
3
C
o
o
V)
0.60
0.40
0.20
0.00
u
ENERGY STAR Standard requires
that products consume less than
1 watt in standby mode
¦ ENERGY STAR DVD
Products
~ Non-ENERGY STAR
DVD Products
T1
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39
DVD Products
Source: OIG analysis.
6
-------
10-P-0040
Computer Monitor Testing Results
ENERGY STAR performance standards for monitors vary based on product
resolution and pixel configurations. Figure 2-2 illustrates the deviation in
performance (positive or negative) relative to the applicable standards. The zero
line on the chart below shows the value needed to meet the ENERGY STAR
specification for this product.
Figure 2-2: Computer Monitor Performance
¦ ENERGY STAR
Monitors
~ Non-ENERGYSTAR
Monitors
O)
-10
-20
Computer Monitors
Source: OIG analysis.
The testing results showed that ENERGY STAR monitors surpassed the
ENERGY STAR standards by 16 to 55 percent. Eight of the non-ENERGY
STAR monitors performed similarly to the qualified products, performing 14 to
37 percent more efficiently than the standards. One non-ENERGY STAR
monitor performed more efficiently than half of the qualified products.
7
-------
10-P-0040
Printer Testing Results
As with monitors, printer performance standards vary based on the addition of
certain features included with the base product model. Figure 2-3 shows the
performance relative to the key standard, the "Sleep Allowed Measure." Similar
to Figure 2-2, the zero line on this chart shows the value needed to meet
ENERGY STAR specifications for these standards. While Figure 2-3 only
applies to the ink jet printers tested, the test results for laser printers were similar.
Figure 2-3: Ink Jet Printer Performance
100
90
80
70
60
50
40
30
20
10
¦ ENERGYSTAR InkJet
Printers
~ Non-ENERGYSTAR
InkJet Printers
9 11 13 15
Ink Jet Printers
17 19 21 23
Source: OIG analysis.
Testing results showed that ENERGY STAR ink jet printers surpassed the
ENERGY STAR standards by a low of almost 7 percent to a high of 88 percent.
The laser printers (not shown) exceeded the applicable standards by a low of
2 percent to a high of 81 percent. In addition, the performance of some of the
non-ENERGY STAR ink jet and laser printers exceeded the key ENERGY STAR
efficiency standards by a considerable margin.
8
-------
10-P-0040
ENERGY STAR Benefit Claims May Not Be Valid
Based on our test results, the energy and greenhouse gas reductions claimed for
qualified products may not be valid. EPA uses the formula shown in Figure 2-4
to calculate the annual energy savings benefits and greenhouse gas reductions
resulting from the ENERGY STAR program:
Figure 2-4: ENERGY STAR Energy Savings Calculation
Non-ENERGY STAR product energy consumption
minus
ENERGY STAR product minimum energy consumption
multiplied by
ENERGY STAR product sales
equals
ENERGY STAR Product's Energy Savings
Source: EPA.
This formula and our test results ultimately affect the accuracy of the computed
energy savings attributable to the program. We did not find the ENERGY STAR
products in our sample to have performance levels equal to the minimum
standard, nor did we find all ENERGY STAR products for a particular category to
perform similarly. Both the generalizations inherent in the formula and the final
benefits reported are affected by (1) the overlap between ENERGY STAR and
non-ENERGY STAR products for energy savings, and (2) the variation in
performance within the ENERGY STAR products.
Impact on Consumer Cost Savings
ENERGY STAR products are advertised as offering consumers a means to reduce
home and business energy costs, as well as an opportunity to help reduce
greenhouse gas emissions. In a 2008 survey of consumers, EPA found 63 percent
of households associated the ENERGY STAR label with "efficiency or energy
savings." In addition, of the households that recognized the ENERGY STAR
label and purchased a product in a relevant product category within the past
12 months, 73 percent purchased an ENERGY STAR-labeled product.
Our results for the non-ENERGY STAR products tested show that many of these
products met and exceeded program requirements. However, within the
ENERGY STAR products selected, there were significant performance variations.
These variations between qualified products and overlap with nonqualified
products may mean that the cost savings advertised to consumers is misleading.
Consumers who base their purchase of energy-efficient products on the ENERGY
9
-------
10-P-0040
STAR label do not necessarily realize savings over some non-ENERGY STAR
products. Additionally, there can be a significant difference in efficiency among
ENERGY STAR-qualified products.
Conclusions
The ENERGY STAR program is widely recognized but misunderstood. The
ENERGY STAR label does not necessarily assure consumers superior energy
savings over products that are not labeled ENERGY STAR. Despite the
performance of all products in a given ENERGY STAR category, only those
products produced by manufacturers that choose to join the ENERGY STAR
program may advertise their products with the ENERGY STAR label.
If our sample results are representative of the universe of ENERGY STAR
products, these results call into question the ability of the program to identify
products with superior energy efficiency. Because manufacturer participation in
the program is voluntary, the ENERGY STAR designation does not necessarily
identify the best performing products in the marketplace. Additionally, a high
rate of compliance by non-ENERGY STAR products will affect the EPA's
method of computing energy savings. EPA could improve the integrity of the
program with enhanced product testing and adjustments to the greenhouse gas
benefits calculation.
Recommendations
We recommend that the Assistant Administrator for Air and Radiation:
2-1 Verify estimated energy savings and greenhouse gas reduction
calculations using a market-based performance testing program that
includes testing non-ENERGY STAR products.
2-2 Revise the ENERGY STAR Website to include the established standard
alongside qualifying product performance data and to provide a summary
listing of the highest performers.
Agency Comments and OIG Evaluation
The Agency concurred with our recommendations and agreed to implement them.
However, the Agency disagreed with the report's conclusions. The Agency did
not concur with conclusions drawn based on a select number of non-ENERGY
STAR products performing comparably to or better than the ENERGY STAR
products. The Agency concluded that the value of the ENERGY STAR label is
ultimately in the assurance the label provides consumers that the ENERGY STAR
product they purchase will consistently save them energy. We disagree with this
conclusion and maintain our concerns about the integrity of the ENERGY STAR
10
-------
10-P-0040
program as well as the associated energy and cost savings reported to consumers.
The Agency's complete response is included in Appendix A.
11
-------
10-P-0040
Status of Recommendations and
Potential Monetary Benefits
RECOMMENDATIONS
POTENTIAL MONETARY
BENEFITS (In $000s)
Rec.
No.
Page
No.
Subject
Status1
Action Official
Planned
Completion
Date
Claimed
Amount
Agreed To
Amount
2-1 10 Verify estimated energy savings and greenhouse
gas reduction calculations using a market-based
performance testing program that includes testing
non-ENERGY STAR products.
2-2 10 Revise the ENERGY STAR Website to include the
established standard alongside qualifying product
performance data and to provide a summary listing
of the highest performers.
Assistant Administrator for
Air and Radiation
Assistant Administrator for
Air and Radiation
1 0 = recommendation is open with agreed-to corrective actions pending
C = recommendation is closed with all agreed-to actions completed
U = recommendation is undecided with resolution efforts in progress
12
-------
10-P-0040
Appendix A
Agency Comments on Draft Report
(Received on November 6, 2009)
MEMORANDUM
Subject: Comments on the Draft Evaluation Report: ENERGY STAR
Program Integrity Can Be Enhanced Through Expanded Product Testing
From: Elizabeth Craig
Deputy Assistant Administrator
To: Jeffrey Harris, Director
Cross-Media Issues, Office of Program Evaluation
Thank you for the opportunity to comment on the Draft Evaluation Report: ENERGY STAR
Program Integrity Can Be Enhanced Through Expanded Product Testing. We are pleased to see
that testing done by the Inspector General's Office indicates that the ENERGY STAR qualified
products tested are at least as efficient as advertised and no issues with the manufacturer self-
certification process were indicated.
While we appreciate and share your interest in protecting the integrity of the ENERGY STAR
Program, we do not concur with the conclusions drawn based on a select number of non-
ENERGY STAR qualified products performing comparably to or better than the ENERGY
STAR products. The ENERGY STAR label's image as a trusted symbol for environmental
protection through superior efficiency is ultimately affected by the performance of products
bearing the label, not by non-participating products. In fact, based on the compliance rates the
OIG found, the consumer choosing a product with the ENERGY STAR label receives the energy
efficiency they expect; choosing a non ENERGY STAR product would provide the consumer
with a range of possible outcomes as the non-ENERGY STAR products may or may not be
efficient.
Although we disagree with the report's conclusions, we recognize potential benefit associated
with the recommendations it makes and would be pleased to implement them.
13
-------
10-P-0040
OAR Response to IG Recommendations in Draft Report and Status of Implementation
Recommendation 2-1: Verify estimated energy savings and greenhouse gas reduction
calculations using a market-based performance-testing program that includes testing non-
ENERGY STAR products
OAR Response: As part of the Obama Administration's commitment to enhancing the
ENERGY STAR Program (as articulated in a new MOU signed by EPA and DOE on
September 30, 2009), we plan to institute new ENERGY STAR qualification testing
requirements that will leverage market-based mechanisms to broaden and improve the
performance test data available to us for use in verifying estimated energy savings and
greenhouse gas reduction calculations. In conjunction with this effort, we will be
working with the Department of Energy to expand verification testing of ENERGY
STAR qualified products, with the intent being to test both qualified and non-qualified
products.
Recommendation 2-2: Revise the ENERGY STAR Website to include the established standard
along side qualifying product performance data and to provide a summary listing of the highest
performers.
OAR Response: We agree that it may be useful to provide interested consumers a web-
based means of comparing and ranking qualifying product performance against
established ENERGY STAR standards. We are currently exploring options for
effectively addressing this while minimizing added administrative burden.
cc: Brian McLean
Ann Bailey
David LaRouche
14
-------
10-P-0040
Appendix B
Distribution
Office of the Administrator
Assistant Administrator, Office of Air and Radiation
Deputy Assistant Administrator, Office of Air and Radiation
Agency Follow-up Official (the CFO)
Agency Follow-up Coordinator
General Counsel
Associate Administrator for Congressional and Intergovernmental Relations
Associate Administrator for Public Affairs
Audit Follow-up Coordinator, Office of Air and Radiation
Acting Inspector General
15
------- |