I
3
U.S. ENVIRONMENTAL PROTECTION AGENCY
OFFICE OF INSPECTOR GENERAL
Catalyst for Improving the Environment
Evaluation Report
Improvements Needed to Validate
Reported ENERGY STAR Benefits
Report No. 09-P-0061
December 17, 2008
-------
Report Contributors:
Laurie Adams
Natasha Besch-Turner
Daniel Carroll
Jerri Dorsey
Jeffrey Harris
Abbreviations
CCAP Climate Change Action Plan
CFLs Compact Fluorescent Lights
DOE Department of Energy
DSM demand-side management
EPA U.S. Environmental Protection Agency
kWh kilowatt hours
MMTCE million metric tons of carbon equivalent
OIG Office of Inspector General
PBC public benefits charge
Cover art: The illustrations depict the four primary components of EPA's
ENERGY STAR program, from left: products, commercial,
industrial, and residential (from EPA).
-------
PRtf
U.S. Environmental Protection Agency
Office of Inspector General
At a Glance
09-P-0061
December 17, 2008
Catalyst for Improving the Environment
Why We Did This Review
We initiated this review to
examine the validity and
accuracy of the reported
energy savings for the U.S.
Environmental Protection
Agency's (EPA's) ENERGY
STAR program. This was part
of our agenda to assess new
approaches to environmental
protection. We specifically
sought to determine whether
the savings reported were
valid and fully supportable.
Background
ENERGY STAR is a
voluntary program designed to
help businesses and
individuals enhance their
energy efficiency. In 2006,
the ENERGY STAR
program reported avoiding a
total of 37.6 million metric
tons of carbon equivalent. It
further reported that ENERGY
STAR helped prevent
greenhouse gas emissions
equivalent to those from
25 million vehicles while
savings Americans $14 billion
on their energy bills.
Improvements Needed to Validate Reported
ENERGY STAR Benefits
For further information,
contact our Office of
Congressional, Public Affairs,
and Management at
(202)566-2391.
To view the full report,
click on the following link:
www.epa.gov/oig/reports/2009/
20081217-09-P-0061.pdf
/2009/
What We Found
Reported ENERGY STAR benefits represented one-half of EPA's total
greenhouse gas emissions avoided in 2006. ENERGY STAR benefits are a major
component of efforts reducing such emissions. The accuracy of the program's
reported energy savings is important in monitoring the United States' efforts to
reduce greenhouse gas emissions.
We found the ENERGY STAR program's reported savings claims were inaccurate
and the reported annual savings unreliable. We identified several deficiencies
with the shipment data and the process used in calculating benefits. Deficiencies
included the lack of a quality review of the data collected; reliance on estimates,
forecasting, and unverified third party reporting; and the potential inclusion of
exported items. Also, EPA included savings for one Department of Energy (DOE)
product that DOE also claimed.
Additionally, sales of formerly qualified products are used to determine ENERGY
STAR's market transformation benefits, but we found that this benefit was
computed inconsistently. Also, the methodology used to compute the ENERGY
STAR commercial sector benefits uses unverified assumptions.
What We Recommend
We recommended that EPA:
Establish and implement improved quality controls.
Develop and consistently apply a data-driven methodology to compute
market transformation effects.
Validate the model for calculating the benefits of the ENERGY STAR
commercial sector to ensure it accurately reflects the sector's impacts.
EPA disagreed with many of our conclusions, but stated it had implemented some
of the recommendations. However, some of EPA's planned actions do not meet
the intent of our recommendations, and we consider these recommendations open
and unresolved.
-------
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
OFFICE OF
INSPECTOR GENERAL
MEMORANDUM
SUBJECT:
FROM:
TO:
December 17, 2008
Improvements Needed to Validate Reported ENERGY STAR Benefits
Report No. 09-P-0061
Wade T. Najjum
Assistant Inspector General for Program Evaluation
Robert J. Meyers
Principal Deputy Assistant Administrator
Office of Air and Radiation
This is our report on the ENERGY STAR benefits evaluation conducted by the Office of
Inspector General (OIG) of the U.S. Environmental Protection Agency (EPA). This report
contains findings that describe the problems the OIG has identified and corrective actions the
OIG recommends. This report represents the opinion of the OIG and does not necessarily
represent the final EPA position. Final determinations on matters in this report will be made by
EPA managers in accordance with established audit resolution procedures.
The estimated cost of this report - calculated by multiplying the project's staff days by the
applicable daily full cost billing rates in effect at the time - is $538,867.
Action Required
In accordance with EPA Manual 2750, you are required to provide a written response to this
report within 90 calendar days. You should include a corrective actions plan for agreed upon
actions, including milestone dates. We have no objections to the further release of this report to
the public. This report will be available at http://www.epa.gov/oig.
If you or your staff has any questions regarding this report, please contact me at 202-566-0827;
or Jeffrey Harris, Director of Cross Media, at 202-566-0831 or harris.jeffrey@epa.gov.
-------
Improvements Needed to Validate 09-P-0061
Reported ENERGY STAR Benefits
Table of Contents
Chapters
1 Introduction 1
Purpose 1
Background 1
Noteworthy Achievements 3
Scope and Methodology 4
Prior Review 5
2 Inaccurate and Unsupported Data Used to Calculate
2006 ENERGY STAR Product Annual Benefits 6
Process Used to Calculate Product Sector Savings 6
Shipment Totals Used in Savings Calculations Are Unreliable 8
Other Factors Contribute to Uncertainties 10
DOE Product Included in EPA ENERGY STAR Reported Savings 12
Conclusion 13
Recommendation 13
Agency Comments and OIG Evaluation 14
3 ENERGY STAR Market Transformation Benefits
Calculated Inconsistently 15
Market Transformation Caused by Innovations in Efficiency 15
Market Transformation Benefits Calculated with Assumed Data 16
Market Transformation Obscures Program Accomplishments 17
Conclusion 18
Recommendation 18
Agency Comments and OIG Evaluation 18
4 ENERGY STAR Commercial Benefits Model Utilizes
Unverified Assumptions 20
Formula Used to Calculate Commercial Sector Savings 20
Methodology Not Tailored for ENERGY STAR 20
Formula Methodologies and Assumptions Need to Be Verified 23
Conclusion 23
Recommendation 24
Agency Comments and OIG Evaluation 24
Status of Recommendations and Potential Monetary Benefits 25
- continued -
-------
Improvements Needed to Validate 09-P-0061
Reported ENERGY STAR Benefits
Appendices
A Logic Model for ENERGY STAR Products 26
B Logic Model for ENERGY STAR Commercial & Industrial Programs 27
C Details on Scope and Methodology 28
D Agency Comments and OIG Evaluation 30
E Distribution 43
-------
09-P-0061
Chapter 1
Introduction
Purpose
We initiated this review to examine the validity and accuracy of the reported
energy savings for the U.S. Environmental Protection Agency's (EPA's)
ENERGY STAR program. This was part of our agenda to assess new approaches
to environmental protection. We specifically sought to determine whether the
savings reported in 2006 were valid and fully supportable.
Background
In 1990, Section 103(g) of the Clean Air Act directed the EPA Administrator to
"conduct a basic engineering research and technology program to develop,
evaluate, and demonstrate non regulatory strategies and technologies for air
pollution prevention." In 1992, EPA's Office of Air and Radiation established the
ENERGY STAR Product Labeling Program (the ENERGY STAR program) as an
innovative approach to environmental protection. Congress formally authorized
the ENERGY STAR program in the Energy Policy Act of 2005.l
ENERGY STAR is a voluntary program designed to help businesses and
individuals protect the environment through superior energy efficiency. The
ENERGY STAR program was designed to overcome selected market barriers.
The program was first introduced to recognize and promote energy-efficient
computers. It has since grown to cover many additional consumer products and
services. In 1996, EPA partnered with the Department of Energy (DOE) to
promote the ENERGY STAR label and broaden the range of products covered.2
The ENERGY STAR program is one part of a larger U.S. government-wide
agenda to address climate change. In 2002, the President announced a goal of
reducing America's greenhouse gas intensity 18 percent by 2012. EPA's
programs are expected to contribute over 70 percent of the emissions reductions
needed to meet the President's greenhouse gas intensity goal. ENERGY STAR is
the most significant of the EPA's greenhouse gas avoidance programs. In 2006,
the program accounted for over 50 percent of EPA's contribution. The success of
ENERGY STAR is therefore central to whether the United States will meet the
President's goal for reducing greenhouse gas intensity.
1 Public Law 109-58-Aug. 8, 2005; Subtitle C-Energy Efficient Products, Sec. 131. ENERGY STAR program.
2 A Memorandum of Cooperation was signed jointly on May 29, 1996.
-------
09-P-0061
The ENERGY STAR program reported avoiding a total of 37.6 million metric
tons of carbon equivalent (MMTCE) in 2006. According to the EPA, the program
helped prevent greenhouse gas emissions equivalent to those from 25 million
vehicles while saving Americans $14 billion on their energy bills. ENERGY
STAR has four program sectors: products, commercial, industrial, and residential.
The 2006 reported carbon emissions avoided by program sector are illustrated in
Figure 1-1; the paragraphs that follow describe the four sectors.
Figure 1-1: Carbon Emissions Avoided by Sector for 2006
Industrial 17.35
Residential
1.1 %
Source: OIG Analysis of ENERGY STAR 2006 Annual Report
Products
According to ENERGY STAR staff the product sector is designed to promote
energy-efficient products for purchase by consumers. This sector is the original
component of EPA's ENERGY STAR program and is still the most recognizable
part of the program.3 EPA reports that approximately 1,700 manufacturers are
using the ENERGY STAR label on over 40,000 product models across more than
50 product categories. Annually, consumers are reportedly purchasing
approximately 300 million ENERGY STAR-qualified products and have invested
in over 2 billion products since 1992. ENERGY STAR reported within its 2006
annual report that the product sector was responsible for 15.5 MMTCE emissions
avoided.
A logic model illustrating how the ENERGY STAR program is designed to promote energy efficiency among
consumer products is depicted in Appendix A.
-------
09-P-0061
Commercial and Industrial Sectors
The commercial and industrial sectors are designed to promote superior corporate
energy management approaches and provide partners guidance on assessing
current energy use and developing plans that will lead to energy reductions and
overall energy efficiency. Annually, commercial and industrial buildings use
approximately $200 billion worth of electricity and natural gas, which results in
nearly half of the U.S. greenhouse gas emissions. ENERGY STAR reported
within the 2006 annual report that the commercial and industrial sectors avoided
combined total emissions of 21.7 MMTCE.4 EPA reported that increasing energy
prices and greater concern for the environment and global warming have recently
increased interest and participation in ENERGY STAR'S commercial and
industrial program efforts.
Residential Sector
The residential sector is designed to help make residential homes more energy
efficient. This sector is composed of two primary components: home
improvements and new homes. ENERGY STAR reported that households can
reduce their energy use up to 30 percent and save $600 annually on utility bills by
using qualified products and services. In the 2006 annual report, EPA reported
that ENERGY STAR'S residential sector avoided total emissions of 0.4 MMTCE.
Noteworthy Achievements
According to ENERGY STAR, the program has grown steadily in terms of the
energy efficient solutions it offers, the variety of partners, and the benefits it
delivers. EPA has been successful in marketing the ENERGY STAR brand and
label. In 2007, EPA reported that 74 percent of households nationwide recognize
the ENERGY STAR label and 62 percent associated the label with "energy
efficiency or energy savings."
Since 2000, the ENERGY STAR program has consistently outperformed its
annual goals, as illustrated in Figure 1-2.
4 A logic model illustrating how the ENERGY STAR program is designed to promote energy efficiency among
commercial buildings is depicted in Appendix B.
-------
09-P-0061
Figure 1-2: ENERGY STAR Goals Are Surpassed
0 oU -
T3
1
^ in
5.
n -
^=
2000 2001
2002
D Goal
2003 2004
Year
D Reported
2005 2006
Source: ENERGY STAR and Other Climate Protection Partnership Annual Reports,
2000 through 2006
The ENERGY STAR program's overall reported achievements in 2006 exceeded
the program's goals as illustrated in Table 1-1.
Table 1 -1: ENERGY STAR 2006 Goals and Achievements by Program
ENERGY STAR Program
Product Labeling
Residential
Commercial Buildings
Industrial
Total
2006 Goal
(MMTCE Avoided)
14.5
0.5
11.5
3.7
30.2
2006 Achievements
(MMTCE Avoided)
15.5
0.4
15.2
6.5
37.6
Source: Based on data presented in the ENERGY STAR and Other Climate Protection
Partnerships 2006 Annual Report
Scope and Methodology
We conducted this performance evaluation in accordance with generally accepted
government auditing standards. Those standards require that we plan and perform
the evaluation to obtain sufficient, appropriate evidence to provide a reasonable
basis for our findings and conclusions based on our objectives. We believe that
the evidence obtained provides a reasonable basis for our findings and
conclusions based upon our objectives. We performed our field work from
January through September 2008.
Our review included an examination of applicable laws and regulations as well as
Agency guidance. We reviewed those internal controls that were relevant to our
objectives. We reviewed ENERGY STAR annual reports, and Agency guidance
-------
09-P-0061
documents. We reviewed planning documents, including logic models. We met
with Agency staff and contractors. We reviewed the methodologies governing the
savings benefit calculations for the commercial and products program sectors.5
Additionally, for the products sector, we reviewed documentation in support of
the 2006 reported savings benefits.
Appendix C provides further details on our scope and methodology.
Prior Review
An EPA Office of Inspector General (OIG) report, ENERGY STAR Program Can
Strengthen Controls Protecting the Integrity of the Label, reported that
improvements should be made to better assure the integrity of the ENERGY
STAR label for home and office products. Criteria for revising specifications
were unclear, and EPA did not have reasonable assurance that the self-
certification process was effective. EPA relied on some alternative verification
mechanisms, but lacked any quality assurance or review of these reported results.
EPA did not agree with all of the conclusions, but agreed to implement the OIG
recommendations to strengthen management controls to protect the integrity of
the ENERGY STAR label.
5 The residential and industrial program sectors were excluded from our review. The commercial and products
sector combined encompassed 81.6 percent of the 2006 reported carbon emissions avoided.
6
EPA OIG Report No. 2007-P-00028, issued August 1, 2007, at www.epa.gov/oig/reports/2007/20070801-2007-P-
00028.pdf.
-------
09-P-0061
Chapter 2
Inaccurate and Unsupported Data Used to Calculate
2006 ENERGY STAR Product Annual Benefits
The ENERGY STAR program reported savings for 2006 that were inaccurate or
unsupported. A key component of the program's annual and lifetime savings
calculation process for consumer products is the total ENERGY STAR shipments
for that year. We identified several concerns with shipment data, including: lack
of quality review of the data submittals; reliance on estimates, forecasting, and
third party reporting; and the potential inclusion of international shipments with
domestic ENERGY STAR product shipments. When annual shipment data are not
supportable or reliable, the validity of the savings calculation process is in
question.
Process Used to Calculate Product Sector Savings
The product sector's annual savings are computed using the Climate Change
Action Plan (CCAP) model. This model is used to compute the energy savings for
each qualified EPA ENERGY STAR product. A key number in the computation
is the annual ENERGY STAR-qualified shipment total by product category.7
For the majority of the EPA product categories (29 of 37), the manufacturers
annually submit shipment data in one of two ways. They either submit data
directly via unit shipment data submittal forms or indirectly through third party
associations. One EPA contractor has sole responsibility for the receipt of these
data and the production of the annual summary report. This report includes the
overall numbers of qualified ENERGY STAR units shipped by product category,
which are used in the annual savings calculation process. Figure 2-1 depicts the
process. Figure 2-2 illustrates how the annual shipments per qualified ENERGY
STAR product category are multiplied by the unit energy savings8 to derive the
annual energy savings attributable to the program.
7 The ENERGY STAR program uses the term shipments interchangeably with sales. When a qualified ENERGY
STAR manufacturer ships its ENERGY STAR-qualified product, the shipment is considered sold for annual savings
calculation purposes. For the reporting period reviewed, EPA did not require manufacturers to submit annual
shipment data for office products or for programmable thermostats. Office products consist of: computers, copiers,
fax machines, mailing machines, multifunction devices, printers, and scanners.
8 Unit energy savings for each product type is the difference between the business-as-usual unit energy consumption
and the ENERGY STAR unit energy consumption in a given year. Unit energy savings change over time for most
product types due to specification revisions, usage pattern changes, and changes to the business-as-usual efficiency.
-------
09-P-0061
Figure 2-1: ENERGY STAR Products Annual Savings Calculation Process
Partners Report
Shipment Data to EPA*
Energy savings from
existing program
stock + (2006
Product Shipments
due to program x Unit
Energy Consumption
- 2006 Adjustments)
(CCAP model results
as supplied by
contractor are
modified by
adjustment factors)
T
Final 2006 Results
75.2 billion kWh
58.5 trillion British
thermal units natural
gas and other fuels
1. Data incorporated
into Annual Report
2. Summary Memos
written
EPA prepares
adjustments to reflect
understanding of
third party promotion
Legend
Pink - Contractor/Partner
Actions
Blue - EPA Actions
Orange - Methodology
Formula
Yellow - Final 2006 Results
Green - Post-Analysis
Actions
*Manufacturer submits data directly to EPA contractor or via third party associations
Source: EPA ENERGY STAR Immediate Office
Figure 2-2: ENERGY STAR Products Annual Savings Calculation
Current Year
ENERGY STAR
Qualified Shipments
Current Year
Unit Energy Savings
Savings from
Products Sold in
Previous Years
Source: OIG based on ENERGY STAR program materials
The total annual saving calculation also includes a portion of savings computed
for products shipped in previous years but are assumed to still be in use. EPA
continues to claim savings for shipped qualified products over the estimated
lifetime of the product. The lengths for which the lifetime annual savings benefits
are calculated vary from 4 to 20 years after initial product shipment.
Additionally, the total annual savings calculation includes an estimate of benefits
from sales of products that previously met ENERGY STAR specifications but no
longer qualify (further details are in Chapter 3).
-------
09-P-0061
Shipment Totals Used in Savings Calculations Are Unreliable
2006 Annual Shipment Data are Irreconcilable
We found that the annual shipment data used for the 2006 savings calculations
were unsupportable. Based on our review of the documents provided, we were
unable to reconcile the original shipment data with final reported totals for 18 of
the 29 product categories. These reported totals are used to calculate ENERGY
STAR program savings. Shipment totals for 13 product categories were
understated and 5 were overstated. The results of our analysis are in Table 2-1.
Table 2-1: OIG Reconciliation of Reported Product Shipment Totals for
18 of 29 ENERGY STAR Categories
ENERGY STAR
Qualified Product
Audio/DVD
Boilers
Commercial Hot
Food Holding
Cabinets
Commercial
Refrigerators &
Freezers
Dehumidifiers
End Use Products
with Qualified
External Power
Supplies
Exit Signs
External Power
Supplies
Furnaces
Monitors
Residential Light
Fixtures
Roofing -
Commercial Gallons
Roofing -
Commercial &
Residential Sq. Ft.
Room Air Cleaners
Telephony
Transformers
(Commercial &
Industrial)
TV/VCR
Ventilating Fans
2006 Unit Shipments
Used in Benefits
Calculation9
3,572,316
382,770
17,059
83,493
1,203,216
18,446,331
1 ,425,554
128,849,173
1,172,843
13,859,666
11,361,701
21,352,516
1,916,655,032
287,354
6,761,627
173,390
17,685,967
853,406
2006 Unit Shipment
Totals Reported by
Manufacturers and
Associations10
3,881,346
405,573
17,403
106,658
776,431
19,623,346
3,450,117
128,939,237
1,170,124
34,581,203
11,363,633
15,966,045
2,227,937,102
289,815
7,006,979
166,669
13,166,140
857,232
Difference
-309,030
-22,803
-344
-23,165
+426,785
-1,177,015
-2,024,563
-90,064
+2,719
-20,721,537
-1,932
+5,386,471
-311,282,070
-2,461
-245,352
+6,721
+4,519,827
-3,826
Impact
Understated
Understated
Understated
Understated
Overstated
Understated
Understated
Understated
Overstated
Understated
Understated
Overstated
Understated
Understated
Understated
Overstated
Overstated
Understated
Source: OIG analysis of shipment data submittals
9 Per ENERGY STAR Unit Shipment Data Report Calendar Year 2006, July 1, 2007. These figures are reported by
the contractor and used in the annual savings calculation process.
10 Total compiled by OIG based on supporting documents (manufacturer data shipment submittal forms and
third party association summary spreadsheets) provided by contractor.
-------
09-P-0061
According to EPA's contractor, the differences may have been caused by a
number of reasons, including:
Subsequent to shipment data submittals, manufacturers may have called to
update or correct previously provided written submissions.
The contractor may have called the manufacturer to follow up concerning
the data submittal and new information received.
Some private labelers11 and original manufacturers submit the shipment
data for the same products, which causes double reporting of shipments.
Some manufacturers who reported shipments were not qualified
manufacturers for that product category.
Manufacturers made errors in reporting.
However, the contractor could not provide evidence in support of these possible
explanations.
If the shipment data used in the annual savings calculation are incorrect or
unsupportable, the Agency may be overstating or understating the savings
benefits for the program. For example, in our reconciliation of the 2006 shipment
data, the monitor category was understated by nearly 21 million unit shipments or
150 percent. Monitors are one of the six ENERGY STAR products that account
for 70 percent of the ENERGY STAR product sector carbon reductions to date.
TV/VCR, another one of the six large contributors to carbon reductions, was
overstated by approximately 4.5 million unit shipments or 26 percent.
ENERGY STAR overall program benefit are the sum of the savings calculated for
each individual product. The importance of accurate data is discussed within a
recently published article12 that summarizes the methodology used to compute the
annual ENERGY STAR product program sector benefits. The article stated:
We implement the bottom-up model with awareness that uncertainty
for each product type contributes to uncertainty in total ENERGY
STAR impacts. This means that many small inaccuracies are
additive overall and any one inaccuracy for a product type with
large energy savings can significantly affect the overall results.
We could not reconcile shipment data in 2006 for 62 percent of the product
categories reviewed. The significance of this deficiency extends beyond a single
year's benefits calculation. The deficiency impacts the savings calculations for
future years as well because savings are computed and claimed annually over the
11 Private labeling is defined as licensing a product to another company to sell under its own name, rather than under
the name of the manufacturer.
12
Maria C. Sanchez, Richard E. Brown, Carrie Webber, and Gregory K. Homan. 2008. Savings estimates for the
United States Environmental Protection Agency's ENERGY STAR voluntary product labeling program. Energy
Policy, vol. 36, no. 6, pp. 2098-2108.
-------
09-P-0061
lifetime of a product. As previously discussed, the lifetime of calculated annual
benefits can reach up to 20 years after initial product shipment.
Lack of Quality Review of Shipment Data
The reported shipment data submitted annually by manufacturers were not being
adequately reviewed. According to ENERGY STAR'S guidance document,13 the
ENERGY STAR Product Labeling branch collects national shipment data from
ENERGY STAR partner manufacturers and reconciles inconsistencies through
the contractor. As described by the contractor, the contractor performs a limited
review that involves reviewing the information to see if the data appears
reasonable. EPA officials have not requested the supporting documentation;14
therefore we found no evidence that EPA conducts any quality reviews of
shipment data submittals. If a reconciliation of shipment data had occurred, the
discrepancies we identified in our analysis could have been noted and addressed
in a timely manner.
Number of Manufacturers Submitting Data May be Wrong
We were unable to reconcile the number of manufacturers who submitted their
annual data to the number reported. In some cases, we found fewer
manufacturers submitting data than the contractor reported. In other cases, more
manufacturers reported than were qualified. Additionally, we noted some
manufacturers reporting annual ENERGY STAR product shipments that were not
listed as a manufacturer of ENERGY STAR-qualified products. For these, there
was no evidence of any follow-up to address why this occurred.
Other Factors Contribute to Uncertainties
Third Party Association Reporting Inconsistent and Unsupported
Reporting from third party associations was inconsistent and primarily included
only summary totals by manufacturer. For most of the associations reporting
shipment data, there was no way to verify the numbers based on any support or
actual shipment data documentation, but we identified some problems. For
example, there were cases where the manufacturer reported directly to both the
contractor and the association, which resulted in shipment number differences.
There was no evidence that the contractor questioned the differences or conducted
any follow-up. The contractor acknowledged that the third party association
reporting is inconsistent, lacks uniformity, and data has been submitted past
agreed due dates. According to the contractor, in instances when there is a
13OAR (Office of Air and Radiation) Climate Protection Partnerships Division's ENERGY STAR Performance
Management Plan.
14 The contractor had not been maintaining the supporting documentation for previous shipment totals, and the
contractor is not required to do so by the contract. However, the contractor recently began retaining supporting
documentation for reported overall numbers.
10
-------
09-P-0061
reporting difference, it has been its general practice to use the shipment totals
reported by the third party association rather than the manufacturer. We could not
verify from documentation provided whether these numbers were double counted.
Use of Estimates and Forecasting Increases Uncertainty
Selected manufacturer shipment data submittal forms provided shipment
estimates rather than actual shipment totals. We found no documentation of any
follow-up with these manufacturers as to why estimates were used, nor did the
contractor mention this as a step taken.
EPA did not require manufacturers to submit annual shipment data for office
products. For the 2006 annual reported savings, forecasting reports were used to
estimate shipment totals for some office products. We reviewed the two primary
forecasting reports used. We noted that one report used for this forecasting was
dated 2001. The other report did not clearly indicate how domestic shipments
were separated from international shipments. Concerns associated with using a
dated forecasting report may be corrected due to recent office product
specification revisions.15 Also, beginning in 2008, manufacturers were required
to submit shipment data annually for office products.
Annual programmable thermostat shipments were estimated by the contractor
based on past conversations with one U.S.-based manufacturer. Currently there
are over 50 qualified manufacturers in this category, including many companies
outside of the United States. Further, the ENERGY STAR Partner Commitment
Agreement for Programmable Thermostats16 specifically requires that
manufacturers submit the total number of qualified products shipped annually.
We could not verify or reconcile the accuracy of support for benefits from this
item since no actual shipment data were submitted, despite EPA's requirements.
ENERGY STAR Totals May Include International Shipments
According to ENERGY STAR staff, shipment totals used to calculate energy
savings are supposed to be limited to U.S. shipments. However, ENERGY
STAR's management plan states that as part of the partnership agreement,
manufacturers must supply EPA with the "total number of ENERGY STAR
qualified products shipped in the United States / Canada on annual basis." We
noted discrepancies in manufacturer shipment submittal forms. For example, on
one submittal the manufacturer was identified as Company X, Canada, but had
been changed to Company X, USA, within the final contractor's reports. Also, on
selected forms, the manufacturer said it could provide total international
shipments with only an estimate of U.S. shipments. EPA's contractor said that it
15 In 2007, new specifications were made to the following office products: computers, copiers, facsimile, mailing
machines, multifunction devices, printers, and scanners.
16
To participate in the ENERGY STAR program and become eligible to become a partner, each manufacturer must
agree to and adhere to this commitment.
11
-------
09-P-0061
was unaware of domestic and international shipments being commingled, but
noted it has not done any detailed work in this area. We believe that being
unaware of whether international shipments are being included in calculations is
not a reason to conclude they are not.
Late Shipment Data Not Included in Benefits Calculations
We found that approximately 33 million units reported as shipped were not
included in the shipment totals used to calculate saving benefits for 2006. Per the
Performance Management Plan, manufacturers are to provide shipment data by
March 1 of the subsequent year (i.e., March 2007 for 2006 shipments). EPA's
contractor then compiles this information and prepares a July report of ENERGY
STAR shipments to be used in the savings benefit calculations. The contractor
prepares an updated report in October that includes shipment data submitted after
the deadline. However, due to late submittals, we noted 33 million units captured
in the October 2007 report that had not been captured in the July 2007 report.
Therefore, savings benefits from these products were not captured in the 2006
reporting period.
According to the EPA Office of Air and Radiation's Director of the Climate
Protection Partnership Division, these shipments were also not included in the
2007 calculations because the amount was considered immaterial. According to
EPA's contractor, the late shipment numbers were folded into the 2007
calculations. However, the contractor was unable to provide evidence that this
had occurred. Therefore, we concluded that the late shipments are not being
captured by the current methodology. The lack of a methodology to capture
potential savings benefits for late submittals reduces the reliability of ENERGY
STAR reporting.
DOE Product Included in EPA ENERGY STAR Reported Savings
EPA reports a portion of savings benefits from the shipments of compact
fluorescent lights (CFLs) even though CFLs are managed by DOE. EPA's 2006
ENERGY STAR Annual Report claimed 75.2 billion kilowatt hours (kWh) in
energy savings. However, 5.7 billion kWh of those savings, or 8 percent, came
from the DOE-managed CFLs. EPA did not disclose in its Annual Report that
savings benefits reported include benefits from DOE-managed products.
Also, we found differences in the total benefits claimed for CFLs by EPA and
DOE. DOE reported 6.2 billion kWh in energy savings versus EPA's 7.6 billion
kWh for 2006. Rather than obtain total savings benefits directly from DOE, EPA
calculated the CFLs energy savings benefits in the same manner as other EPA
product savings (this process is described earlier in this chapter). This difference
in energy savings between DOE and EPA for the same product questions the
reliability of the computing and reporting of energy savings.
12
-------
09-P-0061
EPA ENERGY STAR staff said they claim one-third of CFL energy savings
because of EPA's marketing efforts. They also said they were involved in the
specification development for CFLs. For these reasons, they claim a portion of
the CFLs savings benefits. However, DOE ENERGY STAR staff were not aware
that EPA claimed this percentage of CFL savings, and had claimed 100 percent of
their calculated CFL savings in 2006. This resulted in double-counting of CFL
benefits by EPA and DOE. A DOE ENERGY STAR representative said that
while EPA does contribute to the success of CFLs by EPA's marketing efforts,
EPA should coordinate with DOE regarding the calculation and reporting.
Conclusion
We found that the reported savings claims were inaccurate and the reported
annual savings unreliable. We identified several concerns with shipment data,
including: lack of quality review of the data submittals; reliance on estimates,
forecasting, and third party reporting; and potential inclusion of international
shipments with domestic ENERGY STAR product shipments. After attempting
to reconcile the support for these numbers by product category, we found the
overall claims to be inaccurate due to irreconcilable manufacturer shipment data
and use of estimates. We conclude that the savings reported are not accurate or
verifiable. The accuracy of the program's reported energy savings is important in
the United States' efforts to reduce greenhouse gas emissions.
Recommendation
2-1 To improve the validity of reported annual savings for the ENERGY
STAR program, we recommend that the Principal Deputy Assistant
Administrator for the Office of Air and Radiation establish and perform
quality controls to ensure that:
Data in benefits calculations, whether from partners or third
parties, are timely, complete, valid, and documented.
The contractor and third party associations receiving the
manufacturer data submittal forms reconcile submittals in a
manner that ensures the total annual shipments reported by product
category are accurate and reflect actual numbers (not estimates)
and are for domestic shipments only.
Agency officials improve contractor oversight by obtaining actual
support for annual savings in a manner that demonstrates that the
numbers are valid and can be reconciled.
Data in benefits calculations attributable to DOE products should
be clearly identified and developed in consultation with DOE to
avoid redundancy.
13
-------
09-P-0061
Agency Comments and OIG Evaluation
EPA did not agree to the portions of Recommendation 2-1 that related to
improvements needed to ensure data used in benefits calculations are timely,
complete, valid, and documented. The Agency requested the OIG to delete the
related portions of the recommendation. EPA said it addresses the quality of
annual shipment numbers used for ENERGY STAR benefits calculations through
a systematic review and improvement of the data submitted based on additional
follow-up with reporting manufacturers. However, based on our findings in this
area, we do not agree that the Agency currently has a systematic review process in
place. The OIG evaluated the existing process and the findings show that the
process used to collect and verify the data was deficient and needs improvement.
OIG identified deficiencies with the process of collecting, reviewing, and
reporting product data shipment information both from third parties or actual
manufacturers. EPA also stated that estimates were not used as part of estimating
annual program benefits. However, at the time of our review, estimates of
shipment totals were used for both office products and programmable thermostats
in calculating annual benefits.
EPA provided corrective actions to address the portions of the recommendation
that relate to improving contractor oversight and reporting savings from DOE
products. Specifically, EPA agreed to remove the benefits of CFL products from
the program's 2007 benefits estimates and will only reintroduce these benefits
after appropriate coordination with DOE. The OIG concurs with the Agency's
plan to address the reporting of CFLs. However, the Agency's corrective actions
for contractor oversight do not fully meet the intent of the recommendation. The
OIG identified deficiencies with the process of collecting, reviewing, and
reporting product data shipment information. The Agency's corrective action
plan will need to address these shortcomings.
The recommendation remains open and unresolved. The Agency's comments and
our evaluation of those comments are in Appendix D.
14
-------
09-P-0061
Chapter 3
ENERGY STAR Market Transformation Benefits
Calculated Inconsistently
The reporting of ENERGY STAR market transformation benefits is not
transparent. ENERGY STAR does not disclose that it includes savings from
products that are not ENERGY STAR qualified. Further, the process does not
compute the effect for products with stable or rising sales after a specification
revision. EPA's market transformation benefits are generated when innovations
are introduced into the marketplace and are accepted by a large portion of the
market. However, EPA calculates the market transformation effect only when
ENERGY STAR product shipments fall after a specification revision.
Market Transformation Caused by Innovations in Efficiency
The ENERGY STAR programs attempt to transform the markets (i.e,. consumer
electronics, office equipment, lighting, etc.) from less to more energy-efficient
products. Market transformation is the process whereby these efficiencies are
introduced into the marketplace and are increasingly accepted by a large portion
of the market. According to EPA, the ENERGY STAR program produces such
an effect on the market. ENERGY STAR staff asserted that the program has two
primary mechanisms to spur market transformation for labeled products:
Introduce standards for new product categories.
Revise specifications for existing products to further increase their energy
efficiency.
According to EPA, the program has revised specifications for over 50 ENERGY
STAR product categories. After a specification is revised, shipments of
ENERGY STAR units may decrease until manufacturers institute product design
changes to meet the new requirements. Nonetheless, products that once met
program standards but are then no longer ENERGY STAR-qualified continue to
be sold.17 Shipments of these products are included in reported ENERGY STAR
benefits; this assumes that the products continue to be manufactured as energy
efficient despite not meeting the new ENERGY STAR revised standard.
17 The methodology does not address the loss of potential energy savings that occurs because the consumer
purchased a formerly qualified product over a newly qualified higher performing alternative.
15
-------
09-P-0061
Market Transformation Benefits Calculated with Assumed Data
EPA's ENERGY STAR market transformation benefits process calculates and
reports savings based on a combination of assumed and actual shipment numbers.
The ENERGY STAR program does not have data to support the assumptions the
market transformation effect uses. Further, the assumed sales that are attributed
to the market transformation effect represent non-qualified products.
A peer reviewed article18 describing ENERGY STAR'S market transformation
effect included the following limitations statement:
"General limitations to a bottom-up approach occur in two main
areas: (1) the model requires numerous detailed inputs to generate
the end result and (2) uncertainty in those inputs are additive
through the process. These limitations mean that collecting and
documenting high-quality inputs is essential... targeting data
collection and verification activities at those areas is key to
successful results."
In this market transformation effect the main input - previously qualified
ENERGY STAR products - is based on assumptions rather than collected or
verified data. Program requirements do not require manufacturers to report sales
of formerly qualified ENERGY STAR products. EPA uses a model based on
assumptions to calculate the market transformation effect in the absence of actual
shipment data. By design, the model ensures that shipment amounts of products
are never lower than the amount in the year prior to the specification change.
The specification revision for DVD players illustrates EPA's method of
accounting for the market transformation effect. New specifications for DVD
players took effect January 1, 2003. Shipments of ENERGY STAR-qualified
DVD players rose that year, so no adjustment for the market transformation effect
was made. However, in 2004 and beyond, manufacturer-reported shipments of
ENERGY STAR-qualified DVD players fell below the 2003 level. To make up
for this drop in sales, which was attributed to the specification revision, the model
adjusted the current-year shipment figure based on the market transformation
effect. The model increased the shipment totals for 2004 by adding in an
estimated number of non-qualified ENERGY STAR DVD players shipped. This
assumption is based on the theory that even though this estimated number
represents non-qualified ENERGY STAR products, these products are more
energy efficient than they would have been without Energy Star. Therefore,
ENERGY STAR's impact was to transform the market to produce more efficient
products. The model continued to rely on this estimated level of product
18 Maria C. Sanchez., Richard E. Brown, Carrie Webber, and Gregory K. Homan. 2008. Savings estimates for the
United States Environmental Protection Agency's ENERGY STAR voluntary product labeling program. Energy
Policy, vol. 36, no. 6.
16
-------
09-P-0061
shipments for 2005 and 2006 despite further decline in the number of units
shipped. This is shown in Figure 3-1.
Figure 3-1: DVD Players Market Transformation Effect
Q
Q
10000
9000 -
8000-
7000 -
6000-
5000 -
4000 -
3000-
2000 -
1000-
0
2001
2002
2003
2004
2005
2006
Market transformation assumed shipments
D Actual Energy Star shipments
Source: OIG analysis of CCAP
Market Transformation Obscures Program Accomplishments
Based on our review of ENERGY STAR products that had specification
revisions, the market transformation effect was calculated for nine product
categories in 2006:
DVD players
Monitors
Telephony
Home audio equipment
TVs
TVs/VCRs
VCR & VCR/DVDs
Air source heat pumps
Central air conditioners
The savings derived from the market transformation effect may have a significant
impact on ENERGY STAR reported savings, because energy savings will
continue to accumulate for the lifetime of a product. To illustrate the impact of
the market transformation effect, we recomputed ENERGY STAR benefits by
eliminating assumed shipment data and comparing these to the original benefit
calculation. Based on our recalculation, ENERGY STAR energy savings reported
in 2006 were overstated, as shown in Table 3.1.19
This calculation represents one year; it does not include the potential overstatement from prior or future year
benefits calculated for the same products.
17
-------
09-P-0061
Table 3-1: Market Transformation Effect
Energy Saved
(Billion kWh)
Carbon Avoided
(MMTCE)
ENERGY STAR
Summary20
70.5
14.3
OIG
Re-calculation
66.3
13.5
Overstatement
4.2
0.8
Source: OIG analysis of EPA data
Shipment amounts were not based on actual or reported shipment data. EPA
counted benefits from non-qualified ENERGY STAR products into its annual
ENERGY STAR benefits amounts for 2006. Also, EPA did not compute a
market transformation effect for products with stable or rising sales after a
specification revision. The model is therefore inconsistently applied, since there
should be a benefit derived from products with increasing as well as decreasing
shipments.
Conclusion
ENERGY STAR does not disclose that it includes savings from products that are
not ENERGY STAR-qualified. The computation of the market transformation
effect does not use shipment or sales data, and only applies to products with both
specification revisions and decreasing sales. Further, reported ENERGY STAR
energy savings for 2006 omit mention of the market transformation effect, though
it is included in individual product energy savings calculations. ENERGY STAR
needs to improve controls over estimating and reporting benefits from the market
transformation effect. Partnership agreements should include requirements for
reporting shipment data of formerly qualified products, and the "effect" should be
applied consistently. Also, market transformation benefits should be separately
disclosed in ENERGY STAR'S annual reporting.
Recommendation
3-1 We recommend that the Principal Deputy Assistant Administrator for the
Office of Air and Radiation develop and consistently apply a data-driven
methodology to compute the market transformation effect of all product
categories and report the benefits separately from ENERGY STAR-
qualified products.
Agency Comments and OIG Evaluation
EPA said that the OIG did not provide a basis for recommending a change in the
market transformation methodology and suggested removing the
recommendation. The methodology is based on assumed shipment amounts and
1 The 70.5 billion kWh of energy saved comes from EPA-managed ENERGY STAR products.
18
-------
09-P-0061
reports savings from non-qualified products as ENERGY STAR qualified product
savings without proper disclosure. Therefore, the OIG does not agree with EPA's
suggestion to remove the recommendation. The recommendation is open and
unresolved. The Agency's comments and our evaluation of those comments are
in Appendix D.
19
-------
09-P-0061
Chapter 4
ENERGY STAR Commercial Benefits Model Utilizes
Unverified Assumptions
The methodology used to compute the ENERGY STAR commercial sector
savings was based on unverified assumptions. EPA calculates program savings as
the difference between all commercial sector gains in energy efficiency and
claims by utility and State programs for that year. However, we do not believe
the formula is sufficiently tailored to the specific needs of the ENERGY STAR
program. The formula needs to be verified to ensure that the end results -
benefits attributable to ENERGY STAR'S commercial activities - are valid and
accurately represent ENERGY STAR'S contributions to energy savings.
Formula Used to Calculate Commercial Sector Savings
EPA's ENERGY STAR program uses a formula created, operated, and
maintained by its contractor to compute the annual commercial sector savings. In
this formula, the contractor (a) calculates all commercial sector gains in energy
efficiency, (b) subtracts amounts from utility and State programs, and
(c) concludes that the remaining energy savings are attributed to the ENERGY
STAR program. ENERGY STAR staff said they review the formula with the
contractor but they do not document the review results. The formula uses data
from other agencies, along with related energy studies; it does not use any data
from the ENERGY STAR program.
Methodology Not Tailored for ENERGY STAR
The methodology used for ENERGY STAR is not tailored specifically to
calculate impacts attributable to that program and contains unverified
assumptions. The methodology, which calculates the commercial benefits
amount, is described in a 2007 paper by EPA's contractor.21 This paper states:
"In this study, energy efficiency programs refer collectively to all such
governmental efforts, irrespective of organizational origin or mode." This
methodology looks at all energy efficiency programs in total; it is not tailored to
pull out the impacts of the ENERGY STAR program. There is only one mention
of ENERGY STAR in the paper, because it is used to define the start of the
treatment period.
21 Horowitz, Marvin J. (2007). "Changes in Electricity Demand in the United States from the 1970s to 2003." The
Energy Journal, Vol. 28, No. 3, pp. 93-119.
20
-------
09-P-0061
The paper's goal was to answer the following three questions:
How has electricity demand changed in the United States over the past
three decades?
If electricity demand changed, do these changes differ between States
depending on their degree of commitment to encouraging energy
efficiency?
Does program commitment have observable, long-term impacts on the
behaviors that affect electricity demand?
These questions are not related directly to ENERGY STAR'S impact on
electricity demand. Rather, they seek to determine energy efficiency programs'
effects on energy demand.
The methodology was peer reviewed, but its application as the basis for
measuring ENERGY STAR'S impact on the commercial sector was not part of the
published methodology. EPA needs to verify that the benefits attributed to
ENERGY STAR are supported. Specifically, EPA should verify that the
methodology: (a) avoids double counting savings from the product sector,
(b) accounts for impacts of other federal energy efficiency programs, and
(c) accounts for impacts of non-federal energy efficiency programs.
Method to Avoid Double Counting Products Savings Unclear
The formula used to compute ENERGY STAR commercial sector energy savings
does not use any data, results, or statistics from the EPA's ENERGY STAR
program. Rather, the formula uses databases maintained by other government
agencies, such as the Federal Reserve Board, National Climate Data Center, and
Energy Information Administration. EPA ENERGY STAR staff said that the
formula uses the best information available. To account for benefits from the
ENERGY STAR products sector, the formula uses a Federal Reserve Bank's
market group index for information processing and related equipment.
Approximately half of all ENERGY STAR product savings in 2006 were for
commercial products, totaling 37.1 billion kWh. When the Agency reports total
annual savings it includes savings from products, residential, industrial and
commercial. If the commercial savings do not account for the savings from
commercial products reported by the Agency, there may be the potential for
overstatement of savings. The contractor and EPA officials asserted there was no
double counting, but they could provide no evidence to support that position.
Method for Accounting for Impacts of DOE Program Unclear
The ENERGY STAR program's methods for accounting for the impacts of a
DOE program are unclear. The Energy Policy Act of 1992 set standards for
certain commercial equipment and set schedules requiring DOE to make, review,
21
-------
09-P-0061
and update standards. The DOE's Building Technologies Program works to
improve the efficiency of buildings and the equipment, components, and systems
within them. These DOE programs should contribute to some of the energy
savings within the commercial sector. However, EPA's formula does not
evaluate or compensate for the energy efficiency contributions from DOE
programs. Rather, the formula only deducts savings from utility, State, and local
programs. ENERGY STAR staff said that the formula accounts for the DOE
programs through the use of the Federal Reserve Board's market index group.
The accuracy of using this index compared to actual reported data from both EPA
and DOE should be validated to determine if the formula is reporting ENERGY
STAR commercial benefits accurately.
Method for Accounting for Impacts of Non-federal Energy Efficiency
Programs Unclear
The method for accounting for the impacts of non-federal energy efficiency
programs is unclear. The formula subtracts energy savings that are attributable to
utility companies, States, and local programs - such as demand-side management
(DSM) and public benefits charges (PBC) from total commercial savings - to
arrive at net ENERGY STAR savings for the commercial sector. The contractor
calculates savings from these other programs22 by using information on DSM and
PBC programs from the study A Nationwide Assessment of Utility Sector
Efficiency Spending, Savings, and Integration with Utility System Resource
Acquisition. The contractor performs the following alterations to the numbers
taken from this study:
1. Project or estimate DSM and PBC savings from 2004 to 2006. (The
study's results were from 2004; the contractor updated this data to 2006 by
using a growth rate that it determines.)
2. Allocate the share of energy savings that occur in the commercial
sector. (The study's reported amounts were a combination of residential,
commercial, and industrial savings.)
3. Apply a net-realization rate that lowers the computed DSM and PBC
savings because, as explained by the contractor, the full amount of energy
saved reported by electric utilities for their DSM programs cannot be
taken at face value.
Based on the first two steps, the computed savings from the DSM and PBC
programs were 38,502 gigawatt hours, but EPA's contractor took 42.5 percent23
of this amount, lowering the savings figure to 16,363 gigawatt hours. This was
Other programs consist of State, utility and local energy efficiency programs.
This percentage is supported by a memorandum prepared by the contractor that spells out how the program
allocation factor for the formula was calculated. Specifically, the 42.5 percent is referred to as a net-to-gross factor,
or a net realization rate.
22
-------
09-P-0061
then attributed to the other energy efficiency programs. The application of the
net-realization rate significantly impacts the ENERGY STAR-derived energy
benefit. The lower the rate the greater the portion of savings is attributable to
ENERGY STAR. ENERGY STAR staff said that all formula inputs are from the
best available and/or peer reviewed data.
Formula Methodologies and Assumptions Need to Be Verified
The spreadsheet used to calculate the energy savings for the commercial sector
was not verified to ensure that its results are an accurate computation of energy
savings attributable specifically to the ENERGY STAR commercial sector.
ENERGY STAR plans to expand the use of this formula to compute energy
savings for the industrial sector, so it is additionally important to verify and
validate the formula's results.
The EPA Quality Manual for Environmental Programs CIO 2105-P-01-0
(formerly 5360 Al) defines a peer review as:
A documented critical review of work by qualified individuals (or
organizations) who are independent of those who performed the
work, but are collectively equivalent in technical expertise. A peer
review is conducted to ensure that activities are technically
adequate, competently performed, properly documented, and satisfy
established technical and quality requirements. The peer review is
an in-depth assessment of the assumptions, calculations,
extrapolations, alternate interpretations, methodology, acceptance
criteria, and conclusions pertaining to specific work and of the
documentation that supports them.
A peer review could accomplish two objectives. First, it could validate that the
formula reasonably computes all energy savings. Second, it could provide
assurance that savings are attributable to EPA's ENERGY STAR commercial
sector's activities. The validation should be conducted in accordance with EPA
Quality Manual for Environmental Programs CIO 2105-P-01-0.
Conclusion
The methodology used to compute the ENERGY STAR commercial savings is
based on unverified assumptions that impact the accuracy of the reported energy
savings. The methodology is based on a study of the impact of all energy
efficiency programs on electric demand, not only ENERGY STAR; it assumes
that all net savings are attributable to the ENERGY STAR commercial program.
EPA needs an independent assessment or validation of how ENERGY STAR
savings are computed to establish a reasonable assurance that the energy saved is
accurate.
23
-------
09-P-0061
Recommendation
4-1 We recommend that the Principal Deputy Assistant Administrator for the
Office of Air and Radiation validate the formula (methodology) used for
calculating the benefits of the ENERGY STAR commercial program in
accordance with EPA Quality Manual for Environmental Programs CIO
2105-P-01-0, to ensure that it accurately reflects the impacts of EPA
actions.
Agency Comments and OIG Evaluation
EPA maintains that the estimates generated by the methodology are rigorous, that
where marginal uncertainties are present they have been handled conservatively,
and that the results are sound. EPA said that it will secure additional outside
expert review of the entire methodology being used to estimate the benefits of the
ENERGY STAR program in the commercial sector. The EPA action meets the
intent of the recommendation; however the recommendation remains open until
the completion of the expert review. The review should assure that assumptions,
data sources, and methods used to estimate the ENERGY STAR commercial
benefits savings are reasonable and supported. The Agency's comments and our
evaluation of those comments are in Appendix D.
24
-------
09-P-0061
Status of Recommendations and
Potential Monetary Benefits
RECOMMENDATIONS
Rec.
No.
2-1
Page
No.
13
Subject
To improve the validity of reported annual savings
Status1
0
Action Official
Principal Deputy
Planned
Completion
Date
POTENTIAL MONETARY
BENEFITS (in SOOOs)
Claimed Agreed To
Amount Amount
3-1
4-1
18
24
for the ENERGY STAR program, establish and
perform quality controls to ensure that:
Data in benefits calculations, whether from
partners or third parties, are timely, complete,
valid, and documented.
The contractor and third party associations
receiving the manufacturer data submittal
forms reconcile submittals in a manner that
ensures the total annual shipments reported
by product category are accurate and reflect
actual numbers (not estimates) and are for
domestic shipments only.
Agency officials improve contractor oversight
by obtaining actual support for annual
savings in a manner that demonstrates that
the numbers are valid and can be reconciled.
Data in benefits calculations attributable to
DOE products should be clearly identified
and developed in consultation with DOE to
avoid redundancy.
Develop and consistently apply a data-driven
methodology to compute the market transformation
effect of all product categories and report the
benefits separately from ENERGY STAR-qualified
products.
Validate the formula (methodology) used for
calculating the benefits of the ENERGY STAR
commercial program in accordance with EPA
Quality Manual for Environmental Programs CIO
2105-P-01-0, to ensure that it accurately reflects
the impacts of EPA actions.
Assistant Administrator,
Office of Air and Radiation
Principal Deputy
Assistant Administrator,
Office of Air and Radiation
Principal Deputy
Assistant Administrator,
Office of Air and Radiation
0 = recommendation is open with agreed-to corrective actions pending
C = recommendation is closed with all agreed-to actions completed
U = recommendation is undecided with resolution efforts in progress
25
-------
09-P-0061
Resources
Logic Model for ENERGY STAR Products
Activities
Outputs
Reassessment
Assessment/
Outcomes
Intermediate
Outcomes
Appendix A
Long Term
-Dollars
-Staff
Program Administration
-- Up to date product lists
Up to date partner
information
-- Data collection on
product shipments
Build Supply of Efficient products
Add new products (using
established guidelines)
Revise Product Specs
as appropriate (using
established guidelines)
Build Demand Outreach
-- robust web-based info
consumer hotline
-- direct consumer efforts
-- key product strategies
partner by partner
assistance
coordinated national
campaigns
train distribution channels
-- product procurement
Ensure Quality
require proper use of
logo in ads
-- spot test products
improve retail experience
Evaluation
-- consumer awareness,
understanding, etc
-- progress assessments
Products
-- # product categories
# product models
# product sales (annual)
market penetration
Partners
# manufacturers
-- # retailers
# program sponsors
-- # states and others
Partner activity
-- activity/investment by
retailers, program sponsors,
manufacturers
Outreach
-- # web visitors
key visitor interests
-- # media impressions
# media impressions by
key strategy and product
Compliance/Quality
-# advertising clips
# companies advertising
-- # violations
# companies with
violations
Indication of manufacturer
QA/QC issues
-- lighting fixture violations
Indication of retailer issues
Brand Information
-- level of consumer
awareness, understanding,
etc
Benefits Projections
-- estimates by product of
future sales/benefits
based on continued efforts
Annual review of key outputs
against program near, mid,
and long term goals for
program and key product
areas and revision of
strategies, as appropriate
Energy Savings
-inkWh
by product area
program level
Reductions in GHG
in mmtce
-- by product area
program level
Investment in technology
-- in $$
-- program level
Net Savings on Energy bills
- in $$
program level
Program Cost-effectiveness
Behavioral Change
Improved Energy Efficiency
Policies for Product
Efficiency
Cleaner Air
Reduced Risk of
Climate Change
Improved Public Health
Cost Savings
Institutionalized
Behavior Change
Source: ENERGY STAR program
26
-------
09-P-0061
Appendix B
Logic Model for ENERGY STAR Commercial & Industrial Programs
Resources
Activities
Program Administration
- Up to date partner
information
Maintain portfolio manager
and other key tools
Build Demand -- Outreach
-- robust web-based info
customer hotline
-- national campaign on
building improvements
-- encourage state/local lead
by example
web-based and other
trainings for end-users
"Dollars ~~ rec°9niti°n f°r excellence
- broad corporate leadership
"Staff achieving key milestones
-- exemplary buildings
Build Supply of Services
-- partner with service
providers
- train service providers on
benchmarking and other
ENERGY STAR resources
-- partner with utilities; train;
and help develop programs
offering benchmarking.
Ensure Quality
require proper use of
logo in ads
audit labeled buildings
Outputs
Reassessment
Partners Committed to
Improvements
# private sector organ
-- # public sector
# key sub sectors
- # states
Partner activity
activity/investment by
end-user partners, program
sponsors, third-party
providers, etc
-- number of ENERGY STAR
leaders
Standard Measurement System
# building types
- % of market covered
number of benchmarked
buildings, % of key subsectors
benchmarked, square footage
benchmarked
-- number of buildings labeled,
% of key subsectors labeled,
square footage labeled
Building Service Market
- # SSP partners
# third party providers of
benchmarking services
--Strainings
number of benchmarks
provided
Outreach
- # web visitors
key visitor interests
-- # media impressions
-- number campaign participants
Compliance/Quality
-# advertising clips
# companies advertising
-- # violations
# companies with
violations
Assessment/
Outcomes
Annual review of
key outputs
against program
near, mid, and
long term goals
for program and
sales of key
products and
revision of
strategies, as
appropriate
Intermediate
Outcomes
Energy Savings
- in kWh
- program level
Reductions in GHG
emissions
- in mmtce
- by program level
Investment in technology
- in $$
- program level
Net Savings on Energy
bills
- in $$
- program level
Program Cost-
effectiveness
Behavioral Change
Improved Energy
Efficiency Policies for
Building Efficiency
Long Term
Outcomes
Cleaner Air
Reduced Risk of
Climate Change
Improved Public
Health
Cost Savings
Institutionalized
Behavior Change
Source: ENERGY STAR program
27
-------
09-P-0061
Appendix C
Details on Scope and Methodology
To determine whether the savings reported for products and commercial ENERGY STAR
program sectors were valid and fully supportable, we analyzed supporting documentation for
2006 reported annual savings. We reviewed the ENERGY STAR and Other Climate Protection
Partnerships 2006 Annual Report (September 2007) and the U.S. Climate Action Report - 2006
for savings and benefits claims made by the program. We also reviewed relevant reports and
spreadsheets by Lawrence Berkeley National Laboratory and various contractors.
We conducted a review of the savings benefits for the products and commercial sectors, which
combined encompassed 81.6 percent of the 200624 reported carbon emissions avoided; we did
not conduct a detailed review of the residential and industrial sectors. To analyze the methods
used to calculate the annual savings by sector, we met with EPA ENERGY STAR officials and
the primary contractors involved. For the product sector, we obtained available support data and
documented the roles of both EPA and the contractor in the computation and methodology
process. Specifically, we requested the supporting documentation for the 2006 shipment totals
used in the savings benefit calculation for 2006 (as described in Chapter 2). We were provided
manufacturer shipment data submittal forms and shipment reports from third party associations.
We performed a detailed analysis and reconciliation of the manufacturer shipment data
submittals for all 2006 reported amounts by qualified ENERGY STAR product category. We
documented the forecasting methods for the other EPA product categories for which submission
of direct data shipment data was not required for 2006 reporting. We judgmentally selected a
sample of four qualified products based on a number of factors including: ENERGY STAR
shipment totals for 2006 and specification revision history. We traced and reconciled these four
qualified products through the Lawrence Berkeley National Laboratory CCAP model. These
products were computers, DVDs, external power supplies, and monitors.
For the commercial sector, we reviewed the methodologies used to calculate ENERGY STAR
reported benefits. We met with the contractor to obtain an understanding on how commercial
benefits are calculated. We also analyzed the contractor's spreadsheet, containing the inputs and
assumptions used in calculating the 2006 commercial ENERGY STAR reported benefits.
To determine what management controls EPA had in place to ensure the validity and accuracy of
reported annual savings, we analyzed available data and met with EPA and contracting officials.
We reviewed the OAR (Office of Air and Radiation) Climate Protection Partnerships Division's
ENERGY STAR Performance Management Plan. Our review included a detailed follow-up with
the ENERGY STAR Director on plan specifics. We met with the contractor involved in
implementing the plan to determine the role quality assurance/quality control plays in its work in
the computation process. We also analyzed some of the controls/procedures within the plan to
verify whether they were applicable and had been adequately performed as described within
Chapter 2. All internal control weaknesses noted during our evaluation are discussed in
Chapters 2, 3, and 4 of this report.
' At the time of our evaluation, 2006 was the most current data available.
28
-------
09-P-0061
To determine the basis for goal setting, we meet with ENERGY STAR staff and reviewed the
program's logic models. We also reviewed recent U.S. Climate Action Reports, Guidelines for
Designing EPA Partnership Programs, Government Performance Results Act of 1993, and the
Energy Policy Act of 2005.
29
-------
09-P-0061
Appendix D
Agency Comments and OIG Evaluation
MEMORANDUM
SUBJECT: Response to Draft Evaluation Report: Improvements Needed to Validate Reported
ENERGY STAR Benefits: Project No. 2007-0081
FROM: Robert J. Meyers
Principal Deputy Assistant Administrator
TO: Jeffrey Harris
Director for Special Studies
Office of the Inspector General
We appreciate the opportunity to review the OIG's draft report on the benefits of the
ENERGY STAR program and their recommendations for improving the benefit estimation
methods OAR employs. OAR has or will address two of the three recommendations and asks
that the third recommendation be removed.
OAR maintains a strong commitment to using and continually improving the methods it
employs to provide robust estimates of the benefits of the ENERGY STAR programs. In this
context, OAR will proceed to address the OIG recommendations, where they have not already
been addressed.
OIG Response: OIG understands the ENERGY STAR program reports estimates. The
statement that, "More than 37 million metric tons of greenhouse gas emissions were avoided,
equivalent to the greenhouse gas emissions from 25 million vehicles," does not disclose that
reported program benefits are robust estimates. The other reportable program benefits are
reported in a similar matter.
OAR is very concerned about the tone and exaggeration of issues in the OIG report as
well as the general implication that OAR's estimates of ENERGY STAR program benefits are
not valid. OAR stands by the validity of the reported estimates of the ENERGY STAR program
benefits.
OAR methods and procedures conform to Agency policy and guidance.
OAR employs peer-reviewed methods and appropriate QA/QC procedures.
OAR approaches are consistent with well-documented best practices for developing
estimates of the benefits of energy efficiency programs.
OIG did not identify any issues that have a significant impact on the reported results for
the ENERGY STAR program.
30
-------
09-P-0061
OIG Response: OIG reported that the data used to calculate benefits was inaccurate
and unsupported; therefore we could not attest to the overall validity of the actual
reported numbers. For the product and commercial program sectors we documented
how benefits were calculated, evaluated the supporting data, and outcome of EPA's
calculations with the claimed impacts. In our opinion there were numerous
deficiencies in the processes and data supporting the benefit calculations.
It is important to understand that program benefit estimates are estimates. Such estimates are
routinely based on available data and reasonable assumptions. OAR employs conservative
estimates for assumptions where there is a lack of available data and some uncertainty.
Further, OAR does not believe that the OIG identified any issues that have a significant
impact on the reported results for the ENERGY STAR program. The OIG review of the
ENERGY STAR benefits calculations was narrowly focused on several issues and processes. In
addition, there are important inaccuracies in the OIG report and misunderstandings of the
methods that OAR employs (see supporting information).
OIG Response: It is the OIG's opinion that the Agency approach for estimating program
benefits is not conservative. For example, we do not believe that the Agency's use of the
market transformation effect is a conservative approach. The OIG takes no position on the
validity of the market transformation effect, but we disagree with the methodology used by the
Agency to compute this effect. An additional issue we noted was the lack of disclosure in the
annual report. Benefits derived from the market transformation are not identified in the annual
report and are included along with benefits from ENERGY STAR qualified products.
Furthermore, the OIG's objective was to determine the validity of the ENERGY STAR benefit
calculations. We conducted a comprehensive review of the methodology, data, and analysis
used by EPA to calculate over 80 percent of the ENERGY STAR reported carbon emissions
avoided in 2006. We found deficiencies in the design, data, and execution of the benefit
calculations. We did not attempt to quantify the net numerical impact of these deficiencies.
OIG believes that the net impact of reported deficiencies may be significant. For example,
when calculating individual consumer product benefits if the initial data shipment numbers are
unsupported or inaccurate, not only is the benefit calculation for that current year invalid but the
subsequent calculation of benefits are impacted over the lifetime of that product category. As
noted in the Agency's peer reviewed article:
We implement the bottom-up model with awareness that uncertainty for each product
type contributes to uncertainty in total ENERGY STAR impacts. This means that many
small inaccuracies are additive overall and any one inaccuracy for a product type with
large energy savings can significantly affect the overall results.
We do not agree that the OIG report contains inaccuracies. The Agency's response did not
provide evidence in support of this statement.
31
-------
09-P-0061
The result of the OIG review is that
The OIG identified several ways for OAR to improve its documentation of data and
QA/QC procedures, steps that OAR has already or will undertake this coming year.
The OIG questions an approach that OAR uses to represent the market transformation
benefits of the ENERGY STAR product labeling program; however, because this
approach is a well accepted practice, has been peer-reviewed, addresses the uncertainty in
a conservative manner, and represents about 2 percent of overall program benefits, OAR
does not believe a change is necessary.
The OIG identified two addition errors that would serve to increase the ENERGY STAR
program benefits for 2006 by about 1 percent. These errors have been corrected.
OIG Response: OIG does not agree with the above assessment of the results of the OIG's review.
The supporting information OAR provided in this response will be addressed by each subject as
deemed necessary.
In addition, the OIG made three recommendations to improve the methods and processes OAR
uses for estimating the annual benefits for the ENERGY STAR program. These
recommendations and OAR's responses are provided below.
Recommendation 2-1 To improve the validity of reported annual savings for the ENERGY
STAR program, we recommend that the Principal Deputy Assistant Administrator for the Office
of Air and Radiation establish and perform quality controls to ensure that:
Data in benefits calculations, whether from partners or third parties, are timely, complete,
valid and documented.
The contractor and third party associations receiving the manufacturer data submittal forms
reconcile submittals in a manner that ensures the total annual shipments reported by product
category are accurate and reflect actual numbers (not estimates) and are for domestic
shipments only.
Agency officials improve contractor oversight by obtaining actual support for annual
savings in a manner that demonstrates that the numbers are valid and can be reconciled.
Data in benefits calculations attributable to DOE products should be clearly identified and
developed in consultation with DOE to avoid redundancy.
OAR Response: OAR addresses the quality of annual shipment numbers used for
ENERGY STAR benefits calculations through a systematic review and improvement of the
data submitted based on additional follow up with reporting manufacturers. There are clear
deadlines for the reporting of data and ramifications for partners not reporting data by these
deadlines. Estimates and international shipments, while collected in a few limited cases,
are not used as part of estimating annual program benefits. Documentation of these
requirements, data submittal timelines and penalties for not meeting them is available at
www.energystar.gov/usd.
32
-------
09-P-0061
OIG Response: OIG evaluated the existing process. The findings show that the process used to
collect and verify the data was deficient and needs improvement. Corrective action may include
improved implementation and oversight of the existing process as well as new procedures. For
example, we found large discrepancies in the shipment data used in the benefits calculation for
exit signs and monitors. The table provided by the Agency not only acknowledges errors
occurred, but validates the lack of a quality control system. The large "spreadsheet errors,"
uncorrected until disclosed by OIG, demonstrate deficiencies in the existing Quality
Assurance/Quality Control process. Additionally, at the time of our review, estimates of
shipment totals were used for both office products and for programmable thermostats in
calculating annual benefits.
Status: Suggest removing the first two bullets. The remaining two recommendations have
been addressed: 1) The EPA contractor has been directed to fully document the data
collection and review process for product shipment data for 2007 and subsequent years.
This has been completed for 2007 2) For the future, EPA has removed the benefits
estimates attributable to EPA efforts with compact fluorescent lighting (CFL) products as of
the 2007 program benefits estimates and will only reintroduce these benefits after
appropriate coordination with DOE.
OIG Response: OIG does not agree with the removing of the first two bullets. Further, the
Agency's corrective action plan will need to address the shortcomings related to improving
contractor oversight. The Agency should identify improvements in management controls over
the existing process. OIG identified deficiencies with the process of collecting, reviewing, and
reporting product data shipment information both from third parties or actual manufacturers.
The status of the recommendations reported by the Agency does not meet the intent of the
recommendation. This recommendation is open and unresolved.
Recommendation 3-1 We recommend that the Principal Deputy Assistant Administrator for the
Office of Air and Radiation develop and consistently apply a data-driven methodology to
compute the market transformation effect of all product categories and report the benefits
separately from ENERGY STAR-qualified products.
OAR Response: Calculating market transformation benefits is a widely accepted concept
within the energy community and is a meaningful indicator of the ongoing effect of
ENERGY STAR in the market. The current methodology for calculating market
transformation benefits has been peer reviewed (see below) and is applied consistently
across the product categories, where applicable. Further, the market transformation effect
accounts for a small fraction of overall ENERGY STAR program savings (e.g. less than 2
percent), meaning that uncertainty in the data and assumptions behind the calculations
(uncertainty that has already been addressed through conservative assumptions) have a
relatively low impact on overall modeled results.
Method has been peer reviewed in the journal Energy Policy:
33
-------
09-P-0061
Sanchez, Maria C., Richard E. Brown, Carrie Webber, and Gregory K. Homan.
2008. Savings estimates for the United States Environmental Protection Agency's
ENERGY STAR voluntary product labeling program. Energy Policy, vol. 36, no.
6, pp. 2098-2108
OIG Response: OIG did not question the concept, only the Agency's methodology and data
sources used to compute benefits for ENERGY STAR. The methodology and data used to
calculate ENERGY STAR market transformation benefits has not been validated.
Additionally, the peer reviewed article includes a limitations section which provides the
following:
These limitations mean that collecting and documenting high-quality inputs is
essential, which can be a labor-intensive and expensive process. As a result,
identifying areas of critical uncertainty and sensitivity and then targeting
data collection and verification activities at those areas is key to successful
results. [Emphasis added]
The OIG evaluated reported benefits incurred in 2006. For 2006, the market transformation
effect was 6 percent of the reported benefits for the product category. The significance of
savings from the market transformation calculation can vary depending on the number of new
specifications in a year, as well as the reported shipment totals. We do not consider using
unsupported shipment totals to compute benefit savings in the market transformation
calculation a conservative assumption. Furthermore, the OIG analysis does not take into
account the reported savings that can be realized in future years since these units will be
counted for their expected product life.
Status: Since, the OIG has not provided a basis for recommending a change in
methodology, particularly in light of the complexity and cost associated with doing so,
OAR suggests removing this recommendation.
OIG Response: It is the OIG's opinioin that the computation of the market transformation
effect requires identifying areas of critical uncertainty and sensitivity and then targeting data
collection and verification activities at those areas for successful results. That has not been
done. While the benefits calculated from market transformation make up 6% of the product
savings, given the downturn in the economy and changes to major consumer products (like
DVDs) the impact of a market transformation calculation is likely to grow larger. This
recommendation is open and unresolved.
Recommendation 4-1: We recommend that the Principal Deputy Assistant Administrator for
the Office of Air and Radiation validate the formula (methodology) used for calculating the
benefits of the ENERGY STAR commercial program in accordance with EPA Quality Manual
for Environmental Programs CIO-2105-P-01-0, to ensure that it accurately reflects the impacts of
EPA actions.
34
-------
09-P-0061
OAR Response: OAR's approach to estimating the benefits of the ENERGY STAR
program in the commercial sector has evolved as the program has expanded and adopted
up-to-date econometric analysis and information. Each refinement has been reviewed by
outside experts. OAR believes that the estimates generated by the methodology are
rigorous, that where marginal uncertainties are present they have been handled
conservatively, and that the results are sound. OAR's program benefits estimates are
performed in accordance with the EPA Quality Manual as the Office of Atmospheric
Programs has established a Quality Management Plan and these benefits estimates have
been developed in accordance with that plan.
OIG Response: The peer reviewed methodology does not directly support Office of Air and
Radiation's claimed benefits for the ENERGY STAR commercial sector. EPA uses parts of that
methodology to derive the estimated benefits for commercial sector. The actual computation
methodology used needs to be validated.
Further, OAR recommends that the title for Chapter 4 be modified to be "Further
Validate Commercial Sector Methods
OIG Response: The OIG believes the current title accurately reflects the content of the chapter.
Status: OAR will secure additional outside expert review of the entire methodology
being used to estimate the benefits of the ENERGY STAR program in the commercial
sector. We can provide the OIG for a schedule for this review within two months.
OIG Response: The Office of Air and Radiation action meets the intent of the recommendation.
The expert review should assure that assumptions, data sources, and methods used to estimate the
ENERGY STAR commercial benefits savings are reasonable and supported.
Supporting Information
OAR has identified a number of errors in the OIG report and several important
misunderstandings in the OIG report as detailed below.
2006 Annual Shipment Data
OAR has reviewed the table the OIG presented in its draft report showing that it could not
reproduce the 2006 product shipment totals in a July 2007 ICF report for 18 of the 29 product
categories that it examined. OAR looked at eleven of the eighteen product categories that the
OIG examined, the ones which would potentially have the most significant impact on the 2006
stated program results due to the size of the differences in the OIG table. OAR has the following
observations based on its review (see attached summary table):
OAR has confirmed the validity of the shipment numbers used in the 2006 benefits
calculation for nine of the eleven product categories OAR examined.
35
-------
09-P-0061
The differences that the OIG found are largely explained by the OIG not completing the
QA/QC process that is in place to review and improve the raw submittal data.
Two computational errors were uncovered as a result of the audit. These two errors serve to
increase the ENERGY STAR program benefits and together represent a small fraction (about
1%) of the programs overall reported benefits. These errors have been corrected.
The OIG findings do not support the broad claim that the ENERGY STAR program benefits
are invalid.
OIG Response: The Agency provided no additional documentation or support for its
analysis, so we cannot comment on its accuracy. However, given the quality of the
documentation supporting ENERGY STAR benefit calculations, OIG understands Office of
Air and Radiation's difficulty. OIG analysis and numbers were based on the documentation
provided by the EPA contractor tasked with reporting these numbers. In fact, we met with the
contractor concerning this issue and it was unable to provide us with any further
documentation that would change any of our reported numbers or discrepancies. Our detailed
analysis identified large inaccuracies in the reported shipment totals used and in what the
actual shipment data supported for products such as exit signs and monitors. The Agency's
table provided in this response acknowledges that there were in fact "spreadsheet errors" for
both products and the numbers should have been the much larger number as identified by the
OIG. These two "spreadsheet errors" demonstrate deficiencies with the existing Quality
Assurance/Quality Control process since large errors were not identified. The errors were
reported by the OIG. The Agency work was done after the fact, in an attempt to verify the
original numbers. It should also be noted that the underestimates were already used in the
2006 benefits calculations process and were already reported within the Annual Report.
Further, OAR disagrees with the characterization of the OIG in a number of other areas in the
report and provides the following clarifications.
Quality Review of Data. OAR's contractor responsible for collecting and compiling shipment
data performs a systematic review of all shipment reports that looks for and addresses a number
of potential issues. The contactor has been directed to document the results of this quality
review. The issues the contractor addresses include:
Multiple submissions received from a single partner
Inclusion of a partner's data in an aggregate submission from an association, when the
partner had already reported to contractor directly
An apparent math error on the data collection form
Submission of shipment data for a product that cannot earn the ENERGY STAR, e.g.,
qualified shipments reported for ENERGY STAR camcorders
Submission of data on a non-standard form, e.g., a NYSERDA form
Indication that reported, qualified shipments went to a private labeler, who was also
required to report
Alteration of the standard form, e.g., partner modification of units of measure
36
-------
09-P-0061
Failure of the partner to break data into requested subcategories
ENERGY STAR qualified shipments exceeding total shipments
Indication that the submission is only an estimate
Indication of negative shipments for a partner
A partner's claim of qualified shipments for a product they do not manufacturer, have
not yet qualified with ENERGY STAR, or for which they've not partnered with
ENERGY STAR.
In addition to working with partners and third party associations to resolve data anomalies,
the contractor performs the following steps to improve the quality of the report:
Reviews past year's process recommendations and reviews with EPA if any changes
are needed to solicit the most accurate information from partners in the clearest way.
Shares information including contact lists with third party associations well in
advance, to ensure that all eligible partners contribute to the collection effort.
Contractor product leads review submitted data for their respective categories to flag
any apparent discrepancies or unexpected changes from previous years.
Task Order Manager checks data entry, calculations, and consistency among summary
and breakout tables.
The EPA contractor has been directed to fully document the data collection and review process
for product shipment data for 2007 and subsequent years. This is complete for 2007.
OIG Response: If the Quality Assurance/Quality Control system the Agency summarizes in
this section was implemented and enforced as the Agency implies then the errors we identified
would have been caught by either the contractor or EPA. The Quality Assurance/Quality
Control process existing at the time of the OIG's review was deficient as demonstrated by the
errors identified in the report. Many of these steps were either not in place or had not been
implemented during our review. We were advised by the EPA contractor that follow-up
actions were not always documented, which may have contributed to the discrepancies in
shipment totals used in the benefit calculations. The EPA contractor told us that the third party
reporting was inconsistent, lacked uniformity, and information was often submitted late.
Data Submitted Through Third Parties. The ENERGY STAR partnership agreement
explicitly allows for the submittal of shipment data through a third party aggregator in the
event a partner has competitiveness concerns associated with releasing data. OAR's
contractor works closely with the relevant trade associations to ensure that any potential data
issues are resolved as the data is compiled. This is an appropriate and necessary practice to
employ as part of collecting information for a partnership program.
37
-------
09-P-0061
OIG Response: OIG took no position on the use of third parties. OIG identified
deficiencies with the reporting of the data received by the contractor. The contractor
recognized there were problems with the third party association data submittals. We were
told that it was the contractor's practice to accept the third party reported numbers even if a
manufacturer had self-reported a different set of numbers.
International Shipments. The OIG did not provide any credible basis for OAR to question
that shipment totals used in the benefits calculations include anything but U.S. shipments. In
fact, the two instances cited give no indication other than that the submitter clearly
understood the request to be for U.S. data alone. OAR does not use international shipment
information to estimate program benefits.
OIG Response: OIG provided examples of potential control issues with accounting for
international shipments.
Compact Fluorescent Light (CFL) Bulbs
OAR estimated benefits from EPA efforts with compact fluorescent light bulbs. EPA has made
important contributions to the area of ENERGY STAR CFLs for more than eight years. This
includes contributions to key aspects of the ENERGY STAR specification for CFLs, quality
testing for CFLs, and ramping up consumer awareness and interest in purchasing ENERGY
STAR qualifying CFLs. EPA has removed these benefits from its ENERGY STAR program
benefits as of 2007.
OIG Response: OIG reported that EPA and DOE were both claiming the CFL benefits
which results in double counting of the greenhouse gas savings. OIG did not question
EPA's contribution to CFLs. We agree that removal of the CFL benefits until they are
discussed with the appropriate DOE officials is the appropriate action.
Market Transformation Effect
OAR employs a widely accepted concept within the energy community for estimating the market
transformation benefits of the ENERGY STAR program. This captures the set of products in the
marketplace, that are more efficient than they otherwise would have been, that were once eligible
for the ENERGY STAR, but are no longer due to increased stringency of the ENERGY STAR
specification. The current methodology for calculating these market transformation benefits has
been peer reviewed in the journal Energy Policy:
Sanchez, Maria C., Richard E. Brown, Carrie Webber, and Gregory K. Homan. 2008.
Savings estimates for the United States Environmental Protection Agency's ENERGY
STAR voluntary product labeling program. Energy Policy, vol. 36, no. 6, pp. 2098-2108.
38
-------
09-P-0061
The method is applied consistently across the product categories for which an ENERGY STAR
specification has been revised and a substantial number of products are still in the market that
met the earlier specification. Further, the market transformation effect accounts for a small
fraction of overall ENERGY STAR program savings (about 2%). This means that uncertainty in
the data and assumptions behind the calculations (uncertainty that has already been addressed
through conservative assumptions) has a relatively low impact on program estimates.
Accordingly, the OIG has not provided a basis for recommending a change in methodology,
particularly in light of the complexity and cost associated with doing so.
OIG Response: The method is not applied consistently across the product categories. The
computation for market transformation is only used when shipment of units falls after a
specification change. The market transformation concept as stated would also apply to units
that increased in sales after a specification, which has occurred, but there was no computation
of the savings derived from these units. The ENERGY STAR contractor's formula does not
account for market transformation savings for units that increase after a specification change.
Furthermore, the OIG believes the savings reported from the market transformation units have
a potential to significantly impact reported savings. We do not consider knowingly reporting
unsupported and possibly incorrect data as a conservative practice even if it were properly
disclosed, which is not done in ENERGY STAR reports. The number of formerly qualified
units used in the computation is derived from reported shipments of ENERGY STAR qualified
units. There is no actual relationship, it is an assumption. The result is a flawed process that
will produce an inaccurate result, particularly given current economic conditions.
Boilers
Commercial
Refrigerators/
Freezers
Dehumidifiers
July 2007
Report
Total
382,770
83,493
1,203,216
OIG
Total
405,573
106,658
776,431
Summer
2008
Review*
382,770
83,493
1,203,216
Notes
Reported values add up to 382,770.
No underestimation.
OIG appears to have
excluded one manufacturer's data, because
their brand name, rather than their
manufacturer name, was listed on their
reporting form which ICF addressed through
their QA/QC process
excluded another manufacturer's data because
the company was not on the list of partners
required to submit data, though this was
addressed by ICF through their QA/QC
process
Included one manufacturer's reported data
that was excluded as a result of the ICF
QA/QC process.
No underestimation.
Reported values add up to 1,203,216.
No overestimation.
39
-------
09-P-0061
Exit Signs
Furnaces
Monitors
Roof Products
(Commercial
Gallons)
Room Air
Cleaners
Transformers
TV/VCR
Ventilating
Fans
1,425,554
1,172,843
13,859,666
21,352,516
287,354
173,390
17,685,967
853,406
3,450,117
1,170,124
34,581,203
15,966,045
289,815
166,669
13,166,140
857,232
3,460,176
1,172,843
32,584,882
21,358,657
287,354
173,390
17,689,230
853,406
Spreadsheet error. Compiled number should
have been 3,460,176. This underestimate has
been corrected in subsequent analysis
OIG appears to have excluded data that was
reported by one furnace manufacturer on their
air-source heat pump data form which was
addressed through the QA/QC process.
No overestimation.
Spreadsheet error. Compiled number should
have been 32,584,882. This underestimate has
been corrected in subsequent analysis
Shipments reported by one company were
inadvertently not included. ICF number should
have been 21,358,657.
No significant effect.
OIG appears to have included one
manufacturer's reported data that was excluded
as a result of the QA/QC process.
No underestimation.
OIG appears to have
excluded one manufacturer's data because
their brand name(s) rather than their
manufacturer name was listed on their
reporting forms which ICF addressed through
their QA/QC process
chosen to use data from one of the two forms
submitted by one manufacturer whereas ICF
confirmed with the manufacturer that the
other form was the correct one.
No overestimation.
Shipments reported by one company for one
product type (TV/VCR combination units) were
inadvertently not included. ICF total should
have been 17,689,230.
No significant effect.
OIG appears to have
included one manufacturer's reported data
that was excluded as a result of the QA/QC
process
included incorrect data for one company that
had been addressed through ICF's QA/QC
process.
No underestimation.
40
-------
09-P-0061
* Re-examination of what shipment totals were supportable based on manufacturer information
provided as of May 2007.
OIG Response: The OIG analysis was based on the documentation used to produce EPA's
report of ENERGY STAR benefits in 2006. We do note that the Agency's analysis does not
address all the irreconcilable products as reported on by the OIG within Chapter 2, Table 2-1.
Based on the Agency's chart provided, the Agency only reviewed 11 of the 18 products and thus
the Agency's analysis does not address the differences noted for the following product categories:
Audio/DVD
Commercial Hot Food Holding Cabinets
End Use Products
External Power Supplies
Residential Light Fixtures
Roofing Commercial & Residential Sq. Ft.
Telephony
Commercial Program Methodology (Addressed in Chapter 4 of OIG Report)
OAR uses best available information to estimate program benefits in the commercial market and
updates the methodology and inputs as the program evolves and new information is made
available. These methods can be summarized as follows:
Program estimates are based on well-documented peer-reviewed methods.
The data inputs are either peer-reviewed or publicly available data themselves or based on
reasonable and well-documented assumptions.
In areas of marginal uncertainty, OAR applies program evaluation best practice methods to
avoid overstatement of program benefits.
The current program benefits estimation method includes techniques to account for the
energy savings from commercial products associated with information processing and related
equipment used by businesses. The method was peer-reviewed, found to be appropriate and
sounds, and published in a 2007 Energy Journal article.
The current program benefits estimation method also includes techniques to account for
benefits from energy efficiency programs across the country. The approach of subtracting
state-utility savings from the national total was peer-reviewed and published in a 2001
Energy Journal article.
OAR disagrees with the OIG characterization of the methods used to avoid double-counting with
federal programs like ENERGY STAR qualified products and DOE's Building Technologies
Program (BTP) as "unclear" and "questionable." Again, these methods have been peer-
reviewed by experts in the field and found to be sound and appropriate.
41
-------
09-P-0061
OAR continues to improve the models it uses to estimate ENERGY STAR program benefits.
OAR will undertake another peer-review of the overall approach and its key elements.
OIG Response: The peer-reviewed methodology referenced by the Office of Air and
Radiation does not directly support the claimed benefits for the ENERGY STAR commercial
sector. The contractor who prepares EPA's estimate uses parts of the methodology in its
computation of the benefits. The contractor's computation of estimated benefits for
commercial sector is what needs to be validated.
42
-------
09-P-0061
Appendix E
Distribution
Office of the Administrator
Principal Deputy Assistant Administrator, Office of Air and Radiation
Agency Follow-up Official (the CFO)
Agency Follow-up Coordinator
Office of General Counsel
Associate Administrator for Congressional and Intergovernmental Relations
Associate Administrator for Public Affairs
Audit Follow-up Coordinator, Office of Air and Radiation
Deputy Inspector General
43
------- |