oEPA
Guidebook for Energy Efficiency
Evaluation, Measurement,
and Verification
A Resource for State,
Tribal Air & Energy Of
U.S. Environmental Protection Agency
June 2019
State and Local
Energy and Environment Program
-------
Acknowledgements
This document, Guidebook for Energy Efficiency Evaluation, Measurement, and Verification: A Resource
for State, Local, and Tribal Air & Energy Officials, was developed by EPA's State and Local Energy and
Environment Program within the Climate Protection Partnerships Division of EPA's Office of Atmospheric
Programs. Nikolaas Dietsch managed the overall development of the Guidebook. Julie Rosenberg and
Carolyn Snyder provided organizational and editorial support for the entire update of the document.
This guidebook builds on previous Evaluation, Measurement, and Verification efforts at EPA and reflects
extensive public comments received during those efforts.
EPA would like to acknowledge the many other EPA employees and consultants whose efforts helped to
bring this product to fruition.
The following contributors from EPA (unless otherwise noted) provided significant assistance for this
update through their technical and editorial review of one or more of the Guidebook's sections: Joe
Bryson, Beth Conlin, Robyn DeYoung, Tom Eckman (through Lawrence Berkeley National Laboratory),
Natalie Mims Frick (Lawrence Berkeley National Laboratory), Maureen McNamara, and Denise
Mulholland.
A multidisciplinary team of energy and environmental consultants, led by ICF, provided extensive
research, technical input and review, editorial review, and graphics support for this Guidebook. Key
contributors from ICF (unless otherwise noted) include: Miriam Goldberg (DNV GL), Tara Hamilton, Brad
Hurley, Wendy Jaglom, Cory Jemison, and Logan Pfeiffer. Pat Knight of Synapse Energy Economics and
Steven Schiller of Schiller Consulting, Inc. also provided technical review, research, writing, and graphic
support for the document.
For more information, please contact:
Julie Rosenberg
U.S. Environmental Protection Agency
State and Local Energy and Environment Program
Tel: (202) 343-9154
Email: rosenberg.iulie(S)epa.gov
1
-------
Table of Contents
Acknowledgements 1
Table of Contents 2
Acronyms and Abbreviations 3
1. Introduction 5
1.1. Audiences and Uses 6
1.2. Experience with EM&V 8
1.3. Scope, Contents, and Use with Other EM&V Resources 8
2. EM&V Practices for Quantifying Energy Savings 10
2.1. Establishing a Baseline 10
2.1.1. Discussion 10
2.1.2. Applicable Practices 17
2.2. Selecting a Method 19
Discussion 19
Applicable Practices 20
2.2.1. Deemed Savings 21
2.2.2. Direct M&V 24
2.2.3. Comparison Group 30
2.3. Determining the Time and Location of Energy Efficiency Savings 35
2.3.1. Discussion 35
2.3.2. Applicable Practices 40
2.4. Determining Duration of Savings (i.e., EUL) 43
2.4.1. Discussion 43
2.4.2. Applicable Practices 45
2.5. Verifying Savings 47
2.5.1. Discussion 47
2.5.2. Applicable Practices 47
2.6. Accounting for Additional Aspects of Savings Quantification 48
2.6.1. Independent Variables 48
2.6.2. Interactive Effects 51
2.6.3. Transmission and Distribution Savings and Adders 53
2.6.4. Double Counting 54
2.7. Characterizing Accuracy 56
2.7.1. Discussion 56
2.7.2. Applicable Practices 57
3. EE EM&V Protocols and Guidelines 58
Glossary of Terms 61
References 66
2
-------
Acronyms and Abbreviations
ACEEE American Council for an Energy-Efficient Economy
AE1C Association of Edison Illuminating Companies
AMI advanced metering infrastructure
ASHRAE American Society of Heating, Refrigerating, and Air-Conditioning Engineers
BAU business as usual
C&l commercial and industrial
C&S building energy code and equipment energy standard
CHP combined heat and power
C10IJ California investor-owned utilities
CO? carbon dioxide
CPUC California Public Utilities Commission
CVR conservation voltage reduction
DEQ Departments of Environmental Quality
DER distributed energy resources
DOE (United States) Department of Energy
EE energy efficiency
EERS energy efficiency resource standard
EGU electricity generating unit
EIA (United States) Energy Information Administration
EM&V evaluation, measurement, and verification
EPA (United States) Environmental Protection Agency
ESCO energy services company
EUL effective useful life
EVO efficiency valuation organization
FCM forward capacity market
FEMP (U.S. Department of Energy) Federal Energy Management Program
HVAC heating, ventilating, and air-conditioning
10U investor-owned utilities
IPMVP International Performance Measurement and Verification Protocol
1RP integrated resource planning
ISO independent system operator
ISO-NE ISO New England
kW kilowatt
kWh kilowatt-hour
kWh/h kilowatt-hour per hour
LBNL Lawrence Berkeley National Laboratory
LEAS lifetime equivalent annual savings
LED light-emitting diode
M&V measurement and verification
MW megawatt
MWh megawatt-hour
NAAQS National Ambient Air Quality Standards
NARUC National Association of Regulatory Utility Commissioners
NEEP Northeast Energy Efficiency Partnerships
NGO non-governmental organization
NREL National Renewable Energy Laboratory
3
-------
O&M
operations and maintenance
PUC
public utilities commission
RCT
randomized control trial
RE
renewable energy
RMI
Rocky Mountain Institute
RTF
(Northwest Power and Conservation Council Northwest) Regional Technical Forum
RTO
Regional Transmission Organization
RUL
remaining useful life
SEE Action
State and Local Energy Efficiency Action Network
SIP
State or Tribal Implementation Plan
SPB
stringent practice baseline
T&D
transmission and distribution (system)
TRM
technical reference manual
UMP
(United States Department of Energy) Uniform Methods Project
VSD
variable speed drive
-------
energy officials and EE implementers can also use this EM&V Guidebook as a resource for interpreting
local protocols and guidelines. Finally, representatives of EE-related nongovernment organizations
(NGOs) and advocacy organizations can employ approaches described in this EM&V Guidebook when
examining whether a jurisdiction's EE policy and program regulatory goals are being achieved in a cost-
effective and equitable manner.
1.2. Experience with EM&V
Jurisdictions began to scale up EE as an energy strategy in the 1970s. Since then EM&V has been critical
to EE's success, credibility, and expansion. EM&V methods have been refined and improved overtime as
EE program strategies evolved. The practices in wide use today—and upon which the EM&V approaches
in this EM&V Guidebook are based—are codified in protocols established by the National Renewable
Energy, U.S. DOE's Federal Energy Management Program (FEMP), and several other collaboration-driven
organizations. Today, jurisdictions around the country use EM&V as the metric for determining whether
EE activities are achieving critical electricity reliability, planning, and other policy goals.
The oversight and quantification of EE in each of these contexts may differ slightly depending on
objectives, but always rely on EM&V protocols and guidelines that are robust, transparent, and well-
documented. These oversight mechanisms have generated a set of quantification procedures that are
widely applied to ensure compliance with a broad range of local, state, and regional policy goals and
regulatory requirements.5 The specifics of how EM&V is applied—including the appropriate level of
oversight and review—necessarily vary by policy context and the specific objectives for which EE is
deployed.6
1.3. Scope, Contents, and Use with Other EM&V Resources
1,3.1, Scope of EE Activities Addressed
This EM&V Guidebook is intended to support the range of EE activities that can be included in a SIP/TIP
for purposes of achieving NAAQS compliance, including:
• Building-level EE projects and measures, including those implemented as part of an EE program
(which are currently in place in all 50 states) or to achieve a jurisdiction's energy policy goals;
• EE installed or operating across all customer sectors, including low-income segments of the
population;7 and
• EE implemented by lOUs, public utilities, private companies such as an ESCO, and the owners
and operators of large commercial or industrial (C&l) facilities.
5 For example, the New England Independent System Operator (ISO-NE) established an EM&V protocol for
quantifying capacity savings (expressed in MW) during a limited number of hours, supporting EE as a reliability
resource, and calculating payments to EE providers. See the ISO New England Manual for Measurement and
Verification of Demand Reduction Value from Demand Resources (2014).
6 For more information, see U.S. EPA's EM&V for Energy Efficiency Policies and initiatives (2017).
7 This includes EE activities targeting low-income customers. While low-income EE activities can generate
important energy savings, they are generally implemented to lower the burden of energy costs for a disadvantaged
population. EE programs in this segment are typically designed to deliver co-benefits including but not limited to
improved health, safety, and comfort, beyond just electricity savings.
8
-------
This EM&VGuidebook is not a regulatory or guidance document and therefore does not establish
requirements. The Guidebook also does not apply to or address:
• Market transformation approaches (e.g., the ENERGY STAR Retail Products Platform);8
• EE in the mobile-source sector or implemented onsite at an EGU;
• Estimates of the potential for EE savings that may exist in various sectors of a state or
community's economy;
• Options for implementing, funding, or encouraging EE in a jurisdiction, such as EE policies,
regulations, programs, or projects; or
• Renewable energy (RE), CHP, or other zero- and low-emitting distributed energy resources
(DERs) that generate electricity.
1.3.2. Summary of Contents
This EM&V Guidebook draws from and builds on decades of state, local, and private-sector experience
quantifying and verifying savings from EE projects and measures.9 It defines terms, provides brief
descriptions and context for key EM&V topics, and identifies "applicable practices" for conducting
EM&V. Definitions are compiled into a glossary at the end of this document. For select sections, this
EM&V Guidebook also identifies "considerations for air officials" and presents questions intended to
support dialogue between air officials and their energy counterparts. In addition, the Guidebook
includes a list of existing and complementary EM&V protocols and guidelines.
In providing this EM&V Guidebook, EPA recognizes that the best-practice approaches, protocols, and
procedures that are now used by states, EE implementers, and others will evolve and improve over time
as new technologies and methods emerge, and as the EE marketplace changes. To ensure that it
continues to reflect current practice—and that air officials can continue to have confidence in emission
reductions from EE policies and programs—EPA may periodically provide new versions of the Guidebook
with updated and additional information.
1.3.3. Use with Other EM&V Protocols a Iclelines
Other EM&V guidelines and technical resources are available to support state, local, and tribal officials,
EE implementers, and EM&V practitioners working to quantify EE programs. Several states and ISOs
have their own EM&V protocols for evaluating EE programs funded by energy customers. At the national
level, two widely used EM&V resources are SEE Action's Energy Efficiency Program Impact Evaluation
Guide ("SEE Action Guide," SEE Action, 2012a) and the National Renewable Energy Laboratory's Uniform
Methods Project: Determining Energy Efficiency Savings for Specific Measures ("UMP," NREL, 2018).
The SEE Action Guide establishes basic EM&V definitions and provides high-level descriptions of
foundational concepts, approaches, and methods for quantifying EE savings, avoided emissions, and
other impacts. This EM&V Guidebook maintains consistency with the SEE Action Guide, but also
8 For more information on EM&V for the ENERGY STAR Retail Products Platform, see:
https://www.energvstar.gov/esrpp/emv.
9 EE projects and measures are typically implemented by lOUs, public utilities, private companies such as ESCOs, or
the owners and operators of large commercial or industrial facilities. EE can occur within all sectors of the
economy, including low-income segments of the population and disadvantaged communities.
9
-------
establishes actionable EM&V approaches and "applicable practices" (see Sections 2.1 - 2.7). These
approaches and practices can serve as an information resource for air and energy officials that can
readily be tailored to address a jurisdiction's specific policy objectives and EM&V priorities.
In contrast to this EM&V Guidebook, the UMP provides a detailed set of prescriptive options, methods,
and procedures for quantifying EE savings for specific measure types. It also includes protocols for key
cross-cutting topics such as survey design, sample design, and metering. The UMP therefore provides
EM&V practitioners with a recipe for interpreting and implementing a jurisdiction's overarching EM&V
protocols and guidelines (which can potentially be established or refined using this EM&V Guidebook).
This document is not meant to replace these and other EM&V protocols and guidelines. Rather, it builds
on and compiles existing practices into a single actionable resource supplemented with key questions
and considerations for air officials.
2. EM&V Practices for Quantifying Energy Savings
Quantifying energy savings from an EE activity is a multi-step process, described in Sections 2.1-2.7 of
this Guidebook. The following EM&V topics are addressed:
1. Establishing a Baseline
2. Selecting a Method
3. Determining the Time and Location of Energy Efficiency Savings
4. Determining the Duration of Savings (i.e., effective useful life)
5. Verifying Savings
6. Accounting for Additional Aspects of Savings Quantification
7. Characterizing Accuracy
Each of the sections below defines Key Terms and includes Discussion and Applicable Practices
sections. Where applicable, text boxes highlighting Considerations for Air Officials and Questions that
Air Officials Can Ask are provided.
2.1. Establishing a Baseline
2,1,1, Discussion
Energy savings are the difference between energy consumption with an EE activity in place and the
consumption that otherwise would have occurred during the same period. The consumption that
otherwise would have occurred is called the baseline. Establishing baselines for savings is a key
challenge of EM&V because determining the baseline requires identifying what would have happened
absent the EE activity.
For example, if an EE project involves installing new high-efficiency equipment, the alternatives that
could have occurred absent the EE activity include:
• No change (existing equipment remains in place and unchanged indefinitely);
• Installation of new equipment that is less energy efficient; or
10
-------
• When a dual baseline applies, apply the same code/standards or market baseline as for
replacement on failure for the same equipment type. The code/standards or market baseline
may be different for new construction than for early or on-failure replacement, because code
requirements or market practice may be different for these EE event types.
Maintaining Baseline Specifications
• When markets baselines or an SPB are applied, continually re-evaluate the market or SPB value
to ensure that new EE activities continue to be additional to what is happening naturally in the
market.
When to Establish Baseline Specifications
• Wherever possible, determine and document baselines prior to implementing an EE project or
measure.
Operating Condition Specifications
Calculate savings at operating conditions anticipated over the life of the project or measure. Use one of
the following approaches:
• Assume the operating conditions in the year after the project or measure installation are the
expected average conditions for the lifetime. Calculate baseline consumption at these
conditions.
• In cases where the post-implementation operating conditions are not likely to be typical of the
long-term average, adjust both the observed actual consumption and estimated baseline
consumption to the assumed long-term average operating conditions.
• For weather-dependent energy uses with EE savings accruing over multiple years, adjust both
observed actual consumption and estimated baseline consumption to typical weather
conditions.
2.2. Selecting a Method
Discussion
There are three broad EM&V methods for quantifying savings, including: 1) deemed savings for specific
EE measures, 2) direct measurement and verification (M&V) applied to individual EE projects or
measures, and 3) comparison group methods relying on analysis of consumption data for an affected
group of premises compared to another group. Advanced M&V, a method using automated analysis of
consumption data for either direct M&V or comparison group methods, is discussed in a text box at the
end of Section 2.2.2 below. Best-practice approaches for applying each of these three broad methods
are defined in industry standard protocols or technical guidelines that are commonly used by EE
implementers, oversight agencies, and the firms they hire to quantify and verify savings.
19
-------
spreadsheet, an online searchable database, or similar resource. The term commonly used for such
resources is a technical reference manual (TRM) (U.S. DOE, 2011). As of this document's publication
approximately 28 TRMs are in use across the United States at the state and regional levels (SEE Action,
2017). The methodologies for deriving deemed values can vary across jurisdictions. Some TRMs include
information based on prior-year EM&V. Some TRMs include values based on computer simulations or
engineering algorithms.
Applicable Practices
When to Apply Deemed Savings Methods
• Apply deemed savings methods for relatively simple, well-defined EE projects or EE measures
(such as light bulbs or other electrical equipment) for which the average operating
characteristics that are the basis for the deemed values are well known, or where there is
relatively little uncertainty around average unit savings.
• Do not apply deemed savings methods for unique and custom applications.15 This includes EE
projects encompassing multiple EE measures with complex interactive effects that are
challenging to accurately quantify and document.
How to Apply Deemed Savings Methods
• Implement deemed savings methods by applying the following steps:
1. Establish savings quantification formulas by establishing deemed parameter values,
parameter applicability, and conditions for applying the formula. Deemed parameters
may include per-unit savings values or average values of savings calculation formula
inputs such as hours of use or equivalent full-load hours. The simplest form of a deemed
savings calculation formula is savings per unit multiplied by the number of units.
2. Apply the formulas and documented measure counts to calculate pre-verified savings.
3. Perform installation verification to confirm that units were installed, verify unit
quantities, and confirm appropriate application of deemed values and calculations.
Installation verification may consist of reviewing independent third-party reports on
measure installation rates based on customer surveys and/or onsite verification that
installations were installed according to specification. The verification process may be
based on a valid statistical sample that represents the entire population of EE projects
or EE measures, that is then scaled appropriately to the population.
4. Apply the formulas, parameters, and verified units to determine the total quantified
savings.
• Ensure that deemed values are:
o Based on EE activity type, applicability conditions, assumptions, calculations, and
references that are publicly documented and available.
15 For more complex EE projects or EE measures with significant savings variability, consider the application of
direct M&V or comparison group methods instead of deemed savings. While direct M&V and comparison group
methods may include the use of deemed values for certain parameters used in the calculation of savings, the
incorporation of direct measurement or consumption data analysis moves such methods outside of the deemed
savings category.
22
-------
o Quantified based on typical electricity savings and other factors that determine such
values over the lifetime of the EE measure, such as average occupancy, typical weather,
typical operating hours, and EUL.
o Developed and vetted by independent third parties and developed applying analytical
methods that are widely considered acceptable for the measure, purpose, and data
sources (such as prior metering studies).
o Appropriately adjusted if borrowed from secondary sources from other geographic
areas.
• Apply deemed savings methods as follows:
o Apply deemed values only for EE projects or EE measures similar to the specific EE
projects or measures for which the values were developed.
o When a database or TRM with deemed savings values is updated based on new
information, apply the revised deemed values and quantification methods to EE projects
or EE measures implemented after the effective date of the update. It is generally not
necessary to apply the revised deemed values and quantification methods to EE projects
or EE measures for which EM&V has already been completed, unless revision of prior
completed EM&V is desired and the revised TRM values are considered to be applicable
to the period covered by the prior completed EM&V.
o Use deemed savings values that were calculated as the difference between the
electricity used by the EE project or EE measure and the appropriate baseline for each
EE project or EE measure, as defined based on Section 2.1.
o If savings relative to a particular baseline are desired, and the deemed savings values,
parameters, or formulas produce quantified electricity savings relative to a different
baseline for that EE project or EE measure, document and justify needed adjustments to
the applicable deemed savings values, parameters, or formulas to ensure that electricity
savings are quantified relative to the appropriate baseline. For example, if savings are
needed relative to a SPB and the deemed values are relative to a code/standards
baseline that is less stringent than the SPB, document the adjustments needed to
translate the deemed savings values to the SPB.
• Ensure that savings are adjusted for independent variables that affect energy use, as relevant, in
accordance with Section 2.6.1 and that they account for the interactions between individual EE
measures that comprise the EE project.
• Review the deemed savings values and formulas periodically (e.g., at least every three years),
updating them as necessary to reflect more recent and/or accurate data, and applying them to
all EE projects or EE measures that are installed or begin operating after such an update occurs.
Documentation
• Describe why deemed savings values and formulas are appropriate for each type of EE project or
EE measure.
• Indicate the conditions for which each deemed savings value, parameter, or formula is
applicable (e.g., climate, building type, end use, and measure implementation mechanism).
23
-------
• Include information on the assumed baseline technology and conditions applied to establish the
deemed savings values, to ensure that deemed savings values reflect the appropriate baseline.
• Describe the baseline specification as determined in Section 2.1 for each deemed savings value.
• To increase transparency, document the deemed savings values and formulas in a freely
available database or spreadsheet (e.g., a TRM) that is accessible on a public website, specifies
the conditions for which each deemed savings value or formula may be applied (e.g., climate
zone; building type; and implementation strategy, such as retrofit, replacement on failure, or
new construction), and specifies the source of each deemed savings value or formula.
Resources
When applying deemed savings methods, apply one or more best-practice protocols and guidelines.
Examples include but are not limited to:
• Status and Opportunities for Improving the Consistency of Technical Reference Manuals (ACEEE,
2012)
• Behind the Curtain: Characterization of Measure Technologies within Technical Reference
Manuals (ACEEE, 2016a)
• Technical Reference Manuals Best Practices from Across the Nation to Inform the Creation of
the California Electronic Technical Reference Manual (eTRM) (ACEEE, 2016b)
• Approach to Texas Technical Reference Manual (PUCT, 2013)
• Energy Efficiency Program Impact Evaluation Guide (SEE Action, 2012a)
• The Northwest Power & Conservation Council Regional Technical Forum (RTF, 2018)
• Using Deemed Savings and Technical Reference Manuals for Efficiency Programs and Projects
Webinar (LBNL and U.S. DOE, 2017)
2,2.2, Direct M&V
Discussion
Direct M&V refers to a set of methods that involves obtaining measurements from an individual EE
project or EE measure installation site as a basis for quantifying savings. For direct M&V-based savings
quantification of individual EE projects or EE measures, the selected measurement technique is applied
to a specific piece of equipment, for the site as a whole, or both. When applying direct M&V to an EE
program or group of EE projects or EE measures, analysis may be conducted for each project or measure
in the group. It may also be conducted, as is more common, for a sample of projects or measures, with
the sample results then used to quantify savings for the full group.
The application of direct M&V methods can establish accurate savings for most EE activities. However,
these methods tend to be more expensive than deemed savings or comparison group methods. The cost
for direct M&V is driven by factors such as the measurement equipment required, the measurement
duration, the number of sample points needed at an individual project or measure site, and the number
and complexity of sites to obtain the targeted accuracy. The selection of direct M&V versus other
methods therefore involves tradeoffs between cost and level of uncertainty in the EE savings values.
24
-------
model. For example, one-time changes to a building's occupied floorspace, operating shifts, or
equipment types may involve non-routine adjustments.
Analysis of whole-premise metered consumption data (Option C of the IPMVP) may use similar building-
level models to those applied for comparison group analysis described in Section 2.2.3. Two differences
between building-level models for site-level direct M&V and the comparison group approach are:
1. Site-level direct M&V is designed to estimate savings relative to the appropriate baseline for the
individual site. The comparison group analysis produces savings for a program or group of
similar projects.
2. Site-level direct M&V uses additional information either to confirm that no other changes
affected the facility over the analysis period, or else to support customized analysis to make any
non-routine adjustments to savings estimates required to address changes. This type of custom,
non-routine adjustment is not typically included in comparison group analysis.
Applicable Practices
When to Apply Direct M&V
• Consider the resources available for EM&V and the need for accurate savings values when
considering the application of direct M&V.
o If resources are available and there is a need for highly accurate savings values, direct
M&V may be the most appropriate quantification approach.
o To determine cost, consider the measurement equipment required, the measurement
duration, the number of sample points needed at an individual project or measure site,
and the number and complexity of sites to obtain the targeted accuracy.
• Apply direct M&V only when the physical address(es) of installed measures are known, and
these facilities (or a sample of them) are accessible for conducting the necessary data collection.
• Apply direct M&V methods for:
o EE activities for which reliable deemed savings approaches are not available or not
applicable, and for populations of EE projects or EE measures that are not in sufficient
number or homogeneity for comparison group EM&V methods to be applicable or
feasible, such as because a control group cannot be identified.
o EE activities that have high savings variability or uncertainty due to differences in
physical or behavioral characteristics across individual sites and applications. Large,
complex projects or installations typically have such variability or uncertainty.
o Other situations where the cost is justified by the value in terms of improved reliability
and confidence in the results.
How to Apply Direct M&V
• To quantify savings from an EE program or portfolio of related EE projects applying direct M&V,
do one of the following:
o Conduct direct M&V for each project or measure in the program and sum the results to
determine program-level savings.
26
-------
o Conduct direct M&V for a randomly selected sample of sites and apply statistical sample
expansion16 to determine program-level savings from the sample results.
• Refer to the IPMVP, listed in Section 3, for two direct M&V options applied in the EE industry:
o Retrofit isolation: Assessing savings from each EE measure individually (IPMVP Options
A& B).
o Whole facility: Analyzing savings from each EE measure in a project/facility collectively
(IPMVP Options C & D).
• If statistical sampling and expansion will be applied, ensure there is a large enough sample of EE
projects within an EE program, a sufficient number of EE measures within an individual EE
project site, and sufficient measurement quality across the EE program to meet statistical
accuracy requirements.
• Ensure that direct M&V is conducted by staff who have the appropriate expertise, including:
o Metering and measurement equipment selection, installation, sensing, and calibration.
o Statistical sampling and estimation methods for data collection related to facility
electricity use.
o Engineering analysis for facility electricity use, including baseline specification.
o Field data collection quality control.
• When measured or metered data results are combined with deemed parameters, ensure that
the appropriate deemed parameters are applied for each metered case to ensure accurate
results.
• Ensure that savings quantified by direct M&V methods apply the appropriate baseline as
defined in Section 2.1.17 Before selecting a direct M&V method for EE activities with a baseline
that is not existing conditions,18 ensure that a viable approach exists for modifying existing
condition baseline energy use measurements to equate to the correct baseline. In some
instances, this may not be viable. In other instances, modification can be made. For example:
o With a motor replacement project where the baseline is a new standard-compliant
motor, IPMVP Options A and B can be used to measure existing motor electricity use.
These measurements can then be adjusted using a ratio of the efficiencies of a
standard-compliant motor and the existing motor efficiency.
o With a whole house retrofit project, where the baseline is a building energy code,
IPMVP Option D can be used with a baseline building energy model calibrated to the
16 Sample expansion is statistical estimation of results for the whole group of interest—in this case the EE program
or portfolio—based on the results for the random sample.
17 Direct M&V is often conducted by equipment installers to confirm that measures are working correctly or to
demonstrate to utility customers that they are achieving the expected improvements from the new equipment.
These applications of direct M&V tend to use the existing equipment as the baseline, which may or may not be the
appropriate baseline for the intended EM&V purpose.
18 In the context of utility EE programs and privately implemented EE activities, direct M&V methods are commonly
applied for EE projects and EE measures for which existing condition baselines are appropriate.
27
-------
post-retrofit whole-premise consumption and then adjusted to code-compliant levels to
estimate consumption at the baseline condition.
• Ensure that savings are adjusted for independent variables that affect energy use, as relevant, in
accordance with Section 2.6.1.
• Quantify savings for the long-term, post-installation operating condition. If ongoing
measurement is not used, use appropriate engineering and statistical methods to adjust the
metered and measured data to the long-term annual average condition, normalizing results for
weather, productivity, and other routine and non-routine factors as needed.
• Follow best practices for statistical sampling of sites, EE projects, or EE measures. Also follow
good practices for sample design, sample management, and sample expansion to the full EE
project or full EE program level.
• Because the quantification process ordinarily involves direct observation of installed equipment
or of its effects on whole-facility consumption, a separate step verifying implementation is not
needed for the EE measures subject to this process.
• If direct M&V is conducted for a sample of EE projects or EE measures, an option that can
provide more certainty of savings is to conduct (less expensive) verification for a larger sample
than the direct M&V sample. In this case, combine the quantified savings per measure from the
direct M&V with the verified quantity of measures (e.g., equipment counts) to determine the
total quantified savings.
• Follow rigorous quality assurance, quality control, and training procedures.
• For an EE activity that is an operational improvement, derive the baseline from the efficiency of
the affected equipment without the operational improvement. If the operational improvement
can be cycled on and off at intervals over a full year, calculate the baseline from the periods
when the improvement is off. This approach can be especially useful for EM&V of grid-side EE
activities.19
• Use tools designed to apply an automated analysis of consumption data20 to quantify savings
consistent with guidelines and protocols for the applicable direct M&V method, as well as the
practices described in this section. In particular, describe quantification methods transparently
and show how the automated analysis can provide savings relative to the appropriate baseline
specification. (See the text box on page 30.)
Documentation
Document the following when applying direct M&V:
• Approaches for determining, identifying, and isolating measurement variable(s), including a
description of the measurement variable and why was it selected (e.g., duty factor for a
residential air conditioner, on/off schedule for an industrial process).
19 Examples of grid-side EE activities include voltage and VAR optimization (WO) and conservation voltage
regulation (CVR), which produce electricity savings by reducing voltage at various points along the transmission
and distribution system.
20 Examples of such tools and their uses and performance in EM&V and other contexts are described in DNV GL
(2015); LBNL (2015); and ACEEE (2015).
28
-------
• Sampling and expansion procedures, including how the sample was selected, how the number
of sample points was determined, how the case weights were developed, identification of and
reasoning for the coefficient of variation used to design the sample, how the individual
measurement results were expanded to the population, and how the statistical error metrics
were quantified (e.g., confidence and precision levels).
• Planning documents that describe how direct M&V will be applied at the level of the EE activity,
as appropriate. Planning should address questions such as: What type of direct M&V approach
was applied (e.g., one or more of the four IPMVP methods, a combination, an alternative
method)? How were baselines selected and estimated, including how they conform to the
specifications in Section 2.1? How were metering and monitoring conducted, including for how
long? How was the data collected? What quality assurance and quality-control procedures were
applied? How were electricity savings estimated?
• Reporting procedures, including how the savings results were compiled to produce overall
reported savings estimates relative to the appropriate baseline.
Resources
When applying direct M&V methods, apply one or more best-practice protocols and guidelines.
Examples include but are not limited to:
• International Performance Measurement and Verification Protocols (IPMVP) (EVO, 2016)
• UMP, Chapter 11 - Sample Design Cross-Cutting Protocols (NREL, 2017b)
• California Energy Efficiency Evaluation Protocols (CPUC, 2006), see Measurement and
Verification Protocol and Sampling and Uncertainty Protocol
• California Evaluation Framework (CPUC, 2004), see Chapter 7: Measurement and Verification
and Chapter 13: Sampling
• FEMP protocols and guidelines (U.S. DOE, 2018)
• ASHRAE protocols and guidelines (ASHRAE, 2018)
References for statistical sampling and estimation include the following:
• UMP, Chapter 11 - Sample Design Cross-Cutting Protocols (NREL, 2017b)
• Load Research Manual (AEIC, 2017), see Chapters 4 "Sample Design and Selection" and Chapter
7 "Data Analysis"
• Sampling Techniques (Cochran, 1977)
• Survey Sampling (Kish, 1995)
• Introduction to Variance Estimation (Wolter, 1985)
• Encyclopedia of Survey Research Methods (Lavrakas, 2008)
29
-------
Comparison group methods involve the analysis of whole-premise metered consumption data22 for a
group of customers who participate in an EE program (the treatment group or program participants) and
another group who did not participate (the comparison group). The comparison group provides a basis
for quantifying the consumption or change in consumption the participating group would have had
without the EE program (baseline for net savings), or without the specific projects and measures
installed through the program (baseline for gross savings). Savings is then the difference between
consumption with the EE activity in place and the estimated consumption without—that is, the
difference from the gross or net baseline. Thus, depending on how the analysis is structured, and which
baseline it provides, the savings estimated may be the gross effect of the participants' projects or
measures, or may be the net effect of the program.
When comparison group analysis is correctly applied, the analysis provides a "no-EE-activity"
consumption estimate that represents the combined effect of the changes other than the EE activity
being measured. To the extent the comparison group adequately accounts for other changes on
average, explicit knowledge of and adjustment for these other changes is not necessary.
An appropriate comparison group has minimal identifiable theoretic or empirical systematic differences
from the treatment group, apart from the effect of the EE activity itself. The ideal basis for establishing a
comparison group is by random assignment prior to implementing the EE activity. This technique avoids
the potential for bias and has statistically measurable accuracy. However, random assignment is
compatible only with limited types of EE activities.
When comparison groups are established by methods other than random assignment, two common
risks to comparison group validity should be addressed. These are applicability and self-selection.
Documenting how the comparison method produces savings relative to the appropriate baseline
includes explaining how these two risks are addressed by the comparison group specification and
analysis.
1. Applicability - In addition to being similar to those who participate in an EE activity in other
ways, the comparison group should consist of energy-using consumers or facilities for which the
EE activity would have been applicable. Identifying such consumers or facilities can be
challenging.
2. Self-Selection - Even if the entire pool of consumers is considered eligible, those who choose to
implement an EE activity at a particular time may be different from those in the general
population in ways that can affect electricity use. For example, participants in an EE program
who are interested in installing energy-efficient equipment may have more efficient buildings to
begin with and their consumption may respond differently than that of the typical non-
participant to changes in weather, the economy, or other factors affecting all customers.
22 Analysis of whole-premise metered consumption data can also be applied as a site-level direct M&V method
(IPMVP Option C) as described in Section 2.2.2. Additionally, Advanced M&V (see text box on page 30) is of
potential interest for EE implementers. The automated consumption data analysis tools used in Advanced M&V
may be used to implement direct M&V method "Option C" of the IPMVP protocol as described in Section 2.2.2, or
comparison group approaches as described in Section 2.2.3, provided they are applied consistently with the
guidance for those methods. These tools and approaches are not a different category of EM&V method. Instead,
they can be a means of implementing whole-building consumption analysis for individual cases that is consistent
with the direct M&V category of methods.
31
-------
After random assignment, a next-best basis for a comparison group is a "natural experiment" in which
there are two very similar groups. An example is one group who has a particular EE program offering
available to them and another group who does not. Another example is to implement the natural
experiment over time, using customers who participate in a subsequent year as a comparison group for
the participants who participate in a current year. This approach can be effective provided:
• The EE program and other major economic conditions are similar over the measured year and
the year of subsequent participation.
• There are minimal changes associated with the decision to participate in an EE activity in a
particular year.
Where neither random assignment nor a natural experiment are possible, a matched comparison group
is commonly used. With this approach, one or more "nearest matches" are selected for each participant,
from the overall pool of customers who were eligible for the program but did not participate. The basis
for establishing a nearest match can include any known characteristics such as geography or premise
type, as well as similarity of energy consumption patterns in the pre-participation period. Matching
controls for some but not all self-selection effects.
In jurisdictions where advanced metering infrastructure (AMI) systems or "smart meters" have been
installed for the applicable customer sectors, using daily or hourly consumption data can reduce
statistical uncertainty for the estimated savings. This improvement can make it possible to apply
comparison group methods for smaller magnitude savings than would otherwise be possible. On the
other hand, use of daily or more frequent data involves more complex techniques to determine
correctly the statistical accuracy of the savings estimate.
Comparison group methods are most commonly applied in contexts where the baseline for gross savings
is based on existing conditions. This is because directly calculating savings relative to a market or codes
baseline would require a comparison group of customers who recently installed the market or code-
level new equipment; such customers are challenging to identify. However, with the appropriate
analysis structure, baselines other than existing conditions can also be addressed by comparison group
methods. See Goldberg, Michelman, & Dickerson (1997) and Agnew, Goldberg, & Wilhelm (2009) for
examples.
Applicable Practices
When to Apply a Comparison Croup
• Apply comparison group methods to measure impacts of an EE program or portfolio of projects
as a whole, not to determine savings for individual EE projects or EE measures.
• Apply comparison group methods only if the following are all true:
o The proposed comparison group with the planned analysis structure will provide a good
representation of the participating group absent the EE activity.
o The expected statistical accuracy is adequate based on a power analysis or on the
results from a prior study with similar analysis and conditions to the planned study.
32
-------
(Appendix D of the California Protocols provides an example of how this analysis can be
implemented).23
o Whole-facility metered electricity consumption data are available for the participating
and comparison groups, with at least bimonthly meter reads spanning summer, winter,
and shoulder periods before and after the EE activity.
o Key likely systematic differences between the comparison group and participant group
can be controlled for via observable variables.
o There are minimal identifiable theoretic or empirical systematic differences from the
treatment group, apart from the effect of the EE activity itself.
o The comparison group and analysis method yields savings relative to the appropriate
baseline, per Section 2.1. If this condition is met, separately determining the baseline
efficiency of individual pieces of equipment is not needed.
How to Apply a Comparison Croup
• Ensure that practitioners hired to prepare such analysis have the specialized expertise needed to
implement a random assignment process or specify a comparison group, as well as the expertise
needed to perform analysis to isolate the intervention effect to produce savings relative to the
appropriate baseline.
• Where possible, specify comparison groups applying random assignment following best
practices such as those described in resources from SEE Action (2012) and CALMAC (2016).
Specify the random assignment design in advance of delivery of the EE activity, and ensure that
the EE delivery process follows the design and random assignments.
o If a random assignment process is not practical for the program:
¦ Specify the basis for establishing the comparison group.
¦ Describe likely self-selection effects and qualitatively assess the resulting effects
on savings.
o If random assignment is applied:
¦ Document the random assignment design.
¦ Document the steps taken to ensure delivery of the intervention according to
the random assignments.
• In cases where the comparison group for a particular program-year or set of EE activities is re-
analyzed in successive years to provide direct quantification of savings from surviving EE
projects or EE measures, include a discussion of the basis on which the comparison group
remains appropriate and valid.
• Describe the calculation methods transparently, and provide the basis for interpreting the
results as savings relative to the appropriate baseline.
23 Qualitatively, attaining good statistical precision depends on having sufficiently large savings with a sufficiently
large and homogenous group of facilities or installations, such as several hundred residential or small commercial
customers. That is, the magnitude of expected savings is large compared to the expected random differences
between the participant and comparison group averages (CPUC, 2006).
33
-------
• Design the sample sizes to be large enough to ensure statistically significant savings values.
• Ensure that savings are adjusted for independent variables that affect energy consumption, as
relevant, in accordance with Section 2.6.1.
• Collect sufficient consumption data from before and from after the intervention to include
observations from each season and all operating patterns in each of the two periods (before and
after). Typically, this coverage involves 9 to 12 months of data from each of the two periods.
• If the comparison group consists of participants who did not replace equipment and the
appropriate baseline corresponds to standard new equipment, conduct a separate adjustment
to produce savings relative to the correct baseline. For examples of adjustment processes, see
Goldberg, Michelman, & Dickerson (1997) and Agnew, Goldberg, & Wilhelm (2009).
• If daily or more frequent consumption data are used, document the steps taken to ensure
correct calculation of statistical accuracy.
• If applying tools designed for automated analysis of consumption data24 to quantify savings by
comparison group methods, ensure the general considerations described in this section are
addressed. In particular:
o Describe the calculation methods transparently.
o Clearly describe the comparison group selection process and show the process is
appropriate for the EE activity.
o Show how the analysis can provide savings relative to the appropriate baseline
specification.
Documentation
Include the following as part of a comparison group analysis documentation:
• If random assignment is applied, a description of the randomization design, how it was
implemented, what steps were taken to ensure adherence to the random assignments, and
what deviations, cross-contamination, or dropouts occurred.
• The rationale for the comparison group specification, what the comparison group represents,
what conditions are controlled for by the analysis.
• The estimation method and rationale, including how the analysis provides a valid estimate of
savings with respect to the appropriate baseline, per Section 2.1.
• The metrics of statistical accuracy.
• A description of the data screening criteria used, and the data attrition at each screening stage.
• The response rates if survey data are used in the analysis.
• A discussion of the threats to validity of the analysis, including systematic errors and their
potential magnitude.
24 Examples of such tools and their uses and performance in EM&V and other contexts are described in DNV GL
(2015), LBNL (2015), and ACEEE (2015).
34
-------
• Understanding how targeted investment in EE savings can alleviate electric grid congestion (or
natural gas pipeline capacity constraints)
Given the benefits of using time and locational data for different objectives, jurisdictions can consider
whether and how to design EE programs and EM&V plans to ensure that the necessary data are
collected to quantify the time and locational value of the EE programs or measures. State and local
agencies and their utility partners may also wish to assess whether supplemental research or evaluation
is needed.
Time Periods of Interest for EE Savings
The amount of energy used in homes and businesses varies based on the hour, day, and season, as well
the weather. Grid operators and electric utility companies work together in real time to match EGU
generation with fluctuating demand during these periods and at different locations. When demand
increases, operators have historically responded by increasing production from EGUs already in use, by
purchasing additional power or by adding generation units that are available on reserve or standby (i.e.,
already running at a low level).25
For regional electricity grids, peak demand is defined as the highest electric use during the year or other
period of time. Electricity supply tends to be constrained at times of annual peaks, because systems are
built large enough to meet peak demand with limited excess capacity. These conditions typically result
from extreme summer or winter weather, depending on the region. For example, many areas
experience their highest or peak need for electricity on hot summer afternoons (summer peak) when
homes and businesses maximize their air-conditioning use. In areas where electric heating is prevalent, a
winter peak may occur during very cold weather. In areas where natural gas heat is dominant, the
combined demand for natural gas heating and natural gas-fueled power generation during periods of
extreme cold can constrain regional or local supplies.
In each of these cases, the goal of system planning is to ensure that generation and transmission and
distribution (T&D) capacity are sufficient to serve peak demand. A key step in this process is assessing
the tradeoffs among resource choices such as adding generation capacity, adding transmission capacity
to increase import capability, and reducing demand. Better understanding when system peaks are likely
to occur and whether EE can help reduce these peaks is therefore a common reason why jurisdictions
are interested in evaluating the timing of EE savings.
Supply constraints can also occur not because demand is high but because supply is low. For example,
certain areas of the country that experience supply-constrained conditions—due to a mismatch
between the time when electricity use is highest and the time when large quantities of low-cost solar
generation are available26—are examining ways in which targeted demand reductions from EE may be
useful in mitigating such constraints.
25 For more information about the U.S. electricity system and its impact on the environment, see:
httpsi//www.e|3a.gov/energv/aboyt-ys:electricity-system-and-its:impact-environment
26 Areas of the country that rely heavily on renewable electricity may experience low "net loads" during sunny
afternoons when EGUs are ramped down to accommodate abundant solar generation. This situation can result in
system constraints in the early evening as the sun goes down, the workforce returns home, and electricity loads
spike. During these hours, utilities and system operators must quickly ramp up EGUs to replace solar generation
36
-------
• Congestion Zone: Congestion zones are areas in which local constraints on the distribution grid
impede the flow of electricity. Adding or upgrading the existing T&D infrastructure in these
areas can be expensive and disruptive. Utilities can avoid or delay the need for T&D upgrades by
investing in geographically targeted DERs, including EE. In these cases, the costs and benefits of
the DER alternatives are compared with those of new or upgraded T&D infrastructure.
Quantifying and tracking EE savings by geography may be accomplished by geo-coding EE projects and
measures and aggregating them to the appropriate geographic scale. For certain EE project and measure
types, the physical location is likely to be known and data are likely to be readily available (e.g., for a
typical program where rebate applications provide customer addresses). In other cases (e.g., an
upstream mass-market residential lighting program where the program discounts prices paid by all
customers and has no way to identify purchasers), the geographic distribution of savings impacts needs
to be determined by market characterization studies or models.
For planning and analysis, the geographic distribution of savings can be combined with characteristics
and conditions of the electricity grid serving the area in which the EE project or measure is located.
Examples of the circumstances in which geographic differentiation of EE savings may be useful are
provided in Table 5 in Section 2.3.2.29
Additional Considerations for Quantifying the Time and location of EE Savings
One consideration that may influence a jurisdiction's decision to collect and analyze time and location
savings data is that using EE to displace the operation of fossil-fuel EGUs can provide air quality benefits
during particular periods or at particular points of interest. EE savings may be even more valuable if the
associated fuel costs and air pollutant mitigation costs are taken into account. In general, jurisdictions
have found that collecting and analyzing the time and location of EE savings is more valuable in areas
where air quality concerns are present and generation is more costly (see Considerations for Air Officials
box).
Determination of the time and location of savings is easier and more useful if these factors vary
predictably over the period and in the location being analyzed. Commonly used costing periods (time
blocks over which avoided costs are similar) include on- and off-peak periods in winter and summer but
may also be derived for weekends/holidays or even individual hours for the whole year if that level of
detail is useful for forecasting. Peak demand savings are typically quantified as the average hourly EE
savings (kWh/h or kW) during an on-peak costing period.
Another consideration is that time blocks or locations of interest may vary depending on the planning
objective. For example, the time blocks for electricity system planning may not be the same as those of
most interest for estimating emission reductions. That is, while avoided capacity costs and emission
rates may both vary over time, these factors may not vary in the same way. In such cases, jurisdictions
may wish to establish two or more distinct sets of time blocks for planning—one for costing and one for
emissions.
29 The National Standard Practice Manual is one resource that provides more information on how jurisdictions can
quantify and apply time and locational EE values for various objectives and circumstances.
39
-------
¦ That the quantity of installed EE measures still in place and operating will be
determined each year, via empirical data collection.
¦ How the annual verification results will be used to determine the quantified and
verified savings for each year.
¦ A methodology for empirical data collection to be applied to determine the
number of EE projects and EE measures that remain installed and operating at
the end of each preceding reporting period.
o For the initial year of installation and for each year thereafter, conduct verification in
accordance with Section 2.5 to determine what portion of the total installed EE projects
or EE measures remain in place and operable. Quantify savings based on the portion
that is found to still be in place and operable.
Pre-Specified E lit
• If pre-specified EULs are used, document the source of each pre-specified EUL for EE equipment
installation or operational improvement, consistent with one of the three following categories.
Note that the first category is preferable to the second, which is preferable to the third.
1. Based on a recent, applicable persistence study conducted according to the provisions
of a best-practice protocol for determining EUL values and with EUL estimated at the
appropriate level of confidence and precision. An example of a best practices protocol
for such studies is the Effective Useful Life Evaluation Protocol of the California Energy
Efficiency Evaluation Protocols (CPUC, 2006).
2. Based on an applicable TRM, meeting the Applicable Practices for specifying and
updating deemed values under Section 2.2.1.
3. Based on an independent third-party laboratory lifetime testing protocol.
• When a pre-specified EUL is used, the following lifetime equivalent EUL calculation may be
applied to simplify annual quantification for dual baseline or combination measures. To apply
this calculation, use a single lifetime equivalent annual savings (LEAS). Apply that savings
quantity for each year from the first year of a dual-baseline EE project or EE measure installation
through the full EUL, or for the longest EUL of a combination of measure denoted below by
32 The LEAS may be calculated as follows:
• For a dual baseline measure with annual savings Si from the first year through the RUL, and annual
savings S2 for the remainder of the EUL, calculate the LEAS as: LEAS = (SiRUL + S2(EUL-RUL))/EUL
• For a combination measure with annual savings contributions Sc with EULs EULC for different measure
components c, the LEAS is quantified as: LEAS = Sc(ScEULc)/EULmax
• The LEAS formulas may be applied to successive levels of aggregation of measures using a previously
quantified LEAS in place of the savings Sc, and the corresponding full EUL or EULmax on the right-hand side
of either formula.
46
-------
Controlling for the independent variables means ensuring that:
• The quantified savings do not inadvertently include effects of changes in independent variables.
• The savings are quantified for correct values of the independent variables.
Independent variables are controlled for either by confirming that they are constant over the
quantification periods, or by explicitly adjusting consumption or savings calculations to what would have
occurred at other levels of the variables, applying engineering or statistical methods. For independent
variables that are constant over the periods of interest—and that are consistent with assumptions used
when applying one of the three allowable EM&V methods—no explicit analysis or adjustment is called
for.
As described in Section 2.1, determination of the baseline consumption level depends on both the
baseline and the operating conditions. Operating conditions are specified in terms of independent
variables.
Many EE activities affect equipment and systems in place, but do not affect how the equipment or
facility is used (e.g., hours that a piece of equipment is operating). For such activities, the operating
hours or other indicators of how the equipment is used are among the independent variables that
should be considered in calculating savings. Operating conditions for determining baseline consumption
are the post-installation operating conditions.
Other EE activities, such as installation of equipment control systems or new operating practices, do
affect how equipment or facilities are used. For these activities the operating pattern for determining
baseline consumption consists of the practices that would have been in place during the post-
installation period without the effect of the EE activity. In these cases, the independent variables that
are important to control for may be due to the level of activity in the facility, rather than the runtime of
the equipment. Since the runtime of the equipment is affected by the EE activity, it is not an
independent variable.
For example, if the efficient lighting installed does not affect hours of lighting use, baseline consumption
is calculated for the post-installation hours of use. If the efficient lighting activity includes new controls
to reduce hours of lighting use, baseline consumption is calculated for the hours of lighting use that
would have occurred in the post-installation timeframe absent the new controls. If the facility operating
hours are different between the pre- and post-installation periods, the hours of lighting use for the
baseline consumption calculation are based on the hours of use that would have occurred in the post-
installation period, if the lighting controls were not present.
Each of the three EM&V methods described in Section 2.2 has a mechanism for accounting for
independent variables. For deemed savings values, independent variables are implicitly controlled for
through the associated applicability conditions. For direct M&V, these variables are adjusted for via the
use of regression analyses, computer simulation modeling, or engineering calculation (non-routine)
adjustments. For comparison group methods, independent variables are controlled for through the
comparison group specification and consumption data regression analyses.
49
-------
Applicable Practices
• Identify the independent variables that affect energy consumption and savings for the EE
activity. At a minimum, consider the following and control for them as described below, unless
they can be assumed to be constant over the life of the EE activity, or they will not affect energy
savings for the activity:
1. Weather
2. Equipment or facility hours of operation
3. Facility activity level as measured by variables such as occupancy, number of shifts,
manufacturing production level, or number of meals served
• Document the methodology for adjusting electricity consumption and savings values to account
for the effects of independent variables that can affect energy consumption over the EUL of the
EE project or EE measure.
• Within a single EE program, quantify savings for the constituent EE projects or EE measures
using consistent assumptions for independent variables across different projects and measures.
For example, use consistent forecasts of future weather within a given geographic area, and use
consistent operating hours assumptions within a given market segment. Assumptions may vary
across market segments and geographies based on known characteristics.
• Quantify EE savings using values of independent variables that are expected to apply over the
life of the EE activity, applying one of the following two approaches:
1. Actual conditions that exist over the period when EE savings occur, if these conditions
are measured throughout the EUL (e.g., via ongoing direct M&V or annual verification).
¦ With this approach, adjust baseline electricity consumption data to reflect
actual independent variables observed after the measure is in place and fully
operating.
¦ Examples of independent variables based on actual post-installation conditions
are:
• Observed weather conditions for a residential heating efficiency project
• Observed occupancy rates for a commercial building lighting efficiency
project
• Observed equipment production rates for an industrial efficiency
project
2. Normalized or standardized (typical) conditions that can be reasonably expected to
occur throughout the EUL.
¦ With this approach, both baseline and performance period data on electricity
consumption are normalized to data on the independent variables, where
reasonable and appropriate. Examples of normalized independent variables
based on typical conditions are:
• Typical weather conditions for a residential heating efficiency project
50
-------
o Have multiple EE measures been installed in the same facility at the same time?
o Do installed EE measures indirectly affect energy use of another end use or system?
o Are the EE projects or EE measures influenced by more than one EE program?
• For multi-measure effects, address the interactive effect in one of two ways:
o As an integrated calculation. Determine consumption with the combination of
measures and without any of the measures in place, and take the difference.
o As a sequence of EE measure-specific calculations. In this case, the order in which the
measures are assumed to be installed matters. In the example above, taking the
measures in the indicated order, savings would be calculated for the following:
¦ The high-efficiency electric equipment by itself
¦ The building shell improvements with the high-efficiency electric equipment
included in the baseline specification
¦ The building controls, with the high-efficiency equipment and building shell
improvements included in the baseline specification
Both methods should produce the same savings for the combination, but the second method
allocates savings to the separate EE measures, according to the assumed installation sequence.
If comparison group methods are used to quantify savings, do not make an additional
adjustment for multi-measure effects on the same fuel. With these methods, these interactive
effects are automatically incorporated in the savings calculations
• Calculate other end-use interactive effects using methods appropriate to the broad EM&V
methods used to quantify savings.
o Identify the other end uses affected by the projects and measures, and calculate the
associated effects. Apply the UMP (NREL, 2018) or other applicable protocols and
methods. (See for example UMP Chapter 2 on Commercial and Industrial Lighting
Evaluation.)
o If other end-use interactive effects are treated as zero, justify why this is an appropriate
assumption.
o If deemed savings methods are used to quantify savings, include other end-use
interactive effects in the deemed savings values or separately estimate these effects
applying deemed methods.
o If direct M&V methods are used to quantify savings:
¦ Quantify other end-use interactive effects explicitly if methods based on sub-
facility measurements, such as isolated retrofits or partially isolated retrofits,
are applied.
¦ Incorporate other end-use interactive effects directly into the savings
calculations if building simulation is applied. Most building simulation tools and
approaches are designed to incorporate interactive effects in their savings
calculations.
52
-------
across the country to avoid double counting of EE activities. The purpose of these steps is to avoid the
following circumstances:
• Savings from a single EE activity being claimed by more than one EE implementer. For example:
o Some or all savings from the same retrofit being claimed both by a residential behavior-
based program and a retailer point-of-sale incentive program.36
o Savings from a single retrofit project being claimed by a utility incentive program and
the ESCO that implemented the retrofit.
• Two or more EE activities operating during different years both claiming savings from the same
EE projects or EE measures.
• Two or more EE activities claiming savings that result from interactive effects between EE
projects or EE measures, as described in Section 2.6.2.
• Inconsistent baselines across a portfolio of EE programs. For example:
o One EE program claiming savings from enacting a Building Energy Code and Equipment
Energy Standard (C&S) with 100-percent compliance that results in savings above a prior
C&S or common practice, and another program claiming savings with a baseline defined
below the new C&S (e.g., a baseline defined by a prior C&S) for the same types of EE
activity.
o A state claiming credit for federal actions such as building code determinations or
appliance standards.
Applicable Practices
• Implement systematic tracking and accounting procedures, including the use of well-structured
and well-maintained tracking and reporting systems such as those already being used by many
states and EE implementers. Document procedures and systems.
• Implement the following procedures to avoid or correct for double counting:
o For EE activities with identified consumers, conduct tracking (type and number of EE
projects or EE measures implemented) at the utility-customer level using customer
name, address, account number (where available) and applicable dates for each activity.
o For EE activities without identified consumers, such as point-of-sale rebates and retailer
or manufacturer incentive programs, track applicable vendor, retailer, and manufacturer
data. Include the appropriate specifications and quantities of EE equipment sold or
shipped.
o Where practical, such as where multiple EE implementers share a common tracking
database, use the consumer-level data to identify and correct for duplicate EE activity
records across programs with "trackable" consumers and across non-program projects
such as private-sector transactions for projects sponsored by an ESCO.
36 This potential for double counting is particularly important in the context of randomized encouragement
programs, where part of the savings seen in treatment/control differences is due to increased participation in
general offering programs.
55
-------
The key sources of quantifiable random error that can result from applying EM&V methods include:
• Random sampling error, including error that results from the selection of samples of customers,
EE projects, or EE measures within an EE program; selection of individual EE measures to be
observed within a facility; and random assignment in the context of comparison group methods.
• Modeling or estimation error, when a regression model or other statistical estimation is used to
estimate savings or savings parameters.
2,7,2. Applicable Practices
• Document the approaches used to assess the accuracy of quantified electricity savings and to
control the types of error inherent to the applied EM&V methods. Specify how measurement
error is controlled, as well as how quantifiable random error is estimated. Identify potential
sources of systematic error, plan steps to minimize these sources as practical, and report on the
sources, the mitigation steps taken, and any qualitative assessment of their likely effects.
• Establish targets for the measured accuracy of quantifiable errors, in terms of the relative
precision of a statistical confidence interval. Such targets may be expressed in the form of
confidence level/relative precision. A common target is 90/10, meaning that a 90 percent
confidence interval for estimated savings should be no wider than + 10% of the estimate itself.
Another common target is 80/20, meaning an 80% confidence interval no wider than ± 20% of
the estimate. Confidence/precision targets may be based on established jurisdictional
requirements. Apply such targets for the total savings addressed by an EM&V effort, such as an
overall program or portfolio (i.e., not just for individual projects or measures). Design EM&V
studies to meet these confidence/precision targets, using reasonable assumptions based on
prior similar studies.
• Design assumptions needed for savings quantification to provide neither optimistic savings
estimates (aiming to err on the high side) nor conservative estimates (aiming to err on the low
side).
• If sampling is used to quantify savings values, report the achieved confidence/precision of the
associated estimates.
• Apply and cite applicable best-practice protocols and guidelines documents for sampling.
Examples of best practices for statistical sampling are described in the following resources:
o UMP Sample Design Cross-Cutting Protocol, Chapter 11 (NREL, 2017b)
o Load Research Manual (AEIC, 2017), Chapters 4 "Sample Design and Selection" and
Chapter 7 "Data Analysis"
o Sampling Techniques (Cochran, 1977)
o Survey Sampling (Kish, 1995)
o Introduction to Variance Estimation (Wolter, 1985)
o Encyclopedia of Survey Research Methods (Lavrakas, 2008)
• For states tracking or trading emissions reductions across borders, coordinate across
jurisdictions to apply the same or consistent EM&V approaches to the extent practical, to ensure
the savings values are quantified with comparable levels of accuracy.
57
-------
Glossary of Terms
This glossary includes only terms that are applied in this EM&V Manual.38
Accuracy: How close an estimate is to the true value it estimates. Accuracy can be used in reference to a
point estimate resulting from a sequence of analytic steps, model coefficients, a set of measured data,
or a measuring instrument's capability.
Additional savings (for energy or air quality planning): Energy savings additional to the savings explicitly
or implicitly assumed in an energy or air quality policy baseline, such as the SIP emissions baseline.
Baseline condition: The efficiency level and operating conditions that would have occurred without the
EE activity.
Baseline consumption: The electricity use that would have occurred at the baseline efficiency level and
operating conditions.
Baseline efficiency: The efficiency level that would have been in place without implementation of a
specific EE activity.
Code: Legal EE requirements that apply to the design and construction of buildings, usually for new
buildings and for renovations and additions to existing buildings.
Code/standards baseline: A baseline corresponding to an efficiency level based on applicable federal,
state, or local equipment standards or building codes.
Comparison group (EM&V method): Based on the differences in electricity use patterns between a
population of premises with EE projects or EE measures in place and a comparison group of premises
without the EE projects or EE measures; comparison group approaches include randomized control trials
(RCTs) and quasi-experimental methods using nonparticipants and may involve simple differences or
regression methods.
Compliance (Code): Meeting the code requirements and demonstrating that these requirements have
been satisfied. Compliance is the responsibility of the builder or contractor.
Costing periods: Time blocks over which avoided costs are similar. Costing periods are typically defined
by individual utilities, ISO, or RTO, and tend to be defined by combinations of time of day, day type (e.g.
average weekday, peak weekday, weekend/holiday), month, and season.
Deemed formulas: Pre-specified formulas for quantifying savings, using some deemed parameters and
some inputs that are specific to each project or measure.
Deemed parameter values: Pre-specified values of parameters that are applied to quantify savings using
a deemed formula.
Deemed savings EM&V methods: An EM&V method that applies estimates of average annual electricity
savings for a single unit of an installed EE measure; deemed savings values are developed from certain
38 Certain states, EE implementers, and other stakeholders may currently apply variations of these terms. For
additional information, readers can consult the glossary of the SEE Action EM&V Guide (SEE Action, 2012b).
61
-------
data sources and analytical methods that are widely considered: (1) acceptable for the measure, and (2)
applicable to the situation and conditions in which the measure is implemented. Deemed savings
methods can include deemed savings values, deemed formulas, deemed parameter values.
Deemed savings values: Pre-specified estimates of average annual electricity savings for an EE project or
EE measure.
Derivation from annual savings shapes: Use a library of existing savings shapes established by
simulation or other methods to estimate hourly savings from annual energy savings.
Direct measurement and verification (M&V) EM&V method: An EM&V method that uses onsite
observations, engineering calculations, statistical analyses, and/or computer simulation modeling using
measurements to determine savings from an individual EE project or EE measure.
Dual baseline: A baseline used for programs targeting early replacement; corresponds to existing
efficiency up to the remaining useful life (RUL) of the existing equipment, systems, or construction; and
to either code/standards or market baselines for new installations for the remainder of the effective
useful life (EUL) of the EE activity.
Effective useful life (EUL): The duration of time an EE activity is anticipated to remain in effect with the
potential to save electricity.
Electricity savings: The difference between electricity consumption with an EE activity in place and the
consumption that otherwise would have occurred during the same period.
End-use metered data analysis: Estimate hourly energy savings using metering of the affected end use,
together with engineering analysis. The ability to estimate savings for all hours of the year depends on
the specific time period metered and whether this metering period can be credibly extrapolated to
represent the full year.
Energy efficiency activity (EE activity): An EE measure, EE project, or EE program.
Energy efficiency measure (EE measure): A single technology, energy-use practice, or behavior that,
once installed or operational, results in a reduction in the electricity use required to provide the same or
greater level of service at an end-use facility, premise, or equipment connected to the delivery side of
the electricity grid. EE measures may be implemented as part of an EE program or an EE project.
Energy efficiency program (EE program): Organized activities sponsored and funded by a particular
entity to promote the adoption of one or more EE projects or EE measures that, once installed or
operational, result in a reduction in the electricity use required to provide the same or greater level of
service in multiple end uses, facilities, or premises.
Energy efficiency project (EE project): A combination of measures, technologies, and energy-use practices
or behaviors that, once installed or operational, result in a reduction in the electricity use required to
provide the same or greater level of service. EE projects may be implemented alone or as part of an EE
program.
Engineering algorithms: Apply a formula to estimate peak demand savings based on characteristics of
the installed equipment and its operating patterns.
62
-------
Evaluation, measurement, and verification (EM&V): The set of procedures, methods, and analytic
approaches used to quantify the impacts of EE activities, renewable energy, and other measures, to
ensure that these impact estimates are reliable.
Ex ante savings: Projected savings prior to implementation of an EE activity.
Ex post savings: Savings determined after implementation of an EE activity.
Existing conditions baseline: A baseline corresponding to the efficiency level of equipment, systems, or
construction in place prior to the EE activity.
Facility: All buildings, structures, or installations located in one or more contiguous or adjacent
properties under common control of the same individual or organization.
Factor transfer: Apply factors from savings shapes developed in other territories to estimate peak
demand savings from the quantified energy savings.
Gross savings: The difference in energy consumption with an EE project or EE measure in place versus
the baseline consumption without the project or measure in place.
Hourly building simulation models: Estimate hourly energy savings by estimating consumption with and
without the EE project or measure in place for each hour of a standard year, based on detailed
specifications of building characteristics and operating conditions.
Incremental savings (e.g., for SIP compliance): See Additional savings.
Independent variables: Variables (e.g., weather, occupancy, production levels) that affect electricity
consumption and savings, and vary independently of the EE activity under study.
Interactive effects: Indirect impacts of EE activities that increase or decrease the use of electricity or
fossil fuels in end-use systems outside of the targeted end use. Interactive effects include "multi-
measure effects" or equipment and facility improvement interactions, "other-system effects" or inter-
end-use interactions, and "EE program overlap."
Load shape: The distribution of annual energy used over the year. It may be represented by the fraction
of annual energy falling into each hour of the year.
Market baseline: A baseline corresponding to an efficiency level based on the common practice for new
equipment or installations in the market.
Measurement: (a) The act of metering or monitoring, or (b) a measured or monitored metric
(dimension).
Metering: The collection of energy-use data over time. These data may be collected at the end use, a
circuit, a piece of equipment, or a whole building (or facility).
Monitoring: The collection of data relevant to how a piece of equipment operates, including but not
limited to energy use or emissions data (e.g., energy and water use, temperature, humidity, volume of
emissions, hours of operation).
Net savings: The difference in energy consumption with an EE program in place versus the consumption
without the program in place.
63
-------
Operating conditions: The conditions in which the EE project, measure or facility is operated, including
but not limited to weather, occupancy, and hours of operation.
Peak demand savings: Energy savings that occur at the time of the electricity system's peak demand.
Peak savings are typically quantified as the average hourly savings (kWh/h or kW) over the time block in
which the system peak typically falls.
Policy baseline: A baseline corresponding to business-as-usual projected conditions (e.g., energy
consumption, emissions) used to assess the effect of a potential new policy or policy change.
Post-installation operating conditions: The average operating conditions in the period after the EE
activity is implemented, over the EUL of the activity.
Random error: Estimation errors occurring by chance that may cause an estimate (such as an electricity
savings value) to be overestimated or underestimated with no systematic tendency in either direction,
resulting from uncontrolled and unobservable factors affecting the underlying measurements.
Savings shape: The distribution of annual energy savings over the year. The savings shape may be
represented by the fraction of annual savings falling into each hour of the year.
Site inspections: Site visits to facilities at which an EE project or EE measure was implemented.
Inspections document the existence, characteristics, and operation of baseline or EE project equipment
and systems and the factors that affect energy use.
Standards efficiency (baseline efficiency level): The efficiency level for the applicable federal, state, or
local equipment standard or building code (if any) in place prior to the EE activity.
Stringent practice baseline (SPB): A baseline corresponding to the more stringent39 of any applicable
codes or standards, and the common market practice for the situation.
Surplus savings (e.g., for SIP compliance): See Additional savings.
Systematic error: Estimation errors that may cause an estimate (such as an electricity savings value) to
be consistently either overstated or understated. Systematic errors are also referred to as bias, and may
result from incorrect assumptions, a methodological issue, or a flawed reporting system.
Technical reference manual (TRM): Resource document that includes information used in program
planning and reporting of EE programs. It can include savings values for measures, engineering
algorithms to calculate savings, impact factors to be applied to calculated savings (e.g., net-to-gross ratio
values), source documentation, specified assumptions, and other relevant material to support the
calculation of measure and program savings—and the application of such values and algorithms in
appropriate applications.
Time- and locational savings: Energy savings that occur at different times of the day (e.g., morning or
evening), by season (e.g., summer or winter), or annually, or in a defined geographic area.
Time-differentiated savings: Annual savings split out by time period. The time periods may be broad
costing periods, or may be individual hours for a year with a specified calendar.
39 Most stringent means requiring the lowest energy use.
64
-------
Transmission and distribution (T&D) loss: The difference between the quantity of electricity that serves
a load (measured at the busbar of the generator) and the actual electricity use at the final distribution
location (measured at the onsite meter).
Verification (of EE project or EE measure installation): An assessment by an independent entity to
ensure that the EE activities have been installed correctly and can generate the predicted savings.
Verification may include assessing baseline conditions and confirming that the EE activities are operating
according to how they were designed to operate. Site inspections, phone and mail surveys, and desk
review of program documentation are typical verification activities.
65
-------
References
American Council for an Energy-Efficiency Economy (ACEEE). 2012. Status and Opportunities for
Improving the Consistency of Technical Reference Manuals. Jayaweera, T., Velonis A., Haeri, H.,
Goldman, C., Schiller, S. Available at:
http://aceee.org/files/proceedings/2012/data/papers/0193-0Q0150.pdf.
ACEEE. 2015. How Information and Communications Technologies Will Change the Evaluation,
Measurement, and Verification of Energy Efficiency Programs. Rogers, E., Carley, E., Deo, S., and
Grossberg, F.
ACEEE. 2016a. Behind the Curtain: Characterization of Measure Technologies within Technical Reference
Manuals. Tamble, Z., Brown, M., Parnell, B., Lynch, S., Buckley, R., Maxwell, A. Available at:
http://aceee.Org/files/proceedings/2016/data/papers/2 1182.pdf.
ACEEE. 2016b. Technical Reference Manuals Best Practices from Across the Nation to Inform the Creation
of the California Electronic Technical Reference Manual (eTRM). Beitel, A., Melloch, T., Harley, B.,
Mejia, A. Available at: http://aceee.Org/files/proceedings/2016/data/papers/6 1027.pdf.
ACEEE. 2018a. Energy Efficiency Resource Standards. Available at: https://aceee.org/topics/energy-
efficiency-resource-standard-eers.
ACEEE. 2018b. Keeping the Lights On: Energy Efficiency and Electric System Reliability, Report U1809.
Relf, G., York, D., Kushler, M. Available at: https://aceee.org/research-report/yl809.
Agnew, K, Goldberg, M, Wilhelm, B. 2009. A Pacific Northwest Efficient Furnace Program Impact
Evaluation. Proceedings of the 2009 International Energy Program Evaluation Conference.
American Society of Heating, Refrigerating and Air-Conditioning Engineers (ASHRAE). 2014. ASHRAE
Guideline 14-2014. Measurement of Energy, Demand, and Water Savings.
Association of Edison Illuminating Companies (AEIC). 2017 Load Research Manual. Available at:
https://www.techstreet.eom/aeic#lrm.
California Investor Owned Utilities (CIOU). 2016. A White Paper: Residential Portfolio Impacts from
Whole-Premise Metering. Prepared by Goldberg, M. and Agnew, K. Available at:
http://www.calmac.org/pyblicatioris/Res Portfolio Impacts White Paper (Final) DNVGL 1-
22-2016 .pdf.
California Public Utilities Commission (CPUC). 2004. California Evaluation Framework. Prepared by The
TecMarket Works Team. Available at:
http://www.calmac.org/publications/California%5FEvaluation%5FFramework%5FJune%5F2004
%2Epdf.
CPUC. 2006. California Energy Efficiency Evaluation Protocols: Technical, Methodological, and Reporting
Reguirements for Evaluation Professionals. Prepared by The TecMarket Works Team. Available
at: http://www.calmac.org/publications/EvaluatorsProtocols Final AdoptedviaRuling 06-19-
2006.pdf.
66
-------
CPUC. 2018. Guidance for Program Level M&V Plans: Normalized Metered Energy Usage Savings
Estimation in Commercial Buildings. Prepared for Energy Division by LBNL: Granderson, J.,
Gruendling, P., Jacobs, P., Torok, C. Available at: ftp://ftp.cpuc.ea.goy/fiopher-
data/energy division/EnergvEfficiencv/RollingPortfolioPgrnGuidance/LBNL NMEC TechGuidanc
e Draft.pdf.
Cochran, W. G. 1977. Sampling Technigues, Third Edition. New York: Wiley.
DNV GL. 2015. The Changing EM&V Paradigm -A Review of Key Trends and New Industry Developments,
and Their Implications on Current and Future EM&V Practices. Prepared for the Northeast
Energy Efficiency Partnership Regional Evaluation, Measurement & Verification Forum.
Efficiency Valuation Organization (EVO). 2016. International Performance Measurement and Verification
Protocol (IPMVP). Available at: https://evo-world.org/en/products-services-niainrnenu-
en/protocols/iprnvp.
Gilleo, A. 2014. Picking All the Fruit: All Cost-Effective Energy Efficiency Mandates. 2014 ACEEE Summer
Study on Energy Efficiency in Buildings. Available at:
https://aceee.org/files/proceedings/2014/data/papers/8-377.pdf.
Goldberg, M, Michelman, T, Dickerson, C. 1997. Can We Rely on Self Control? Proceedings of the 1997
International Energy Program Evaluation Conference. Chicago, IL.
ISO New England (ISO-NE). 2014. Measurement and Verification of Demand Reduction Value from
Demand Resources. Available at: https://www.iso-ne.com/static-
assets/dociimeiits/2017/02/mmvdr measurement-and-verification-demand-
reduction rev6 20140601.pdf.
Kish, Leslie. 1995. Survey Sampling. New York: John Wiley & Sons, Inc.
Lavrakas, Paul J. 2008. Encyclopedia of Survey Research Methods. Sage. Available at:
http://rriethods.sagepub.com/Reference/encvclopedia-of-survev-research-methods.
Lawrence Berkeley National Laboratory (LBNL). 2015. Assessment of Automated Measurement and
Verification (M&V) Methods. Granderson, J., Touzani, S., Custodio, C., Fernandes, S., Sohn, M.,
Jump, D. LBNL#-187225.
LBNL. 2017a. Application of Automated Measurement and Verification to Utility Energy Efficiency
Program Data. Granderson, J., Touzani, S., Fernandes, S., Taylor, C. Available at: http://eta-
publications.lbl.gov/sites/default/files/1007286.pdf.
LBNL. 2017b. The State of Advanced Measurement and Verification Technology and Industry
Application. Granderson, J„ Fernandes, S. Available at: http://eta-
publications.lbl.gov/sites/default/files/sam fernandes - report -
state of advanced measurement and verification technology and industry application O.p
df.
LBNL. 2017c. Time-Varying Value of Electric Energy Efficiency. Mims, N., Eckman, T., Goldman, C.
Available at: https://emp.lbl.gov/publications/time-varying-value-electric-energy.
67
-------
LBNL. 2018. Time-Varying Value of Energy Efficiency in Michigan. Mims, N., Eckman, T., Schwarz, L.
Available at: https://emp.lbl.gov/publications/tirne-varving-value-energv-efficiency.
Lawrence Berkeley National Laboratory and U.S. Department of Energy. June 27, 2016. Using Deemed
Savings and Technical Reference Manuals for Efficiency Programs and Projects [Webinar],
Available at: https://emp.lbl.gov/sites/all/files/EMVWebinar June2Q16.pdf and
https://www.youtube.com/watch?v=PLnBkglQh68&feature=voutube.
Massachusetts Program Administrators and Energy Efficiency Advisory Council. 2017. Massachusetts
Commercial/Industrial Baseline Framework, http://rna-eeac.org/wordpress/wp-
content/uploads/M A-Cornrnercial-and-lndustrial-Baseline-Framework-l.pdf.
Northeast Energy Efficiency Partnerships (NEEP). 2010. Regional EM&V Methods and Savings
Assumptions Guidelines. Available at: https://neep.org/regional-emv-rnethods-and-savings-
assumptions-guidelines-2010.
Northwest Power & Conservation Council, Regional Technical Forum. About the RTF. Available at:
http://rtf.riwcoyncil.org/aboyt.htm.
National Association of Regulatory Utility Commissioners (NARUC). 2016. NARUC Manual on Distributed
Energy Resources Rate Design and Compensation. Prepared by NARUC Staff Subcommittee on Rate
Design. Washington, D.C. Available at: https://pubs.naruc.org/pub/19FDF48B-AA57-5160-DBAl-
BE2E9CZF7EA0.
National Renewable Energy Laboratory (NREL). 2013. Chapter 8: Whole Building Retrofit with
Consumption Data Analysis Evaluation Protocol. Available at:
https://www.nrel.gov/docs/fyl7osti/63564.pdf.
NREL. 2014a. Chapter 21: Estimating Net Savings - Common Practices. Violette, Daniel M., Pamela
Rathbun. Available at: https://www.nrel.gov/docs/fvl7osti/68578.pdf.
NREL. 2014b. Chapter 23: Estimating Net Savings: Common Practices. Violette, D., Rathbun, P. Available
at: https://www.energv.gov/sites/prod/files/2Q15/Q2/fl9/UMPChapter23-estiniating-net-
savings Q.pdf.
NREL. 2017a. Chapter 10: Peak Demand and Time-Differentiated Energy Savings Cross-Cutting Protocols.
Stern, F., Spencer, J. Available at: https://www.nrel.gov/docs/fyl7osti/68566.pdf.
NREL. 2017b. Chapter 11: Sample Design Cross-Cutting Protocol; The Uniform Methods Project: Methods
for Determining Energy Efficiency Savings for Specific Measures. Prepared by The Cadmus
Group. Available at: https://www.nrel.gov/docs/fvl7osti/68566.pdf.
NREL. 2017c. Chapter 13: Assessing Persistence and Other Evaluation Issues Cross-Cutting Protocol.
Violette, D. Available at: https://www.nrel.gov/docs/fvl7osti/68569.pdf.
NREL. 2018. The Uniform Methods Project: Methods for Determining Energy Efficiency Savings for
Specific Measures. Li, M., Haeri, H., Reynolds, A. Available at:
https://www.nrel.gov/docs/fyl8osti/7Q472.pdf. Individual protocols available at:
https://www.energv.gov/eere/about-us/ump-protocols.
68
-------
PJM. 2016. PJM Manual 18B: Energy Efficiency Measurement & Verification. Available at:
https://www.pim.eom/~/roedia/docurnents/manuals/ml8b.ashx.
Public Utility Commission of Texas. 2013. Approach to Texas Technical Reference Manual - Revised for
Version 3.0 (Final). Prepared by TetraTech.
Rocky Mountain Institute (RMI). 2017. The Status and Promise of Advanced M&V: An Overview of "M&V
2.0" Methods, Tools, and Applications. Franconi, E., Gee, M., Goldberg, M., Granderson, J.,
Guiterman, T., Li, M., Smith, B. Available at: https://www.rmi.org/wp-
content/uploads/2017/03/Advanced M and V Report MarchZOl? RMI.pdf.
State and Local Energy Efficiency Action Network (SEE Action). 2012a. Energy Efficiency Program Impact
Evaluation Guide. Prepared by Schiller, S., Schiller Consulting, Inc. Available at:
https://www4.eere.energv.gov/seeaction/publication/energv-efficiencv-prograrn-impact-
evaluation-guide.
SEE Action. 2012b. Evaluation, Measurement, and Verification (EM&V) of Residential Behavior-Based
Energy Efficiency Programs: Issues and Recommendations. Available at:
https://www4.eere.energy.gov/seeaction/publication/evaluation-measurement-and-
verification-ernv-residential-behavior-based-energy-efficiencv.
SEE Action. 2017. SEE Action Guide for States: Guidance on Establishing and Maintaining Technical
Reference Manuals for Energy Efficiency Measures. Prepared by Lawrence Berkeley National
Laboratory. Available at:
https://www4.eere.energv.gov/seeaction/svstern/files/docurnents/TRM%2QGuide Final 6.21.1
7.pdf.
SEE Action. 2018. SEE Action Guide for States: Evaluation, Measurement, and Verification Frameworks-
Guidance for Energy Efficiency Portfolios Funded by Utility Customers. Prepared by Lawrence
Berkeley National Laboratory. Available at:
https://www4.eere.eriergv.gov/seeaction/systero/files/docuroeiits/EMV-
Framework Jan2018.pdf.
U.S. Department of Energy (U.S. DOE). 2008. M&V Guidelines: Measurement and Verification for Federal
Energy Projects Version 3.0. Available at:
http://portal.hud.gov/hudportal/docurnents/huddoc?id=doc 10604.pdf.
U.S. DOE. 2011. Scoping Study to Evaluate Feasibility of National Databases for EM&V Documents and
Measure Savings. Prepared by Jayaweera, et al.; The Cadmus Group Inc. Available at:
http://energy.gov/sites/prod/files/2013/ll/f5/enivscoping databasefeasibility appendices.pdf.
U.S. DOE. 2017. Quadrennial Energy Review, Second Installment. Available at:
https://www.energv.gov/sites/prod/files/2017/P2/f34/Quadrennial%20Energv%20Review--
Second%20lnstallment%20%28Full%20Report%29.pdf.
U.S. DOE. 2018. Federal Energy Management Program. Available at:
http://energy.gov/eere/femp/federal-energy-rnanagement-program.
69
-------
U.S. Energy Information Administration (U.S. EIA). 2019. Frequently Asked Questions: How much
electricity is lost in transmission and distribution in the United States? Accessed March 21, 2019.
Available at: http://www.eia.gov/tools/faqs/faq.cfni?id=lQ5&t=3.
U.S. EPA. 2012. Roadmap for Incorporating Energy Efficiency/Renewable Energy Policies and Programs
into State and Tribal Implementation Plans. Available at:
https://www.epa.gov/sites/prodyction/files/2016-05/docyments/eeremanyal O.pdf.
U.S. EPA. 2017. EM&Vfor Energy Efficiency Policies and Initiatives. Available at:
https://www.epa.gov/sites/prodyction/files/Z017-06/docyments/emvframeworkpaper 2017-
01-19.pdf.
U.S. Environmental Protection Agency (EPA). 2018a. AVERT Web Edition. Available at:
https://www.epa.gov/statelocalenergy/avert-web-edition.
U.S. EPA. 2018b. Energy Star. Available at: https://www.energystar.gov/.
Wolter, K. M. 1985. Introduction to Variance Estimation. New York: Springer-Verlag.
70
-------
430B19001
June 2019
vvEPA
------- |