UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
                               WASHINGTON D.C. 20460
                                                          OFFICE OF THE ADMINISTRATOR
                                                           SCIENCE ADVISORY BOARD

                               November 25, 2008

EPA-SAB-09-005

The Honorable Stephen L. Johnson
Administrator
U.S. Environmental Protection Agency
1200 Pennsylvania Avenue, N.W.
Washington, D.C. 20460

       Subject: Consultation on Pollution Prevention Measures

Dear Administrator Johnson:

       EPA's Office of Pollution Prevention and Toxics (OPPT) currently uses a suite of
measures to evaluate the effectiveness of EPA's pollution prevention (P2) programs.
OPPT requested the Science Advisory Board (SAB) provide consultative advice on
sound methodologies for measuring P2 program performance results, particularly
recurring results. In response to OPPT's request, the SAB Environmental Engineering
Committee (EEC), augmented with additional SAB members, conducted a consultation
on this subject at a public meeting on September 3-4, 2008. (See Enclosure 1 for the
Committee's roster.)

       The Committee commends OPPT on the creative development and substantial
progress of the seven P2 programs. These programs have resulted in avoidance of
emissions, energy consumption, and  resource depletion and have provided other benefits
documented by the metrics developed by OPPT.  The P2 programs are among the most
forward-looking and important programs of the EPA, defining a new model for
environmental protection and restoration based on voluntary initiative that is prompted by
incentives for cost savings and higher product and process performance.  These are also
among the most difficult programs to implement, as they are based on voluntary
participation by partner organizations, the appropriate data to assess the individual
programs are not readily available, the information can be sensitive as it may provide a
competitive advantage to competing  organizations,  and effective approaches are often
specific to the particular product or process. Measuring progress achieved at the program
level is challenging because, for a variety of good reasons, results are measured in
different ways, and on different time scales. As a consequence, results  are often difficult
to compare and aggregate. Nevertheless, the P2 programs of OPPT have made

-------
significant progress, and the metrics developed to date at least partially document this
success.

       An SAB consultation is a mechanism for individual technical experts to provide
comments for the Agency's consideration early in the development of a technical
product. A consultation does not require achievement of consensus among the
Committee members or preparation of a detailed report.  In this consultation, however,
the Committee would like to highlight key points that were developed in the discussions
in response to the five charge questions (Enclosure 2) which are summarized in the
document enclosed with this letter (Enclosure 3).

       The Committee would like to thank EPA presenters for their expertise,
perspectives, and insights that assisted the Committee's understanding of the P2
programs. Thank you for the opportunity to provide early  advice on this important topic.
As this is a consultation, the SAB does not expect a formal response from the Agency.
                                  Sincerely,

                                        /Signed/

                                  David A. Dzombak, Chair
                                  Environmental Engineering Committee
                                  Science Advisory Board
cc:  Deborah L. Swackhamer,
    Chair, Science Advisory Board

-------
                                                                   Enclosure 1
                  U.S. Environmental Protection Agency
                          Science Advisory Board
    Environmental Engineering Committee Augmented for Pollution
                     Prevention Measure Consultation
CHAIR
Dr. David A. Dzombak, Walter J. Blenko Sr. Professor of Environmental Engineering,
Department of Civil and Environmental Engineering, College of Engineering, Carnegie
Mellon University, Pittsburgh, PA
MEMBERS
Dr. Viney Aneja, Professor, Department of Marine, Earth, and Atmospheric Sciences,
School of Physical and Mathematical Sciences, North Carolina State University, Raleigh,
NC, USA

Dr. James Boyd, Senior Fellow, Director, Energy & Natural Resources Division,
Resources for the Future, Washington, DC

Dr. John C. Crittenden, Richard Snell Presidential Professor , Department of Civil and
Environmental Engineering, Ira A. Fulton School of Engineering, Arizona State
University, Tempe, AZ

Dr. T. Taylor Eighmy, Assistant Vice President for Research and Director of Strategic
Initiatives, Office of the Vice President for Research, University of New Hampshire,
Durham, NH

Dr. Madhu Khanna, Professor, Department of Agricultural and Consumer Economics,
University of Illinois at Urbana-Champaign, Urbana, IL

Dr. Cindy M. Lee, Professor, Department of Environmental Engineering and Earth
Sciences, Clemson University, Anderson, SC

Dr. Reid Lifset, Director of the Industrial Environmental Management Program and a
member of the faculty, Industrial Environmental Management Program, School of
Forestry and Environmental Studies, Yale University, New Haven, CT

Dr. Jill Lipoti, Director, Division of Environmental Safety and Health, New Jersey
Department of Environmental Protection, Trenton, NJ

Dr. Michael J. McFarland, Associate Professor, Department of Civil and
Environmental Engineering, Utah State University, Logan, UT

-------
Dr. James R. Mihelcic, Professor, Civil and Environmental Engineering, State of Florida
21st Century World Class Scholar, University of South Florida, Tampa, FL

Dr. Horace Moo-Young, Dean and Professor, College of Engineering, Computer
Science, and Technology, California State University, Los Angeles, CA

Dr. Catherine Peters, Associate Professor, Department of Civil and Environmental
Engineering, Princeton University, Princeton, NJ

Dr. Susan E. Powers, Associate Dean and Professor, Department of Civil and
Environmental Engineering, Wallace H. Coulter School of Engineering, Clarkson
University, Potsdam, NY

Dr. Mark Rood, Professor, Department of Civil and Environmental Engineering,
University of Illinois, Urbana, IL

Dr. Bryan Shaw, Commissioner, Texas Commission on Environmental Quality, Austin,
TX

Dr. John R. Smith, Division Manager, Environmental Science and Sustainable
Technology, Alcoa Inc., Alcoa Technical Center, Alcoa Center, PA
SCIENCE ADVISORY BOARD STAFF
Ms. Kathleen White, Designated Federal Officer, 1200 Pennsylvania Avenue, NW,
Washington, DC, Phone: 202-343-9878,  Fax: 202-233-0643, (white.kathleen@epa.gov)

-------
                                           Enclosure 2
       Charge for the Consultation on



   Pollution Prevention Program Measures



                 with the



   Environmental Engineering Committee



                  of the



       EPA's Science Advisory Board
EPA Office of Pollution Prevention and Toxics



            September 3, 2008

-------
Pollution Prevention Program Measures

The  Pollution Prevention  (P2)  Program  seeks  feedback from the SAB  on considerations
associated with technical elements for the development of a sound methodology for measuring
recurring performance  results  for P2  Program Centers.  It seems like there are a variety  of
measures that could be used, some of which are depicted in the illustration, below.
            Generic Depiction of Performance Results
     0)
    13
    s
     u
     g
     c
     3
                                                         -Baseline without P2 Program
                                                          Intervention
                                                         -With Pollution Prevention Program
               Annual Recurring Results

            |  Cumulative Recurring Results
                 New Annual Result
               from P2 Program FY2004
        1999    2000   2001    2002    2003
                           Fiscal Year
2004
2005
This illustration addresses a single hypothetical P2 Program Center which may undertake one or
more activities in any given year. It presents a baseline curve of toxic materials generated over a
period of years without any P2 Program intervention; then a separate curve depicting generation
over the same span of years as a result of P2 Program interventions.  One measure of results
would be to capture new annual results, the measure of new interventions in a year, which is
calculated by taking the  difference between the pounds of toxic materials generated in one year
from the previous year.  Many P2 Center interventions can also generate results over a span of
years. One could consider an annual recurring result which is the vertical distance between the
curves during any  single year.  Alternatively, cumulative  recurring results  are depicted by the
shaded area between the curves.  It is the recurring results measures that are the focus of this
consultation with the Science Advisory Board.

In developing sound methodology for measuring different performance results, the P2 Program
is particularly interested in understanding technical factors that are important to consider when
measuring recurring results.  This understanding should help us,  for example, to decide on
appropriate modifications to the methodology when facing  constraints on the collection of results
data, particularly on a year-after-year basis. The P2 Program has developed  some understanding
for counting recurring results in certain contexts, and is seeking  additional input in order to
develop sound methodology for counting recurring results across P2  Program  Centers.  The
methodology will need to meet the following technical parameters:

-------
    1)  completeness - a methodology for recurring results must work for all Centers because all
       Centers must report performance results into a unified Agency reporting system; and
    2)  comparability - a methodology for recurring results must support the ability to generate
       time-series data to analyze changes in performance over time, must support performance
       comparisons between Centers, must be sufficiently flexible to accommodate the design
       and implementation differences among Centers, and must  be sufficiently  neutral  and
       transparent  so that  report users  can understand the factors  that may contribute to
       differences in performance among Centers.

Key Terms

Please  refer to the Annex 1 (Glossary) for a list of key terms that will be used throughout this
document.
Background

The P2 Program has a number of reasons for wanting to improve its measures of performance.
The P2  Program is guided by statute to measure  and  report its performance.   The Pollution
Prevention Act (PPA) of 1990 directs the Agency to  develop an approach to measuring P2 (42
USC 13101).  The Government Performance and Results Act (GPRA) directs federal agencies to
develop strategic plans  outlining what programs intend to accomplish, how they measure their
progress, and how they  communicate that information about their performance to Congress and
to the public, among other requirements.  In response to GPRA, the Office of Management and
Budget  (OMB)  developed the  standardized Program Assessment Rating  Tool  (PART) to
evaluate  the  effectiveness of program  performance  in budget submissions  for individual
programs and program groupings.  Beyond  these statutory reasons for measurement, in this age
of results orientation it is important to be able to measure performance to demonstrate the value
of P2 Programs to government and private sector stakeholders to foster support and participation.
Supporting  Materials

Additional information on the P2 Program, its Centers, and other topics relevant to performance
measures and recurring results is provided in a supporting materials document.  The additional
information includes:
   -   a narrative overview of the seven P2 Centers and their streams of results, and
   -   a table on data management within the P2 Centers.

In addition, weblinks are provided for the following:
   -   the Pollution Prevention Act of 1990,1
   -   Executive Order 13423, Strengthening Federal Environmental, Energy, and
       Transportation Management,2
1 http://www.epa.gov/oppt/p2home/pubs/p2policy/actl990.htm
2http://www.ofee.gov/eo/EO_13423.pdf
                                      Page 2

-------
   -  the Pollution Prevention Program PART Program Assessment (Fall 2006).3

Please keep in mind that some of these materials (e.g., PART assessment questions and answers)
are provided as background only and are not directly relevant to this consultation.

Charge to the SAB

The  P2 Program is seeking the consultative  input of the SAB Engineering Committee on the
technical elements  necessary to consider in the development of a sound methodology for
measuring P2 program performance results, particularly recurring results.  The Program is at a
fairly early  stage of methodological development,  and is seeking the benefit of Committee
members' thinking and expertise  at this  stage  of  the  process.  The Program is specifically
interested in receiving  input from the SAB on the questions presented below on the technical
elements identified thus far.
1. Over what time period is it appropriate to count recurring results?

   It is generally accepted that entities adopting a pollution prevention practice will frequently
   keep such a practice in place for more than a year, generating pollution reduction results over
   a span of years.  The life span over which it is appropriate to count these results likely varies
   by industry.  What technical elements should be considered when determining how long to
   count  recurring results? What data sources exist to estimate the typical lifecycle of a
   commercial technology and how might this typical lifecycle vary by industry?

   Are there other factors that are relevant to consider when identifying the end of a benefits
   stream of results from a P2 practice?
2. What are appropriate data sources?

   What data sources are appropriate for substantiating recurring results from P2 Centers whose
   technical assistance activities are referral driven, and facility driven, but not sector driven?
   Discussion:  Could innovation rates, and/or the birth and death rate of firms, be used towards
   building a basis  for counting recurring results  from Centers that work with  small- and
   medium-sized businesses broadly  across sectors on a technical assistance basis?  What data
   sources exist to estimate the typical length of time an innovation is retained in small U.S.
   businesses, and for medium-sized U.S. businesses?   What other factors can  be considered
   when determining how to substantiate recurring results for technical assistance programs that
   work broadly across sectors?

   What should we do with data source gaps over time for producing time-series data?
   Discussion:    See Table 3  in the supporting materials  comparing our data sources and
   literature search data sources and identifying gaps to explore.
1 http ://www. whitehouse. gov/omb/expectmo re/summary/I 00043 04.2006. html


                                       Page 3

-------
3. How should aggregation issues be addressed?

   Assuming practices have different life cycles, thus different time periods over which results
   would presumably be counted, what are the technical issues faced when aggregating results
   for all practices at a center or across all centers?
4. How do other programs and organizations consider or measure
   recurring results, and can we benchmark and learn from their
   experience?

   The experiences of other programs and  organizations regarding  recurring  results may be
   informative to  the P2  Program's decision  on a path forward on its  own methodology
   development.   The  P2  Program would  like  to  understand how other programs or
   organizations considered or measured recurring results  and, if so, whether there are existing
   benchmarks  that the P2 Program can utilize.   The Program has conducted  an  initial
   benchmarking exercise  and  is interested in expanding the  scope and depth of information
   searched and  compiled.  The Program is particularly interested in additional information on
   the following:

   Are there other programs and organizations that count recurring results? What is the scope
   and purpose of these programs and are they similar to the voluntary P2 Program?

   -   What data do these programs collect and what is (are) the source(s) of the  data?

   -   What are the typical performance results that are measured and what is the methodology
       used to compute the performance results?

   -   Do any of their  experiences relate to specific P2 Program Centers?

   -   Are there  other  indicators as to how recurring benefits are viewed (regardless whether
       they are reflected in measurement approaches)?

   See Section III in  the  supporting materials  for the P2 Program's initial findings  from a
   benchmark literature search summarized in Table 2 organized by program/organization with
   a description  of each approach to recurring results.  Section III provides a discussion of the
   P2 Program's initial benchmark literature search findings, including the relevance of specific
   findings to the individual seven P2 Program  Centers as well  as  across the P2 Program.
   Information  gathered includes government  programs (state and  federal),  some industry
   practices (at the individual company level),  as well as global environmental measurement
   principles.  The benchmark information in Section III is provided solely as a source of ideas
                                      Page 4

-------
   building on the experience of others who have addressed similar performance measurement
   questions OPPT is facing. It is not intended that they will be part of this consultation in any
   other manner.

5. How can we best use the initial benchmark literature search findings to
   help us further inform  our investigation on recurring results and begin
   methodology development?

   The P2 Program is seeking to relate findings of the initial benchmark literature search to the
   data sources that differentiate our Centers and are relevant to resolving how to approach
   recurring results across the P2 Program.  Additionally, the Program is interested in how to
   use the initial findings to further focus and prioritize our investigation on recurring results for
   the purpose of methodology development.  The P2 Program has identified the following
   preliminary list of task-related considerations on which we are seeking consultative advice.
   See also Section IV of the supporting materials for more information.

       - In  analyzing the current data  sources for each Center,  how does the availability or
         limitations in data sources for an individual Center (such as publicly available market
         data or statutory barriers to data collection) affect P2 Program options for measuring
         recurring results program-wide?  Table  3 in Section IV of the supporting materials
         compares P2 Program data sources with data sources inferred from examples in the
         literature.

       - Where data gaps exist for certain Centers, are there other data sources that the Centers
         could consider  (such  as  economic,  market  trends,  market  demographics) in
         methodology development to allow for measuring recurring results consistently across
         the P2 Program?

       - In  continuing  a  benchmarking exercise, are  there other benchmarks we  might
         consider?
                                      Page 5

-------
                               ANNEX 1 - Glossary
P2 Centers. The P2 Program has seven Centers or distinct program units that produce
   performance results. Each P2 Center has its own program design, targets, and goals. All P2
   Centers use common outcome performance measures - pounds of hazardous materials
   reduced, BTUs of energy/million metric tons of carbon equivalent reduced (program is
   switching from BTUs to MMTCE), gallons of water reduced, dollars saved.  Not every
   center reports to all four measures, but all of them report to at least three of these measures.
   Details on the Centers appear in attached background documents, identified immediately
   below.

Program intervention (or P2 intervention).  This is a Center activity that produces a specific
   environmental outcome (pounds reduced, energy saved, etc.) at a point in time.  There is also
   a view that it includes the ongoing administrative investments in building and/or maintaining
   the overall  project which produces the series of specific environmental outcomes.

Results. For purposes of this consultation, "results" refers to program performance results
   expressed in terms of specific indicator measures.  These specific P2 Program indicator
   measures are pounds of hazardous materials (releases to air, water, land, and material inputs)
   reduced, billion BTUs of energy reduced (this measure is being changed to million metric
   tons of carbon equivalent reduced), gallons of water reduced, and dollars saved from P2
   interventions.  Provided on the following pages is a graph that further illustrates how a P2
   Center may represent annual and recurring results.   Equations and examples are also
   provided to illustrate how the various types of results may be interpreted from the example
   graph.

Baseline Year. The baseline year is the year from which results in following years are
   measured.  In the example below, there are no Annual Results in the baseline year (FY2000).

(New) Annual Results. New Annual Results or simply Annual Results refer to one year's worth
   of performance results from a program intervention that occurred in a given year.

   Example:  To calculate the New Annual Result in FY2004 in the graph below, subtract the
   amount of toxic material generated in FY2004 (with the P2 Program in place) from the
   amount of toxic material generated in FY2003.
       e.g. New Annual ResultpY2oo4 = 16,500 - 15,000 = 1,500 Ibs of toxic materials reduced

Annual Recurring Results (Cumulative Annual Results). Annual Recurring Results are
   benefits that occur in a single year from P2 interventions initiated in previous years.  The P2
   Programs include any New Annual Results in this total as well. Note: Annual Recurring
   Results  are equivalent to Cumulative Annual Results which are the sum of New Annual
   Results  year-after-year, starting from a baseline year of measurement.

   Assuming a baseline year of FY2000, New Annual Results begin to occur in FY2001 and
                                       Page 6

-------
   continue every year through FY2004.  Then the Annual Recurring Results for the P2 Center
   inFY2004wouldbe:
                                                 2004
          Annual Recurring Results FY2004 = ^ Annual Re suits FYk
                                                k=2001

   Example:  To calculate the Annual Recurring Result in FY2004 in the graph below, add the
      Annual Results in each year. This calculation can be simplified to subtract the total
      amount of toxic waste generated with the P2 Program in place from the baseline level of
      waste.
      e.g. Annual Recurring ResultFY2oo4 = 20,000 Ibs - 15,000 Ibs = 5,000 Ibs of toxic
      materials reduced.

Cumulative Recurring Results.  Cumulative recurring results are the sum of recurring results
   over two or more years, starting from the baseline year of measuring.

   Starting with a baseline year of FY2000, the cumulative recurring results for the P2 Center in
   FY2004 would be:
                                             2004
Cumulative Recurring ResultsFY2004  =  ^ Annual Recurring ResultsFYk
                                            k=2001

   (Note: In FY2001, annual recurring results are equivalent to new annual results.)

   Example:  To calculate the Cumulative Recurring Results in FY2004 in the  graph below, add
   the Annual Recurring Results in each year.
      e.g. Cumulative Recurring ResultsFY2oo4 = (20,000-19,000) + (20,000-17,800) + (20,000-
      16,500) + (20,000-15,000) = 11,700 Ibs of toxic materials reduced.
                                     Page 7

-------
u
'x
3  0)
                Depiction of Performance Results
                                                    • Baseline without P2 Program
                                                    Intervention

                                                    -With Pollution Prevention Program
                                                 | Annual Recurring Result FY2004 \

                                                 Cumulative Recurring Results FY2004
                                                             New Annual Result
                                                           from P2 Program FY2004
        7999    2000   200?    2002    2003    2004    2005

                          Fiscal Year
                                        2004
Annual  Recurring Results
                               FY2004
                                           Annual Results
                                                           FYk
                                       k = 2001
     Cumulative  Recurring Results
                                        2004
                               FY2004  =  XAnnual Recurring  Results FYk
                                       k = 2001
                                    PageS

-------
                                                                    Enclosure 3


   EPA SAB Consultation on Pollution Prevention Program Measure
                                Methodology
                            September 3-4, 2008

                          Summary of Key Points

Charge Question 1: Over what time period is it appropriate to count recurring results?

   1.  It is appropriate to count results over the life of a technology (product or process),
      when the technology use is stable and not largely displaced by a new technology.
      An argument can be made for only counting results during the time period of
      technology distribution in the marketplace, but the majority  opinion on the
      committee favored counting results over the life of the technology use.

   2.  Appropriate time periods depend on the particular technology. For example, a
      time period relevant for some devices could be the mean time to failure and its
      replacement with a new technology.

   3.  By focusing on the technology rather than the company, OPPT can avoid
      counting problems associated with the birth and death of companies.

   4.  The baselines or benchmarks used for counting are extremely important, they
      should be carefully selected and the reasoning for their selection should be both
      clear and defensible.  Because EPA P2 programs are not the only source of
      innovation, defining a baseline needs to account for the conventional innovation
      rate and ranges of outcomes.

   5.  OPPT currently uses amounts of materials that are not emitted to the environment
      and reduction in energy consumption as measures. Resources not expended and
      cumulative impacts avoided could also be used as measures  of results that
      consider preservation of natural capital more broadly.


   Charge Question 2:  What are appropriate data sources?

   6.  OPPT has chosen appropriate data sources  and evaluated them. OPPT also has
      opportunities to use other EPA data sources not directly related to P2 programs,
      e.g., air and water permitting data, and TRI data.

   7.  OPPT should continue to strengthen processes for data validation.

   8.  OPPT should continue to develop uniform  methods of data collection across the
      P2 centers.

-------
9.  The EEC recognizes that, because of the government's interest in not over-
   burdening individuals and organizations with requests for information and the
   need to limit barriers to voluntary participation in P2 programs, collection of
   additional data is neither a simple question nor is it a choice fully within the
   domain of OPPT or EPA. However, if it was possible, it would be useful to
   collect data that document how EPA is accelerating innovation and economic
   competitiveness. The acceleration of innovation and related economic
   competitiveness  are also measures of the P2 programs' results.

10. Ideally and over the long-term, more meaningful measures of the results of
   programs, including pollution prevention programs, should incorporate risk
   averted in addition to chemical releases averted.  Unfortunately, the data required
   to implement this measure for all EPA P2  programs do not exist and will not exist
   any time soon.  Similarly, measures of energy conservation could be improved by
   weighting energy sources by human health and environmental impacts because
   not all amounts of energy have equal impacts on resources and the environment.

11. To support better analyses when complete suites of data are lacking,  OPPT should
   examine the scientific literature on missing data and develop methodologies based
   on this literature, e.g., the use of proxy data for particular facilities or processes.
Charge Question 3:  How should aggregation issues be addressed?

12. While aggregation makes it difficult to convey the influence of P2 programs on
   systems, it is acceptable if similar data that describe, for example, source
   reductions or changes in consumption, are being aggregated. As noted in (1),
   health risks averted could be a valuable common measure, but it is not yet
   possible to determine this for the range and variety of technologies in the P2
   programs. This is primarily due to the lack of available data, the constraints on
   collecting the relevant data, and the lack of resources with which to do so.

13. Work to segregate product and process impacts because a process that produces a
   product can have a different lifetime than its product. Also, the duration and
   degree of the impact to the environment by the process can be quite different than
   its product.

14. If OPPT considers different ways of normalizing results, e.g., per unit of
   production or per functional unit (per action, facility, household, etc), it is likely
   to find a measure robust enough that changes of production can be considered
   more realistically when describing the impact of results from P2 activities over a
   wide range of programs.

-------
Charge Question 4:  How do other programs and organizations consider or measure
recurring results?

15. Companies track performance to change behavior, with objectives for the
   behavior change desired. Using this model, OPPT could consider setting specific
   targets for P2 programs.

16. Experience with other organizations demonstrates that, for P2 metrics to be
   effective, they need to be clear and simple.

17. OPPT should establish metrics only for activities that can be clearly and
   defensibly influenced by the P2 programs.

18. Experience with other organizations demonstrates that it really matters what is
   measured, as performance metrics motivate and demotivate behavior.

19. OPPT may find it helpful to examine the Global Reporting Initiative
   (www.globalreporting.org) and how this organization handles recurring results
   and includes life cycle considerations.

20. The Intergovernmental Panel on Climate Change has developed methods of
   accounting for recurring and cumulative emission reductions achieved by control
   measures for greenhouse gases. The IPCC methods may serve as useful models
   for counting recurring results for P2 programs.

21. Industrial trade associations have developed methods for counting recurring and
   cumulative results of pollution prevention activities, e.g., the Aluminum
   Association, the  American Chemistry Council, and the World Business Council
   for Sustainable Development.

22. The U.S. Center for Disease Control and the World Health Organization track
   recurring results from interventions for disease prevention.  The methodologies
   developed may be useful in determining methods for counting recurring results
   from P2 program interventions.
Charge Question 5:  How can EPA best use the initial benchmark literature search
findings to help further inform recurring results method development?

23. Information in peer-reviewed journals should be explored in much greater depth.

24. It would be useful to explore how other organizations (e.g., IPCC) reflect
   uncertainty in new or recurring results from interventions.

-------
25. The EPA should examine international sources to inform recurring results method
   development, e.g., emission accounting compendia developed by the Institute for
   International Studies and Training (IIST) and by the European Union.
Other Key Observations

26. There are multiple audiences for the P2 program metrics:  Measures of P2 can be
   used to evaluate OPPT's programs, as OMB does in PART evaluations.  Another
   important audience is the companies/organizations being solicited to participate in
   the program. While EEC recognizes that development of metrics to document
   performance is a priority, it suggests some thought be given to development of
   additional metrics that capture innovation, economic competitiveness, and the
   evolution of the P2 discipline that should be directed to the participants in these
   voluntary programs.

27. It is important to convey the barriers to data collection faced by the P2 programs.
   The programs involve voluntary participation and data submittal by the
   organizations that partner with EPA. Too much pressure for data submittal  could
   result in the loss of participation by the partner organizations.  Further, the EPA is
   limited in the surveys it can conduct by federal information collection request
   (ICR) regulations.

28. OPPT has been careful to be conservative in its estimates  of impacts of P2
   program interventions. This care is commendable, but the Committee encourages
   the OPPT to be less conservative in its data compilation and analysis in order to
   demonstrate more clearly the range of benefits.  The current bias toward
   underestimating benefits is counterproductive with respect to attracting more
   organizations as participants in the P2 programs. In this respect there is harm in
   underestimating benefits. Defensible approaches to estimation of benefits that are
   not so conservative could help expand P2 program participation more rapidly.

-------