United States
              Environmental Protection
              Agency
                    Office of
                    The Administrator
EPA
September 1999
&EPA
Office of Cooperative Environmental Management
               NATIONAL ADVISORY COUNCIL
     ,._  ,        ^     ^   FOR	
    ENVU^DNMENTAL POLICY AND TECHNOLOGY (NACEPTY
                                              ' 1   1
           FINAL REPORT AND RECOMMENDATIONS
           THE REINVENTION CRITERIA COMMITTEE
       IDENTIFICATION OF EVALUATION CRITERIA
           FOR EPA'S REINVENTION PROGRAMS
                                1999

-------

-------
 Law Offices
 HOLLAND & KNIGHT LLP
 2100 Pennsylvania Avenue, N.W.
 Suite 400
 Washington, D.C. 20037-3202

 202-955-3000
 FAX 202-955-5564
 http://www.hklaw.com
Atlanta
Boca Raton
Boston
Fort Lauderdale
Jacksonville
Lakeland
Melbourne
Mexico City
Miami
New York
Northern Virginia
Orlando
San Francisco
St. Petersburg
Tallahassee
Tampa
Washington, D.C.
West Palm Beach
  September 30, 1999                                      ROBERT L. RHODES
                                                          202-457-5943
                                                          Internet Address:
                                                          rrhodes@hklaw.com

 Administrator Carol M. Browner
 U.S. Environmental Protection Agency
 401 M Street, S.W. (Mail Code 1101)
 Washington,  D.C.  20460

 Dear Administrator Browner:

       I am pleased to forward to you the first of two reports that the National
 Advisory Council for Environmental Policy and Technology (NACEPT)
 Reinvention Criteria Committee (RCC) have been working on.

       This report, Identification of Evaluation Criteria for EPA's Reinvention
 Programs is the result of your charge to NACEPT that it identify criteria the
 Agency could use to evaluate the progress and success of specific reinvention
 programs. As EPA commenced its reinvention initiatives, the RCC was able to
 work closely with many EPA programs to identify evaluation criteria for
 measuring their progress.

       Additionally, the Committee was able to provide recommendations on how
 EPA might be able to evaluate reinvention, as a whole, as well as an evaluation
 framework and set of criteria to measure progress.

       I hope that you will find this report useful as EPA continues to streamline
 its regulatory processes and introduce innovations and new approaches to
 environmental protection.  The RCC and NACEPT Council welcome your review
 and response to their work.
                                     Sincerely,
                                     Robert L. Rhodes, Jr.
                                     Chair, NACEPT
RLRrnct
Attachment

-------

-------
                                      NOTICE
       This report and set of recommendations have been written as part of the activities of the
National Advisory Council for Environmental Policy and Technology (NACEPT), a public
advisory committee providing extramural policy advice to the Environmental Protection
Agency's Administrator and other officials of the EPA. The Council is structured to provide
balanced, and expert assessment of policy matters related to the effectiveness of the
environmental programs of the United States. This report has not been reviewed for approval by
the EPA and, hence, the contents of this report,  and its recommendations, do not necessarily
present the views and policies of the EPA, nor of other agencies in the Executive Branch of the
federal government, nor does mention of trade names, companies, or commercial products
constitute a recommendation or endorsement for use.

-------

-------
 National Advisory Council for Environmental Policy and Technology
 Reinvention Criteria Committee
                                                                          September 1999
                               EXECUTIVE SUMMARY

 In March 1995, President Clinton, Vice-President Gore, and EPA Administrator Carol Browner
 announced an ambitious agenda to reinvent environmental protection as part of the larger goal of
 creating a federal government that works better and costs less. The agenda, which evolved and
 broadened over time, cut across all areas of EPA's regulatory responsibility. It focused on
 achieving a safer, cleaner environment for the public by correcting the everyday inefficiencies
 and obstacles that limited the effectiveness of environmental programs, and by designing and
 testing fundamentally new approaches and systems equal to current and future environmental and
 public health challenges.1-

 In the Spring 1996, the Deputy Administrator requested that NACEPT help the Agency identify
 criteria it could use to evaluate the progress and success of specific reinvention programs. This
 request resulted in the formation of the NACEPT Reinvention Criteria Committee. The
 Committee did not evaluate EPA's reinvention efforts as the Agency was in the early stages of
 the reinvention process. Later in that year, the Committee was asked to broaden its focus by,
 identifying evaluation criteria for measuring the progress of reinvention activities in several of
 the more traditional EPA programs and to provide advice on how to evaluate reinvention as a
 whole.

 The Committee met several times between April 1996 and November 1997 (Attachment A).  The
 Committee dialogued with the reinvention program managers and several regional reinvention
 ombudsmen on ways that their programs could be evaluated. This real-time feedback and an
 interim report developed in March 1998 allowed for pertinent suggestions to be incorporated as
 the reinvention programs were being developed. An evaluation framework and criteria that EPA
 could use to measure the success of relevant reinvention programs were recommended as
 follows:

 Evaluation Framework:

   •   Each reinvention program should have a clear and succinct statement of its particular
       goals .and objectives.

   •   Based upon those identified goals and objectives, EPA should develop a specific
       evaluation strategy for each program.

   •   In order for reinvention to be successful, it should involve a more pervasive
       organizational culture change within the Agency, beyond the specific reinvention
       programs.

 ' U.S. EPA, Office of the Administrator, "Managing for Better Environmental Results, A Two-Year Anniversary
Report on Reinventing Environmental Protection," EPA 100-R-97-004, March 1997.

-------
 National Advisory Council for Environmental Policy and Technology
 Reinvention Criteria Committee
                                                                          September 1999
 Key Recommendations and Conclusions:

    •  In order to evaluate the progress and success of reinvention, EPA must clearly define its
       purpose, goals, and objectives.

    •  There are actually multiple goals for reinvention. EPA should make each goal explicit.

    •  To succeed, the Agency's reinvention efforts must:
          Maintain public confidence
          Involve stakeholders                j
          Recognize that EPA is a partner hi environmental protection
          Serve as a catalyst for change in collaboration with other stakeholders.

    •   EPA should address the question of what as needed to meet the environmental challenges
       of the future.

    •   It is inevitable that cross-program comparisons will be made. To minimize "bureaucratic
       Darwinism," EPA should establish a more explicit process for deciding which programs
       are worthy and feasible to pursue.

    •   Reinvention must eventually move from its current pilot/laboratory phase to systemic
       change.

    •   There are opportunities for consolidation and coordination of individual reinvention
       programs.  In order to pursue them, EPA should group similar types of reinvention
       programs for evaluation.  This will help to clarify the commonalties and differences. It
       will also help the Agency to extract the "lessons learned."

    •   Barriers to the adoption of positive innovations 'include uniformity myths and risk
       aversion. Both should be recognized by EPA,  and strategies should be developed to
       overcome them.

    •   Just as reinvention is an evolutionary process, the function of the Office of Reinvention
       should evolve over time.
	                                          i
The Committee's preliminary recommendations were presented to the NACEPT Council at the
November 5-6, 1997 plenary meeting. The preh'minary recommendations were also documented
in the March 1998 interim report.

It should be noted that this Committee has continued to address issues affecting the Agency's
reinvention activities. However, in April 1998 changes were made to the membership because
several members either changed sector affiliation or their terms expired.  The Committee was
                                           11

-------
National Advisory Council for Environmental Policy and Technology
Reinvention Criteria Committee
                                                                           September 1999
reconstituted with several new members and focused on a revised charge affecting the Agency's
reinvention efforts. The reconstituted Committee has completed its work in response to the new
charge and will submit findings and  recommendations to the NACEPT Council for review and
approval in a separate report.
                                          in

-------

-------
National Advisory Council for Environmental Policy and Technology
Reinvention Criteria Committee
                                                                September 1999
                            TABLE OF CONTENTS



EXECUTIVE SUMMARY

I.   INTRODUCTION

II.  APPROACH

HI.  RECOMMENDATIONS

ATTACHMENT A - Schedule of Committee Meetings

ATTACHMENT B -List of Committee Members

ATTACHMENT C -Reinvention Programs Addressed by the Committee
Page

  i

  1

  3

  5

  9

  10

  11

-------

-------
 National Advisory Council for Environmental Policy and Technology
 Reinvention Criteria Committee	.	September 1999

           Identification of Evaluation Criteria For EPA's Reinvention Programs
                           Final Report and Recommendations
                                           of
                           The Reinvention Criteria Committee

                                  I. INTRODUCTION

 The National Advisory Council for Environmental Policy and Technology (NACEPT) is a public
 advisory committee originally chartered on July 7, 1988. The Council provides
 recommendations and advice to the Administrator and other EPA officials on ways to improve
 the development and implementation of domestic and international environmental management
 policies and programs. The NACEPT membership includes senior-level officials and experts
 representing  federal, state, and local government agencies and tribal organizations,
 business/industry, academia, environmental organizations, and NGOs. As principal constituents
 and stakeholders of EPA, these members provide advice and recommendations on policy
 issues/questions and serve as a sounding board for new strategies that the Agency is developing.

 In March 1995, President Clinton, Vice-President Gore, and EPA Administrator Carol Browner
 announced an ambitious agenda to reinvent environmental protection as part of the larger goal of
 creating a federal government that works better and costs less. The agenda, which evolved and
 broadened over time, cut across all areas of EPA's regulatory  responsibility. It focused on
 achieving a safer, cleaner environment for the public by correcting the everyday inefficiencies
 and obstacles that limited the effectiveness of environmental programs, and by designing and
 testing fundamentally new approaches and systems equal to current and future environmental and
 public health challenges.

 EPA's reinvention agenda included 25 high-priority projects, which cut across all areas of
 environmental regulatory responsibility. They were designed  to promote innovation and
 flexibility, increase community participation and partnerships, improve compliance with
 environmental laws, and cut red tape and paperwork. K

 EPA also took steps to bolster reinvention internally by announcing a new Office of Reinvention
 led by an Associate Administrator.  Senior Agency officials were appointed to serve as
 "Reinvention Ombudsmen" for their respective program areas, based on the recognition that
 reinvention is most successful when senior managers assume an active role.

 In April 1996, the Deputy Administrator requested that NACEPT help the Agency identify
 criteria it could use to evaluate the progress and success of specific reinvention programs. The
 ' U.S. EPA, Office of the Administrator, "Managing for Better Environmental Results, A Two-Year Anniversary
Report on Reinventing Environmental Protection," EPA 100-R-97-004, March 1997.

                                          1

-------
 National Advisory Council for Environmental Policy and Technology
 Reinvention Criteria Committee
                                                                        September 1999
Reinvention Criteria Committee was convened to address this charge (see Attachment B).  The
Committee did not evaluate EPA's reinvention efforts as the Agency was in the early stages of
the reinvention process. Later that year, the Committee was asked to broaden its focus by
identifying evaluation criteria for measuring the progress of reinvention activities in several of
the more traditional programs and to provide advice on how to evaluate reinvention as a whole.

This report is being submitted at this time to serve as a record of the Committee's "real-time"
advice and recommendations on the identification of evaluation criteria. The Committee's
preliminary recommendations were presented to the NACEPT Council at the November 5-6,
1997 plenary meeting.  The interim report documenting the preliminary recommendations was
prepared in March 1998.

It should be noted that this Committee has continued to address issues affecting the Agency's
reinvention activities.  Changes were  made to the membership due to affiliation changes and
expiration of terms. The Committee was reconstituted in April 1998 with several new members
and focused on a revised charge affecting the Agency's reinvention efforts.  The Committee has
completed its work in response to the new charge and will submit recommendations to the
NACEPT Council for review and approval in a separate report.

-------
 National Advisory Council for Environmental Policy and Technology
 Reinvention Criteria Committee
                                                                          September 1999
                                   II.  APPROACH

' The Committee began by reviewing and dialoging with the managers of specific reinvention
programs.  The Committee concluded that each reinvention program needed to have a discrete set
of goals and objectives, which were often lacking. However, several reinvention programs
reflected that careful thought had been given to identifying evaluation measures.

The Committee dialogued with the reinvention program managers and provided "real time"
feedback, as the reinvention programs were evolving. The members felt that it was important to
focus on whether the goals and objectives of the reinvention programs had been clearly
articulated within the scope of reinvention overall.

As the discussions progressed, the Committee shifted its focus to identifying the steps required to
develop an evaluation strategy. Several overarching issues were identified and communicated in
letters to the Deputy Administrator on October 1996 and April 1997.

The Agency responded in letters dated December 13,1996 and May 12,1997, respectively.  The
Deputy Administrator acknowledged that the preliminary findings were consistent with the
Agency's ideas about evaluation criteria to be considered.  He urged the committee to continue
with its efforts and requested that the focus be expanded to include:

    •   Examining the reinvention activities in the more traditional media offices and providing
       insight into how their progress can be measured.

    •   Recommending criteria for measuring the success of reinvention as a whole.

In focusing on these broader issues, the committee framed its deliberations with the following
questions:

   •   What evaluation criteria would be appropriate for reinvention as a whole?

   •   Is it possible/useful to employ cross-program comparisons?

   •   Are there opportunities for using "lessons learned" to move to systemic change?

   • •   Is there consistency among the reinvention themes? Are there opportunities for
       consolidation/coordination?

   •   What are the barriers to reinvention and what factors lead to the adoption of positive
       innovations?

-------
National Advisory Council for Environmental Policy and Technology
Reinvention Criteria Committee
                                                                           September 1999
As the Committee began to define these overarching issues, the need to dialog with the regional
reinvention managers and other senior level decision makers was recognized. The Committee
also recognized the need to consider how these issues were disseminated into the Agency's
overall mission.  Senior managers were invited to the meetings to share their perspectives on the
overarching issues and barriers to reinvention.

Section m. of this report contains the Committee's final recommendations on an evaluation
framework, and other key recommendations and conclusions.

-------
National Advisory Council for Environmental Policy and Technology
Reinvention Criteria Committee
                                                                         September 1999
                           III.  RECOMMENDATIONS

The Committee began with the premise of reinvention-that it is important to find more efficient
ways of conducting the business of environmental protection, or as it has succinctly been
summarized, make the Agency's environmental protection efforts "cleaner, cheaper, and
smarter." The Committee added to that succinct summary, that "maintenance of public
confidence" should also be an explicit goal. Within this framework, the committee dialogued
with the reinvention program managers and several regional reinvention ombudsmen on ways
that the programs could be evaluated.  This real-time feedback allowed for pertinent suggestions
to be incorporated as the programs were evolving. A total of eighteen reinvention programs were
reviewed by the Committee (see attachment C).

Evaluation Framework:

    •  Each reinvention program should have a clear and succinct statement of its
      particular goals and objectives. Several programs defined criteria for use in
      implementing their efforts. Examples included criteria for determining which individual
      projects or facilities will be part of the program, or criteria for evaluating individual grant
      proposals.  However, the overall program goals and objectives were often not clear or
      explicit. Within this framework, the Committee recommends adding environmental
      improvements (i.e., environmental results) as an explicit goal of each reinvention program.
      The Committee also noted that several programs appeared to focus primarily on improving
      the process.

   • Based upon those identified goals and objectives, EPA should develop a specific
     evaluation strategy for each program.  The Committee believes that it is not feasible to
     develop meaningful evaluation criteria for the programs without first articulating the goals
     the programs are designed to achieve.

   • The evaluation strategies should include a hierarchy of integrated measures that
     range from short-term to long-term components. There is a continuum of the types of
     measures/that should be examined, ranging from activity and management elements in the
     short-term, to longer term measures that focus on environmental outcomes. While it may
     not be feasible in all of the programs to define actual measures in environmental media,
     EPA can or should develop intermediate measures that relate to specific environmental
     outcomes (such as emissions reductions or pollution prevention measures). The logical
     integration of these measures may enable the program activity measures to ultimately
     focus on improved environmental outcomes. The activity or operation measures would
     also include indicators of whether the "cheaper and smarter" goals are being met.

-------
 National Advisory Council for Environmental Policy and Technology
 Reinvention Criteria Committee                  ;
September 1999
    •  The evaluation strategies should also include measures of environmental quality
      improvement, stakeholder involvement and satisfaction, and maintenance of publlic
      confidence.  Most of the reinvention programs explicitly stated the desire for greater
      stakeholder involvement. However, some of the programs had not considered measures of
      stakeholder involvement or satisfaction as components of their evaluation strategy. Such
      measures can be either direct or indirect. The direct measures, such as simply asking for
      stakeholders' opinions about the program, appeared to be to most frequently overlooked,
      although most obvious.

    •  In order for reinvention to be successful, it should involve a more pervasive
      organizational culture change within the Agency, beyond the specific reinvention
      programs. EPA should ensure that the lessons learned from the implementation of the
      programs are diffused into general practice. Evidence of implementation of this could
      include incentives, such as increased concentration of Agency resources in areas with
      promising evaluation results, with simultaneous disincentives such as decreased (or
      eliminated) allocation of Agency resources in areas with inferior evaluation results. An
      Agency review of factors leading to the adoption of positive innovations may be useful.

At the request of the Agency, the committee expanded its focus to include:
                                            j:
    •  Examining the reinvention activities in the more traditional media offices and providing
      insight into how their progress can be measured.

    •  Recommending criteria for measuring the success of reinvention as a whole.
Key Recommendations and Conclusions:

   •  In order to evaluate the progress and success of reinvention, EPA must clearly define
      its purpose, goals, and objectives. The sound-bite "cleaner, cheaper, smarter" is not, In
      itself, sufficient. It only asks whether the environment is better relative to today's
      baseline. The theme appears agnostic on the question of how much "better" is enough.
      That is, the baseline is known but not the goals or milestones/benchmarks, even in
      qualitative terms.

   •  The Committee believes that there are actually multiple goals for reinvention.  EPA
      should make each goal explicit For example, reinvention is seeking and testing both:
      a) ways to make the current regulatory system work better, and b) new alternative
      approaches to the regulatory system.

-------
National Advisory Council for Environmental Policy and Technology
Reinvention Criteria Committee
                                                                         September 1999
      To succeed, the Agency's reinvention efforts must:
      —  Maintain public'confidence                                        '   ~
      ~  Involve stakeholders
      —  Recognize that EPA is a partner in environmental protection
      —  Serve as a catalyst for change in collaboration with other stakeholders.

      EPA should address the question of what is. needed to meet the environmental
      challenges of the future. This will help the Agency to define the purpose of reinvention.

     • It is inevitable that cross-program comparisons will be made. To minimize
      "bureaucratic Darwinism," EPA should establish a more explicit process.for
      deciding which programs are worthy and feasible to pursue. This approach should
      involve hypothesis testing and measures of success, including at a minimum, qualitative
      estimates of benefits and costs.

      Reinvention must eventually move from its current pilot/laboratory phase to
      systemic change. To do so, EPA must explore options for how successful "experimental"
      reinvention programs can be transitioned and more broadly incorporated into the Agency's
      environmental protection mission.

      There are opportunities for consolidation and coordination of individual reinvention
      programs. In order to pursue them, EPA should group similar types of reinvention
      programs for evaluation. This will help to clarify the commonalties and differences. It
      will also help the Agency to extract the "lessons learned."

     Barriers to the adoption of positive innovations include uniformity myths and risk
     aversion.  Both should be recognized by EPA, and strategies should be developed to
     overcome them. There is a tendency for people to regard others as having uniform
     characteristics. Such views may serve as barriers to  effective adoption of environmental
     and/or regulatory innovations. Uniformity myths block effective communication and
     effective relationship building. EPA's reinvention effort is short-sighted if it does not
     conduct a systemic assessment of such barriers and collaborate with its constituents.

     Just as reinvention is an evolutionary process, the function of the Office of
     Reinvention should evolve over time. At this juncture, this Office should:

     —  Promote better communication throughout the Agency regarding the reinvention
        programs and philosophy.

     —  Coordinate and integrate the reinvention efforts.

-------
National Advisory Council for Environmental Policy and Technology
Reinvention Criteria Committee
                                                                           September 1999
      —  Serve as evaluator and diagnostician (through statutes, regulations, etc.) of the
          "lessons learned."

      --  Develop a plan for applying the "lessons learned" from the individual reinvention
          programs into broader implementation by program offices.

      -  Focus on using the "lessons learned" to change the Agency's culture and system for
          environmental protection.            !

      —  Provide training across the Agency.

      —  Advocate and promote reinvention.

The Committee commends EPA for undertaking the reinvention effort, and believes that it is a
productive approach for the Agency and its stakeholders and for environmental protection in the
future. EPA is also to be commended for recognizing the importance of evaluating its efforts and
attempting to develop criteria for evaluation early hi the process. Evaluation criteria must be
defined early in program development to provide interim feedback on program direction (or
needed redirection) as well as for later measures of progress or success in the programs.

-------
National Advisory Council for Environmental Policy and Technology
Reinvention Criteria Committee
                                                                      September 1999
                                 ATTACHMENT A

                     SCHEDULE OF COMMITTEE MEETINGS

                                 Dates & Location

   April 17-18,1996                 -           Arlington, Virginia

   July 24 -25, 1996                  -           Washington, DC

   September 10-11,1996            -           Washington, DC  "

   December 11 -12, 1996             -           Washington, DC

   April 2-3, 1997                   -      .     Alexandria, Virginia

   July 16 -17, 1997          '        -           Washington, DC

   September 29, 1997                -           Alexandria, Virginia

   November 5 - 6, 1997               -           Arlington, Virginia

-------
                                  ATTACHMENT B
                         LIST OF COMMITTEE MEMBERS
 Chair: Ms. Holly Stoerker
 Executive Director
 Upper Mississippi River Basin Association
 St. Paul, Minnesota
 Designated Federal Officer:
 Gwendolyn C.L. Whitt
 Office of Cooperative
  Environmental Management
 U.S. Environmental Protection Agency
 Washington, B.C.
Dr. Edwin H. (Toby) Clark II
President, Clean Sites, Inc.
Alexandria, Virginia

Ms. Catharine M. DeLacy
Corporate Vice President
Health, Environment, and Safety
AlliedSignal, Inc.
Morristown, New Jersey

Ms. Wilma Delaney
Vice President
Environmental and Regulatory Issues
Dow Chemical Company
Washington, DC

Mr. Nicholas A. DiPasquale
Director
Division of Air and Waste Management
Department of Natural Resources and
Environmental Control
Dover, Delaware

Mr. Gerald Digerness
National Association
 of Conservation Districts
Sumas, Washington
 Mr. Edward Garvey
 Chair, State of Minnesota Public Utilities
 Commission
 St. Paul, Minnesota

 Dr. George Hallberg
 Associate Director, Hygenic Laboratory
 University of Iowa
 Iowa City, Iowa

 Dr. Walter Handy, Jr.
 Assistant Health Commissioner
 Cincinnati Department of Health
 Cincinnati, Ohio

 Ms. Elise Hoerath
 Counsel, Environmental Quality Division
 National Wildlife Federation
 Washington, DC

 Mr. Kevin Mills
 Sr. Attorney/Director
 Pollution Prevention Alliance
 Environmental Defense Fund
 Washington, DC

Dr. Joseph Sullivan
49 Bittersweet Trail
Wilton, Connecticut

-------
National Advisory Council for Environmental Policy and Technology
Reinvention Criteria Committee
                                                                       September 1999
                                 ATTACHMENT C

         REINVENTION PROGRAMS ADDRESSED BY THE COMMITTEE

      Common Sense Initiative (CSI)
      Self-Policing/Audit Policy
      Self Certification
      Sustainable Development Challenge Grants
      National Environmental Performance Partnership System
      Project XL                                                 '
      Environmental Leadership Program
      National Environmental Goals Project
      Small Business Compliance Assistance Centers
      Brownfields Program
      Consolidated Federal Air Rule
      Sustainable Industries Project
      One Stop Reporting
      Sector Facility Indexing
      Open Market Air Trading
      Pollution Prevention Voluntary Programs
      Electronic Public Access
      Effluent Trading in Watersheds
                                        11

-------

-------