United States
Environmental Protection
Agency
Office of
The Administrator
EPA
September 1999
&EPA
Office of Cooperative Environmental Management
NATIONAL ADVISORY COUNCIL
,._ , ^ ^ FOR
ENVU^DNMENTAL POLICY AND TECHNOLOGY (NACEPTY
' 1 1
FINAL REPORT AND RECOMMENDATIONS
THE REINVENTION CRITERIA COMMITTEE
IDENTIFICATION OF EVALUATION CRITERIA
FOR EPA'S REINVENTION PROGRAMS
1999
-------
-------
Law Offices
HOLLAND & KNIGHT LLP
2100 Pennsylvania Avenue, N.W.
Suite 400
Washington, D.C. 20037-3202
202-955-3000
FAX 202-955-5564
http://www.hklaw.com
Atlanta
Boca Raton
Boston
Fort Lauderdale
Jacksonville
Lakeland
Melbourne
Mexico City
Miami
New York
Northern Virginia
Orlando
San Francisco
St. Petersburg
Tallahassee
Tampa
Washington, D.C.
West Palm Beach
September 30, 1999 ROBERT L. RHODES
202-457-5943
Internet Address:
rrhodes@hklaw.com
Administrator Carol M. Browner
U.S. Environmental Protection Agency
401 M Street, S.W. (Mail Code 1101)
Washington, D.C. 20460
Dear Administrator Browner:
I am pleased to forward to you the first of two reports that the National
Advisory Council for Environmental Policy and Technology (NACEPT)
Reinvention Criteria Committee (RCC) have been working on.
This report, Identification of Evaluation Criteria for EPA's Reinvention
Programs is the result of your charge to NACEPT that it identify criteria the
Agency could use to evaluate the progress and success of specific reinvention
programs. As EPA commenced its reinvention initiatives, the RCC was able to
work closely with many EPA programs to identify evaluation criteria for
measuring their progress.
Additionally, the Committee was able to provide recommendations on how
EPA might be able to evaluate reinvention, as a whole, as well as an evaluation
framework and set of criteria to measure progress.
I hope that you will find this report useful as EPA continues to streamline
its regulatory processes and introduce innovations and new approaches to
environmental protection. The RCC and NACEPT Council welcome your review
and response to their work.
Sincerely,
Robert L. Rhodes, Jr.
Chair, NACEPT
RLRrnct
Attachment
-------
-------
NOTICE
This report and set of recommendations have been written as part of the activities of the
National Advisory Council for Environmental Policy and Technology (NACEPT), a public
advisory committee providing extramural policy advice to the Environmental Protection
Agency's Administrator and other officials of the EPA. The Council is structured to provide
balanced, and expert assessment of policy matters related to the effectiveness of the
environmental programs of the United States. This report has not been reviewed for approval by
the EPA and, hence, the contents of this report, and its recommendations, do not necessarily
present the views and policies of the EPA, nor of other agencies in the Executive Branch of the
federal government, nor does mention of trade names, companies, or commercial products
constitute a recommendation or endorsement for use.
-------
-------
National Advisory Council for Environmental Policy and Technology
Reinvention Criteria Committee
September 1999
EXECUTIVE SUMMARY
In March 1995, President Clinton, Vice-President Gore, and EPA Administrator Carol Browner
announced an ambitious agenda to reinvent environmental protection as part of the larger goal of
creating a federal government that works better and costs less. The agenda, which evolved and
broadened over time, cut across all areas of EPA's regulatory responsibility. It focused on
achieving a safer, cleaner environment for the public by correcting the everyday inefficiencies
and obstacles that limited the effectiveness of environmental programs, and by designing and
testing fundamentally new approaches and systems equal to current and future environmental and
public health challenges.1-
In the Spring 1996, the Deputy Administrator requested that NACEPT help the Agency identify
criteria it could use to evaluate the progress and success of specific reinvention programs. This
request resulted in the formation of the NACEPT Reinvention Criteria Committee. The
Committee did not evaluate EPA's reinvention efforts as the Agency was in the early stages of
the reinvention process. Later in that year, the Committee was asked to broaden its focus by,
identifying evaluation criteria for measuring the progress of reinvention activities in several of
the more traditional EPA programs and to provide advice on how to evaluate reinvention as a
whole.
The Committee met several times between April 1996 and November 1997 (Attachment A). The
Committee dialogued with the reinvention program managers and several regional reinvention
ombudsmen on ways that their programs could be evaluated. This real-time feedback and an
interim report developed in March 1998 allowed for pertinent suggestions to be incorporated as
the reinvention programs were being developed. An evaluation framework and criteria that EPA
could use to measure the success of relevant reinvention programs were recommended as
follows:
Evaluation Framework:
• Each reinvention program should have a clear and succinct statement of its particular
goals .and objectives.
• Based upon those identified goals and objectives, EPA should develop a specific
evaluation strategy for each program.
• In order for reinvention to be successful, it should involve a more pervasive
organizational culture change within the Agency, beyond the specific reinvention
programs.
' U.S. EPA, Office of the Administrator, "Managing for Better Environmental Results, A Two-Year Anniversary
Report on Reinventing Environmental Protection," EPA 100-R-97-004, March 1997.
-------
National Advisory Council for Environmental Policy and Technology
Reinvention Criteria Committee
September 1999
Key Recommendations and Conclusions:
• In order to evaluate the progress and success of reinvention, EPA must clearly define its
purpose, goals, and objectives.
• There are actually multiple goals for reinvention. EPA should make each goal explicit.
• To succeed, the Agency's reinvention efforts must:
Maintain public confidence
Involve stakeholders j
Recognize that EPA is a partner hi environmental protection
Serve as a catalyst for change in collaboration with other stakeholders.
• EPA should address the question of what as needed to meet the environmental challenges
of the future.
• It is inevitable that cross-program comparisons will be made. To minimize "bureaucratic
Darwinism," EPA should establish a more explicit process for deciding which programs
are worthy and feasible to pursue.
• Reinvention must eventually move from its current pilot/laboratory phase to systemic
change.
• There are opportunities for consolidation and coordination of individual reinvention
programs. In order to pursue them, EPA should group similar types of reinvention
programs for evaluation. This will help to clarify the commonalties and differences. It
will also help the Agency to extract the "lessons learned."
• Barriers to the adoption of positive innovations 'include uniformity myths and risk
aversion. Both should be recognized by EPA, and strategies should be developed to
overcome them.
• Just as reinvention is an evolutionary process, the function of the Office of Reinvention
should evolve over time.
i
The Committee's preliminary recommendations were presented to the NACEPT Council at the
November 5-6, 1997 plenary meeting. The preh'minary recommendations were also documented
in the March 1998 interim report.
It should be noted that this Committee has continued to address issues affecting the Agency's
reinvention activities. However, in April 1998 changes were made to the membership because
several members either changed sector affiliation or their terms expired. The Committee was
11
-------
National Advisory Council for Environmental Policy and Technology
Reinvention Criteria Committee
September 1999
reconstituted with several new members and focused on a revised charge affecting the Agency's
reinvention efforts. The reconstituted Committee has completed its work in response to the new
charge and will submit findings and recommendations to the NACEPT Council for review and
approval in a separate report.
in
-------
-------
National Advisory Council for Environmental Policy and Technology
Reinvention Criteria Committee
September 1999
TABLE OF CONTENTS
EXECUTIVE SUMMARY
I. INTRODUCTION
II. APPROACH
HI. RECOMMENDATIONS
ATTACHMENT A - Schedule of Committee Meetings
ATTACHMENT B -List of Committee Members
ATTACHMENT C -Reinvention Programs Addressed by the Committee
Page
i
1
3
5
9
10
11
-------
-------
National Advisory Council for Environmental Policy and Technology
Reinvention Criteria Committee . September 1999
Identification of Evaluation Criteria For EPA's Reinvention Programs
Final Report and Recommendations
of
The Reinvention Criteria Committee
I. INTRODUCTION
The National Advisory Council for Environmental Policy and Technology (NACEPT) is a public
advisory committee originally chartered on July 7, 1988. The Council provides
recommendations and advice to the Administrator and other EPA officials on ways to improve
the development and implementation of domestic and international environmental management
policies and programs. The NACEPT membership includes senior-level officials and experts
representing federal, state, and local government agencies and tribal organizations,
business/industry, academia, environmental organizations, and NGOs. As principal constituents
and stakeholders of EPA, these members provide advice and recommendations on policy
issues/questions and serve as a sounding board for new strategies that the Agency is developing.
In March 1995, President Clinton, Vice-President Gore, and EPA Administrator Carol Browner
announced an ambitious agenda to reinvent environmental protection as part of the larger goal of
creating a federal government that works better and costs less. The agenda, which evolved and
broadened over time, cut across all areas of EPA's regulatory responsibility. It focused on
achieving a safer, cleaner environment for the public by correcting the everyday inefficiencies
and obstacles that limited the effectiveness of environmental programs, and by designing and
testing fundamentally new approaches and systems equal to current and future environmental and
public health challenges.
EPA's reinvention agenda included 25 high-priority projects, which cut across all areas of
environmental regulatory responsibility. They were designed to promote innovation and
flexibility, increase community participation and partnerships, improve compliance with
environmental laws, and cut red tape and paperwork. K
EPA also took steps to bolster reinvention internally by announcing a new Office of Reinvention
led by an Associate Administrator. Senior Agency officials were appointed to serve as
"Reinvention Ombudsmen" for their respective program areas, based on the recognition that
reinvention is most successful when senior managers assume an active role.
In April 1996, the Deputy Administrator requested that NACEPT help the Agency identify
criteria it could use to evaluate the progress and success of specific reinvention programs. The
' U.S. EPA, Office of the Administrator, "Managing for Better Environmental Results, A Two-Year Anniversary
Report on Reinventing Environmental Protection," EPA 100-R-97-004, March 1997.
1
-------
National Advisory Council for Environmental Policy and Technology
Reinvention Criteria Committee
September 1999
Reinvention Criteria Committee was convened to address this charge (see Attachment B). The
Committee did not evaluate EPA's reinvention efforts as the Agency was in the early stages of
the reinvention process. Later that year, the Committee was asked to broaden its focus by
identifying evaluation criteria for measuring the progress of reinvention activities in several of
the more traditional programs and to provide advice on how to evaluate reinvention as a whole.
This report is being submitted at this time to serve as a record of the Committee's "real-time"
advice and recommendations on the identification of evaluation criteria. The Committee's
preliminary recommendations were presented to the NACEPT Council at the November 5-6,
1997 plenary meeting. The interim report documenting the preliminary recommendations was
prepared in March 1998.
It should be noted that this Committee has continued to address issues affecting the Agency's
reinvention activities. Changes were made to the membership due to affiliation changes and
expiration of terms. The Committee was reconstituted in April 1998 with several new members
and focused on a revised charge affecting the Agency's reinvention efforts. The Committee has
completed its work in response to the new charge and will submit recommendations to the
NACEPT Council for review and approval in a separate report.
-------
National Advisory Council for Environmental Policy and Technology
Reinvention Criteria Committee
September 1999
II. APPROACH
' The Committee began by reviewing and dialoging with the managers of specific reinvention
programs. The Committee concluded that each reinvention program needed to have a discrete set
of goals and objectives, which were often lacking. However, several reinvention programs
reflected that careful thought had been given to identifying evaluation measures.
The Committee dialogued with the reinvention program managers and provided "real time"
feedback, as the reinvention programs were evolving. The members felt that it was important to
focus on whether the goals and objectives of the reinvention programs had been clearly
articulated within the scope of reinvention overall.
As the discussions progressed, the Committee shifted its focus to identifying the steps required to
develop an evaluation strategy. Several overarching issues were identified and communicated in
letters to the Deputy Administrator on October 1996 and April 1997.
The Agency responded in letters dated December 13,1996 and May 12,1997, respectively. The
Deputy Administrator acknowledged that the preliminary findings were consistent with the
Agency's ideas about evaluation criteria to be considered. He urged the committee to continue
with its efforts and requested that the focus be expanded to include:
• Examining the reinvention activities in the more traditional media offices and providing
insight into how their progress can be measured.
• Recommending criteria for measuring the success of reinvention as a whole.
In focusing on these broader issues, the committee framed its deliberations with the following
questions:
• What evaluation criteria would be appropriate for reinvention as a whole?
• Is it possible/useful to employ cross-program comparisons?
• Are there opportunities for using "lessons learned" to move to systemic change?
• • Is there consistency among the reinvention themes? Are there opportunities for
consolidation/coordination?
• What are the barriers to reinvention and what factors lead to the adoption of positive
innovations?
-------
National Advisory Council for Environmental Policy and Technology
Reinvention Criteria Committee
September 1999
As the Committee began to define these overarching issues, the need to dialog with the regional
reinvention managers and other senior level decision makers was recognized. The Committee
also recognized the need to consider how these issues were disseminated into the Agency's
overall mission. Senior managers were invited to the meetings to share their perspectives on the
overarching issues and barriers to reinvention.
Section m. of this report contains the Committee's final recommendations on an evaluation
framework, and other key recommendations and conclusions.
-------
National Advisory Council for Environmental Policy and Technology
Reinvention Criteria Committee
September 1999
III. RECOMMENDATIONS
The Committee began with the premise of reinvention-that it is important to find more efficient
ways of conducting the business of environmental protection, or as it has succinctly been
summarized, make the Agency's environmental protection efforts "cleaner, cheaper, and
smarter." The Committee added to that succinct summary, that "maintenance of public
confidence" should also be an explicit goal. Within this framework, the committee dialogued
with the reinvention program managers and several regional reinvention ombudsmen on ways
that the programs could be evaluated. This real-time feedback allowed for pertinent suggestions
to be incorporated as the programs were evolving. A total of eighteen reinvention programs were
reviewed by the Committee (see attachment C).
Evaluation Framework:
• Each reinvention program should have a clear and succinct statement of its
particular goals and objectives. Several programs defined criteria for use in
implementing their efforts. Examples included criteria for determining which individual
projects or facilities will be part of the program, or criteria for evaluating individual grant
proposals. However, the overall program goals and objectives were often not clear or
explicit. Within this framework, the Committee recommends adding environmental
improvements (i.e., environmental results) as an explicit goal of each reinvention program.
The Committee also noted that several programs appeared to focus primarily on improving
the process.
• Based upon those identified goals and objectives, EPA should develop a specific
evaluation strategy for each program. The Committee believes that it is not feasible to
develop meaningful evaluation criteria for the programs without first articulating the goals
the programs are designed to achieve.
• The evaluation strategies should include a hierarchy of integrated measures that
range from short-term to long-term components. There is a continuum of the types of
measures/that should be examined, ranging from activity and management elements in the
short-term, to longer term measures that focus on environmental outcomes. While it may
not be feasible in all of the programs to define actual measures in environmental media,
EPA can or should develop intermediate measures that relate to specific environmental
outcomes (such as emissions reductions or pollution prevention measures). The logical
integration of these measures may enable the program activity measures to ultimately
focus on improved environmental outcomes. The activity or operation measures would
also include indicators of whether the "cheaper and smarter" goals are being met.
-------
National Advisory Council for Environmental Policy and Technology
Reinvention Criteria Committee ;
September 1999
• The evaluation strategies should also include measures of environmental quality
improvement, stakeholder involvement and satisfaction, and maintenance of publlic
confidence. Most of the reinvention programs explicitly stated the desire for greater
stakeholder involvement. However, some of the programs had not considered measures of
stakeholder involvement or satisfaction as components of their evaluation strategy. Such
measures can be either direct or indirect. The direct measures, such as simply asking for
stakeholders' opinions about the program, appeared to be to most frequently overlooked,
although most obvious.
• In order for reinvention to be successful, it should involve a more pervasive
organizational culture change within the Agency, beyond the specific reinvention
programs. EPA should ensure that the lessons learned from the implementation of the
programs are diffused into general practice. Evidence of implementation of this could
include incentives, such as increased concentration of Agency resources in areas with
promising evaluation results, with simultaneous disincentives such as decreased (or
eliminated) allocation of Agency resources in areas with inferior evaluation results. An
Agency review of factors leading to the adoption of positive innovations may be useful.
At the request of the Agency, the committee expanded its focus to include:
j:
• Examining the reinvention activities in the more traditional media offices and providing
insight into how their progress can be measured.
• Recommending criteria for measuring the success of reinvention as a whole.
Key Recommendations and Conclusions:
• In order to evaluate the progress and success of reinvention, EPA must clearly define
its purpose, goals, and objectives. The sound-bite "cleaner, cheaper, smarter" is not, In
itself, sufficient. It only asks whether the environment is better relative to today's
baseline. The theme appears agnostic on the question of how much "better" is enough.
That is, the baseline is known but not the goals or milestones/benchmarks, even in
qualitative terms.
• The Committee believes that there are actually multiple goals for reinvention. EPA
should make each goal explicit For example, reinvention is seeking and testing both:
a) ways to make the current regulatory system work better, and b) new alternative
approaches to the regulatory system.
-------
National Advisory Council for Environmental Policy and Technology
Reinvention Criteria Committee
September 1999
To succeed, the Agency's reinvention efforts must:
— Maintain public'confidence ' ~
~ Involve stakeholders
— Recognize that EPA is a partner in environmental protection
— Serve as a catalyst for change in collaboration with other stakeholders.
EPA should address the question of what is. needed to meet the environmental
challenges of the future. This will help the Agency to define the purpose of reinvention.
• It is inevitable that cross-program comparisons will be made. To minimize
"bureaucratic Darwinism," EPA should establish a more explicit process.for
deciding which programs are worthy and feasible to pursue. This approach should
involve hypothesis testing and measures of success, including at a minimum, qualitative
estimates of benefits and costs.
Reinvention must eventually move from its current pilot/laboratory phase to
systemic change. To do so, EPA must explore options for how successful "experimental"
reinvention programs can be transitioned and more broadly incorporated into the Agency's
environmental protection mission.
There are opportunities for consolidation and coordination of individual reinvention
programs. In order to pursue them, EPA should group similar types of reinvention
programs for evaluation. This will help to clarify the commonalties and differences. It
will also help the Agency to extract the "lessons learned."
Barriers to the adoption of positive innovations include uniformity myths and risk
aversion. Both should be recognized by EPA, and strategies should be developed to
overcome them. There is a tendency for people to regard others as having uniform
characteristics. Such views may serve as barriers to effective adoption of environmental
and/or regulatory innovations. Uniformity myths block effective communication and
effective relationship building. EPA's reinvention effort is short-sighted if it does not
conduct a systemic assessment of such barriers and collaborate with its constituents.
Just as reinvention is an evolutionary process, the function of the Office of
Reinvention should evolve over time. At this juncture, this Office should:
— Promote better communication throughout the Agency regarding the reinvention
programs and philosophy.
— Coordinate and integrate the reinvention efforts.
-------
National Advisory Council for Environmental Policy and Technology
Reinvention Criteria Committee
September 1999
— Serve as evaluator and diagnostician (through statutes, regulations, etc.) of the
"lessons learned."
-- Develop a plan for applying the "lessons learned" from the individual reinvention
programs into broader implementation by program offices.
- Focus on using the "lessons learned" to change the Agency's culture and system for
environmental protection. !
— Provide training across the Agency.
— Advocate and promote reinvention.
The Committee commends EPA for undertaking the reinvention effort, and believes that it is a
productive approach for the Agency and its stakeholders and for environmental protection in the
future. EPA is also to be commended for recognizing the importance of evaluating its efforts and
attempting to develop criteria for evaluation early hi the process. Evaluation criteria must be
defined early in program development to provide interim feedback on program direction (or
needed redirection) as well as for later measures of progress or success in the programs.
-------
National Advisory Council for Environmental Policy and Technology
Reinvention Criteria Committee
September 1999
ATTACHMENT A
SCHEDULE OF COMMITTEE MEETINGS
Dates & Location
April 17-18,1996 - Arlington, Virginia
July 24 -25, 1996 - Washington, DC
September 10-11,1996 - Washington, DC "
December 11 -12, 1996 - Washington, DC
April 2-3, 1997 - . Alexandria, Virginia
July 16 -17, 1997 ' - Washington, DC
September 29, 1997 - Alexandria, Virginia
November 5 - 6, 1997 - Arlington, Virginia
-------
ATTACHMENT B
LIST OF COMMITTEE MEMBERS
Chair: Ms. Holly Stoerker
Executive Director
Upper Mississippi River Basin Association
St. Paul, Minnesota
Designated Federal Officer:
Gwendolyn C.L. Whitt
Office of Cooperative
Environmental Management
U.S. Environmental Protection Agency
Washington, B.C.
Dr. Edwin H. (Toby) Clark II
President, Clean Sites, Inc.
Alexandria, Virginia
Ms. Catharine M. DeLacy
Corporate Vice President
Health, Environment, and Safety
AlliedSignal, Inc.
Morristown, New Jersey
Ms. Wilma Delaney
Vice President
Environmental and Regulatory Issues
Dow Chemical Company
Washington, DC
Mr. Nicholas A. DiPasquale
Director
Division of Air and Waste Management
Department of Natural Resources and
Environmental Control
Dover, Delaware
Mr. Gerald Digerness
National Association
of Conservation Districts
Sumas, Washington
Mr. Edward Garvey
Chair, State of Minnesota Public Utilities
Commission
St. Paul, Minnesota
Dr. George Hallberg
Associate Director, Hygenic Laboratory
University of Iowa
Iowa City, Iowa
Dr. Walter Handy, Jr.
Assistant Health Commissioner
Cincinnati Department of Health
Cincinnati, Ohio
Ms. Elise Hoerath
Counsel, Environmental Quality Division
National Wildlife Federation
Washington, DC
Mr. Kevin Mills
Sr. Attorney/Director
Pollution Prevention Alliance
Environmental Defense Fund
Washington, DC
Dr. Joseph Sullivan
49 Bittersweet Trail
Wilton, Connecticut
-------
National Advisory Council for Environmental Policy and Technology
Reinvention Criteria Committee
September 1999
ATTACHMENT C
REINVENTION PROGRAMS ADDRESSED BY THE COMMITTEE
Common Sense Initiative (CSI)
Self-Policing/Audit Policy
Self Certification
Sustainable Development Challenge Grants
National Environmental Performance Partnership System
Project XL '
Environmental Leadership Program
National Environmental Goals Project
Small Business Compliance Assistance Centers
Brownfields Program
Consolidated Federal Air Rule
Sustainable Industries Project
One Stop Reporting
Sector Facility Indexing
Open Market Air Trading
Pollution Prevention Voluntary Programs
Electronic Public Access
Effluent Trading in Watersheds
11
-------
------- |