United States Environmental Protection Agency Office of The Administrator EPA September 1999 &EPA Office of Cooperative Environmental Management NATIONAL ADVISORY COUNCIL ,._ , ^ ^ FOR ENVU^DNMENTAL POLICY AND TECHNOLOGY (NACEPTY ' 1 1 FINAL REPORT AND RECOMMENDATIONS THE REINVENTION CRITERIA COMMITTEE IDENTIFICATION OF EVALUATION CRITERIA FOR EPA'S REINVENTION PROGRAMS 1999 ------- ------- Law Offices HOLLAND & KNIGHT LLP 2100 Pennsylvania Avenue, N.W. Suite 400 Washington, D.C. 20037-3202 202-955-3000 FAX 202-955-5564 http://www.hklaw.com Atlanta Boca Raton Boston Fort Lauderdale Jacksonville Lakeland Melbourne Mexico City Miami New York Northern Virginia Orlando San Francisco St. Petersburg Tallahassee Tampa Washington, D.C. West Palm Beach September 30, 1999 ROBERT L. RHODES 202-457-5943 Internet Address: rrhodes@hklaw.com Administrator Carol M. Browner U.S. Environmental Protection Agency 401 M Street, S.W. (Mail Code 1101) Washington, D.C. 20460 Dear Administrator Browner: I am pleased to forward to you the first of two reports that the National Advisory Council for Environmental Policy and Technology (NACEPT) Reinvention Criteria Committee (RCC) have been working on. This report, Identification of Evaluation Criteria for EPA's Reinvention Programs is the result of your charge to NACEPT that it identify criteria the Agency could use to evaluate the progress and success of specific reinvention programs. As EPA commenced its reinvention initiatives, the RCC was able to work closely with many EPA programs to identify evaluation criteria for measuring their progress. Additionally, the Committee was able to provide recommendations on how EPA might be able to evaluate reinvention, as a whole, as well as an evaluation framework and set of criteria to measure progress. I hope that you will find this report useful as EPA continues to streamline its regulatory processes and introduce innovations and new approaches to environmental protection. The RCC and NACEPT Council welcome your review and response to their work. Sincerely, Robert L. Rhodes, Jr. Chair, NACEPT RLRrnct Attachment ------- ------- NOTICE This report and set of recommendations have been written as part of the activities of the National Advisory Council for Environmental Policy and Technology (NACEPT), a public advisory committee providing extramural policy advice to the Environmental Protection Agency's Administrator and other officials of the EPA. The Council is structured to provide balanced, and expert assessment of policy matters related to the effectiveness of the environmental programs of the United States. This report has not been reviewed for approval by the EPA and, hence, the contents of this report, and its recommendations, do not necessarily present the views and policies of the EPA, nor of other agencies in the Executive Branch of the federal government, nor does mention of trade names, companies, or commercial products constitute a recommendation or endorsement for use. ------- ------- National Advisory Council for Environmental Policy and Technology Reinvention Criteria Committee September 1999 EXECUTIVE SUMMARY In March 1995, President Clinton, Vice-President Gore, and EPA Administrator Carol Browner announced an ambitious agenda to reinvent environmental protection as part of the larger goal of creating a federal government that works better and costs less. The agenda, which evolved and broadened over time, cut across all areas of EPA's regulatory responsibility. It focused on achieving a safer, cleaner environment for the public by correcting the everyday inefficiencies and obstacles that limited the effectiveness of environmental programs, and by designing and testing fundamentally new approaches and systems equal to current and future environmental and public health challenges.1- In the Spring 1996, the Deputy Administrator requested that NACEPT help the Agency identify criteria it could use to evaluate the progress and success of specific reinvention programs. This request resulted in the formation of the NACEPT Reinvention Criteria Committee. The Committee did not evaluate EPA's reinvention efforts as the Agency was in the early stages of the reinvention process. Later in that year, the Committee was asked to broaden its focus by, identifying evaluation criteria for measuring the progress of reinvention activities in several of the more traditional EPA programs and to provide advice on how to evaluate reinvention as a whole. The Committee met several times between April 1996 and November 1997 (Attachment A). The Committee dialogued with the reinvention program managers and several regional reinvention ombudsmen on ways that their programs could be evaluated. This real-time feedback and an interim report developed in March 1998 allowed for pertinent suggestions to be incorporated as the reinvention programs were being developed. An evaluation framework and criteria that EPA could use to measure the success of relevant reinvention programs were recommended as follows: Evaluation Framework: • Each reinvention program should have a clear and succinct statement of its particular goals .and objectives. • Based upon those identified goals and objectives, EPA should develop a specific evaluation strategy for each program. • In order for reinvention to be successful, it should involve a more pervasive organizational culture change within the Agency, beyond the specific reinvention programs. ' U.S. EPA, Office of the Administrator, "Managing for Better Environmental Results, A Two-Year Anniversary Report on Reinventing Environmental Protection," EPA 100-R-97-004, March 1997. ------- National Advisory Council for Environmental Policy and Technology Reinvention Criteria Committee September 1999 Key Recommendations and Conclusions: • In order to evaluate the progress and success of reinvention, EPA must clearly define its purpose, goals, and objectives. • There are actually multiple goals for reinvention. EPA should make each goal explicit. • To succeed, the Agency's reinvention efforts must: Maintain public confidence Involve stakeholders j Recognize that EPA is a partner hi environmental protection Serve as a catalyst for change in collaboration with other stakeholders. • EPA should address the question of what as needed to meet the environmental challenges of the future. • It is inevitable that cross-program comparisons will be made. To minimize "bureaucratic Darwinism," EPA should establish a more explicit process for deciding which programs are worthy and feasible to pursue. • Reinvention must eventually move from its current pilot/laboratory phase to systemic change. • There are opportunities for consolidation and coordination of individual reinvention programs. In order to pursue them, EPA should group similar types of reinvention programs for evaluation. This will help to clarify the commonalties and differences. It will also help the Agency to extract the "lessons learned." • Barriers to the adoption of positive innovations 'include uniformity myths and risk aversion. Both should be recognized by EPA, and strategies should be developed to overcome them. • Just as reinvention is an evolutionary process, the function of the Office of Reinvention should evolve over time. i The Committee's preliminary recommendations were presented to the NACEPT Council at the November 5-6, 1997 plenary meeting. The preh'minary recommendations were also documented in the March 1998 interim report. It should be noted that this Committee has continued to address issues affecting the Agency's reinvention activities. However, in April 1998 changes were made to the membership because several members either changed sector affiliation or their terms expired. The Committee was 11 ------- National Advisory Council for Environmental Policy and Technology Reinvention Criteria Committee September 1999 reconstituted with several new members and focused on a revised charge affecting the Agency's reinvention efforts. The reconstituted Committee has completed its work in response to the new charge and will submit findings and recommendations to the NACEPT Council for review and approval in a separate report. in ------- ------- National Advisory Council for Environmental Policy and Technology Reinvention Criteria Committee September 1999 TABLE OF CONTENTS EXECUTIVE SUMMARY I. INTRODUCTION II. APPROACH HI. RECOMMENDATIONS ATTACHMENT A - Schedule of Committee Meetings ATTACHMENT B -List of Committee Members ATTACHMENT C -Reinvention Programs Addressed by the Committee Page i 1 3 5 9 10 11 ------- ------- National Advisory Council for Environmental Policy and Technology Reinvention Criteria Committee . September 1999 Identification of Evaluation Criteria For EPA's Reinvention Programs Final Report and Recommendations of The Reinvention Criteria Committee I. INTRODUCTION The National Advisory Council for Environmental Policy and Technology (NACEPT) is a public advisory committee originally chartered on July 7, 1988. The Council provides recommendations and advice to the Administrator and other EPA officials on ways to improve the development and implementation of domestic and international environmental management policies and programs. The NACEPT membership includes senior-level officials and experts representing federal, state, and local government agencies and tribal organizations, business/industry, academia, environmental organizations, and NGOs. As principal constituents and stakeholders of EPA, these members provide advice and recommendations on policy issues/questions and serve as a sounding board for new strategies that the Agency is developing. In March 1995, President Clinton, Vice-President Gore, and EPA Administrator Carol Browner announced an ambitious agenda to reinvent environmental protection as part of the larger goal of creating a federal government that works better and costs less. The agenda, which evolved and broadened over time, cut across all areas of EPA's regulatory responsibility. It focused on achieving a safer, cleaner environment for the public by correcting the everyday inefficiencies and obstacles that limited the effectiveness of environmental programs, and by designing and testing fundamentally new approaches and systems equal to current and future environmental and public health challenges. EPA's reinvention agenda included 25 high-priority projects, which cut across all areas of environmental regulatory responsibility. They were designed to promote innovation and flexibility, increase community participation and partnerships, improve compliance with environmental laws, and cut red tape and paperwork. K EPA also took steps to bolster reinvention internally by announcing a new Office of Reinvention led by an Associate Administrator. Senior Agency officials were appointed to serve as "Reinvention Ombudsmen" for their respective program areas, based on the recognition that reinvention is most successful when senior managers assume an active role. In April 1996, the Deputy Administrator requested that NACEPT help the Agency identify criteria it could use to evaluate the progress and success of specific reinvention programs. The ' U.S. EPA, Office of the Administrator, "Managing for Better Environmental Results, A Two-Year Anniversary Report on Reinventing Environmental Protection," EPA 100-R-97-004, March 1997. 1 ------- National Advisory Council for Environmental Policy and Technology Reinvention Criteria Committee September 1999 Reinvention Criteria Committee was convened to address this charge (see Attachment B). The Committee did not evaluate EPA's reinvention efforts as the Agency was in the early stages of the reinvention process. Later that year, the Committee was asked to broaden its focus by identifying evaluation criteria for measuring the progress of reinvention activities in several of the more traditional programs and to provide advice on how to evaluate reinvention as a whole. This report is being submitted at this time to serve as a record of the Committee's "real-time" advice and recommendations on the identification of evaluation criteria. The Committee's preliminary recommendations were presented to the NACEPT Council at the November 5-6, 1997 plenary meeting. The interim report documenting the preliminary recommendations was prepared in March 1998. It should be noted that this Committee has continued to address issues affecting the Agency's reinvention activities. Changes were made to the membership due to affiliation changes and expiration of terms. The Committee was reconstituted in April 1998 with several new members and focused on a revised charge affecting the Agency's reinvention efforts. The Committee has completed its work in response to the new charge and will submit recommendations to the NACEPT Council for review and approval in a separate report. ------- National Advisory Council for Environmental Policy and Technology Reinvention Criteria Committee September 1999 II. APPROACH ' The Committee began by reviewing and dialoging with the managers of specific reinvention programs. The Committee concluded that each reinvention program needed to have a discrete set of goals and objectives, which were often lacking. However, several reinvention programs reflected that careful thought had been given to identifying evaluation measures. The Committee dialogued with the reinvention program managers and provided "real time" feedback, as the reinvention programs were evolving. The members felt that it was important to focus on whether the goals and objectives of the reinvention programs had been clearly articulated within the scope of reinvention overall. As the discussions progressed, the Committee shifted its focus to identifying the steps required to develop an evaluation strategy. Several overarching issues were identified and communicated in letters to the Deputy Administrator on October 1996 and April 1997. The Agency responded in letters dated December 13,1996 and May 12,1997, respectively. The Deputy Administrator acknowledged that the preliminary findings were consistent with the Agency's ideas about evaluation criteria to be considered. He urged the committee to continue with its efforts and requested that the focus be expanded to include: • Examining the reinvention activities in the more traditional media offices and providing insight into how their progress can be measured. • Recommending criteria for measuring the success of reinvention as a whole. In focusing on these broader issues, the committee framed its deliberations with the following questions: • What evaluation criteria would be appropriate for reinvention as a whole? • Is it possible/useful to employ cross-program comparisons? • Are there opportunities for using "lessons learned" to move to systemic change? • • Is there consistency among the reinvention themes? Are there opportunities for consolidation/coordination? • What are the barriers to reinvention and what factors lead to the adoption of positive innovations? ------- National Advisory Council for Environmental Policy and Technology Reinvention Criteria Committee September 1999 As the Committee began to define these overarching issues, the need to dialog with the regional reinvention managers and other senior level decision makers was recognized. The Committee also recognized the need to consider how these issues were disseminated into the Agency's overall mission. Senior managers were invited to the meetings to share their perspectives on the overarching issues and barriers to reinvention. Section m. of this report contains the Committee's final recommendations on an evaluation framework, and other key recommendations and conclusions. ------- National Advisory Council for Environmental Policy and Technology Reinvention Criteria Committee September 1999 III. RECOMMENDATIONS The Committee began with the premise of reinvention-that it is important to find more efficient ways of conducting the business of environmental protection, or as it has succinctly been summarized, make the Agency's environmental protection efforts "cleaner, cheaper, and smarter." The Committee added to that succinct summary, that "maintenance of public confidence" should also be an explicit goal. Within this framework, the committee dialogued with the reinvention program managers and several regional reinvention ombudsmen on ways that the programs could be evaluated. This real-time feedback allowed for pertinent suggestions to be incorporated as the programs were evolving. A total of eighteen reinvention programs were reviewed by the Committee (see attachment C). Evaluation Framework: • Each reinvention program should have a clear and succinct statement of its particular goals and objectives. Several programs defined criteria for use in implementing their efforts. Examples included criteria for determining which individual projects or facilities will be part of the program, or criteria for evaluating individual grant proposals. However, the overall program goals and objectives were often not clear or explicit. Within this framework, the Committee recommends adding environmental improvements (i.e., environmental results) as an explicit goal of each reinvention program. The Committee also noted that several programs appeared to focus primarily on improving the process. • Based upon those identified goals and objectives, EPA should develop a specific evaluation strategy for each program. The Committee believes that it is not feasible to develop meaningful evaluation criteria for the programs without first articulating the goals the programs are designed to achieve. • The evaluation strategies should include a hierarchy of integrated measures that range from short-term to long-term components. There is a continuum of the types of measures/that should be examined, ranging from activity and management elements in the short-term, to longer term measures that focus on environmental outcomes. While it may not be feasible in all of the programs to define actual measures in environmental media, EPA can or should develop intermediate measures that relate to specific environmental outcomes (such as emissions reductions or pollution prevention measures). The logical integration of these measures may enable the program activity measures to ultimately focus on improved environmental outcomes. The activity or operation measures would also include indicators of whether the "cheaper and smarter" goals are being met. ------- National Advisory Council for Environmental Policy and Technology Reinvention Criteria Committee ; September 1999 • The evaluation strategies should also include measures of environmental quality improvement, stakeholder involvement and satisfaction, and maintenance of publlic confidence. Most of the reinvention programs explicitly stated the desire for greater stakeholder involvement. However, some of the programs had not considered measures of stakeholder involvement or satisfaction as components of their evaluation strategy. Such measures can be either direct or indirect. The direct measures, such as simply asking for stakeholders' opinions about the program, appeared to be to most frequently overlooked, although most obvious. • In order for reinvention to be successful, it should involve a more pervasive organizational culture change within the Agency, beyond the specific reinvention programs. EPA should ensure that the lessons learned from the implementation of the programs are diffused into general practice. Evidence of implementation of this could include incentives, such as increased concentration of Agency resources in areas with promising evaluation results, with simultaneous disincentives such as decreased (or eliminated) allocation of Agency resources in areas with inferior evaluation results. An Agency review of factors leading to the adoption of positive innovations may be useful. At the request of the Agency, the committee expanded its focus to include: j: • Examining the reinvention activities in the more traditional media offices and providing insight into how their progress can be measured. • Recommending criteria for measuring the success of reinvention as a whole. Key Recommendations and Conclusions: • In order to evaluate the progress and success of reinvention, EPA must clearly define its purpose, goals, and objectives. The sound-bite "cleaner, cheaper, smarter" is not, In itself, sufficient. It only asks whether the environment is better relative to today's baseline. The theme appears agnostic on the question of how much "better" is enough. That is, the baseline is known but not the goals or milestones/benchmarks, even in qualitative terms. • The Committee believes that there are actually multiple goals for reinvention. EPA should make each goal explicit For example, reinvention is seeking and testing both: a) ways to make the current regulatory system work better, and b) new alternative approaches to the regulatory system. ------- National Advisory Council for Environmental Policy and Technology Reinvention Criteria Committee September 1999 To succeed, the Agency's reinvention efforts must: — Maintain public'confidence ' ~ ~ Involve stakeholders — Recognize that EPA is a partner in environmental protection — Serve as a catalyst for change in collaboration with other stakeholders. EPA should address the question of what is. needed to meet the environmental challenges of the future. This will help the Agency to define the purpose of reinvention. • It is inevitable that cross-program comparisons will be made. To minimize "bureaucratic Darwinism," EPA should establish a more explicit process.for deciding which programs are worthy and feasible to pursue. This approach should involve hypothesis testing and measures of success, including at a minimum, qualitative estimates of benefits and costs. Reinvention must eventually move from its current pilot/laboratory phase to systemic change. To do so, EPA must explore options for how successful "experimental" reinvention programs can be transitioned and more broadly incorporated into the Agency's environmental protection mission. There are opportunities for consolidation and coordination of individual reinvention programs. In order to pursue them, EPA should group similar types of reinvention programs for evaluation. This will help to clarify the commonalties and differences. It will also help the Agency to extract the "lessons learned." Barriers to the adoption of positive innovations include uniformity myths and risk aversion. Both should be recognized by EPA, and strategies should be developed to overcome them. There is a tendency for people to regard others as having uniform characteristics. Such views may serve as barriers to effective adoption of environmental and/or regulatory innovations. Uniformity myths block effective communication and effective relationship building. EPA's reinvention effort is short-sighted if it does not conduct a systemic assessment of such barriers and collaborate with its constituents. Just as reinvention is an evolutionary process, the function of the Office of Reinvention should evolve over time. At this juncture, this Office should: — Promote better communication throughout the Agency regarding the reinvention programs and philosophy. — Coordinate and integrate the reinvention efforts. ------- National Advisory Council for Environmental Policy and Technology Reinvention Criteria Committee September 1999 — Serve as evaluator and diagnostician (through statutes, regulations, etc.) of the "lessons learned." -- Develop a plan for applying the "lessons learned" from the individual reinvention programs into broader implementation by program offices. - Focus on using the "lessons learned" to change the Agency's culture and system for environmental protection. ! — Provide training across the Agency. — Advocate and promote reinvention. The Committee commends EPA for undertaking the reinvention effort, and believes that it is a productive approach for the Agency and its stakeholders and for environmental protection in the future. EPA is also to be commended for recognizing the importance of evaluating its efforts and attempting to develop criteria for evaluation early hi the process. Evaluation criteria must be defined early in program development to provide interim feedback on program direction (or needed redirection) as well as for later measures of progress or success in the programs. ------- National Advisory Council for Environmental Policy and Technology Reinvention Criteria Committee September 1999 ATTACHMENT A SCHEDULE OF COMMITTEE MEETINGS Dates & Location April 17-18,1996 - Arlington, Virginia July 24 -25, 1996 - Washington, DC September 10-11,1996 - Washington, DC " December 11 -12, 1996 - Washington, DC April 2-3, 1997 - . Alexandria, Virginia July 16 -17, 1997 ' - Washington, DC September 29, 1997 - Alexandria, Virginia November 5 - 6, 1997 - Arlington, Virginia ------- ATTACHMENT B LIST OF COMMITTEE MEMBERS Chair: Ms. Holly Stoerker Executive Director Upper Mississippi River Basin Association St. Paul, Minnesota Designated Federal Officer: Gwendolyn C.L. Whitt Office of Cooperative Environmental Management U.S. Environmental Protection Agency Washington, B.C. Dr. Edwin H. (Toby) Clark II President, Clean Sites, Inc. Alexandria, Virginia Ms. Catharine M. DeLacy Corporate Vice President Health, Environment, and Safety AlliedSignal, Inc. Morristown, New Jersey Ms. Wilma Delaney Vice President Environmental and Regulatory Issues Dow Chemical Company Washington, DC Mr. Nicholas A. DiPasquale Director Division of Air and Waste Management Department of Natural Resources and Environmental Control Dover, Delaware Mr. Gerald Digerness National Association of Conservation Districts Sumas, Washington Mr. Edward Garvey Chair, State of Minnesota Public Utilities Commission St. Paul, Minnesota Dr. George Hallberg Associate Director, Hygenic Laboratory University of Iowa Iowa City, Iowa Dr. Walter Handy, Jr. Assistant Health Commissioner Cincinnati Department of Health Cincinnati, Ohio Ms. Elise Hoerath Counsel, Environmental Quality Division National Wildlife Federation Washington, DC Mr. Kevin Mills Sr. Attorney/Director Pollution Prevention Alliance Environmental Defense Fund Washington, DC Dr. Joseph Sullivan 49 Bittersweet Trail Wilton, Connecticut ------- National Advisory Council for Environmental Policy and Technology Reinvention Criteria Committee September 1999 ATTACHMENT C REINVENTION PROGRAMS ADDRESSED BY THE COMMITTEE Common Sense Initiative (CSI) Self-Policing/Audit Policy Self Certification Sustainable Development Challenge Grants National Environmental Performance Partnership System Project XL ' Environmental Leadership Program National Environmental Goals Project Small Business Compliance Assistance Centers Brownfields Program Consolidated Federal Air Rule Sustainable Industries Project One Stop Reporting Sector Facility Indexing Open Market Air Trading Pollution Prevention Voluntary Programs Electronic Public Access Effluent Trading in Watersheds 11 ------- ------- |