United States EPA Science Advisory EPA-SAB-RSAC-02-003 Environmental Board (1400A) December 2001 Protection Agency Washington, DC iviviv.epa.gov/sab &EPA WATER QUALITY AND POLLUTION PREVENTION MULTIYEAR PLANS: AN SAB REVIEW A REVIEW BY THE RESEARCH STRATEGIES ADVISORY COMMITTEE (RSAC) OF THE EPA SCIENCE ADVISORY BOARD (SAB) ------- I UNITED STATES ENVIRONMENTAL PROTECTION AGENCY / WASHINGTON, D.C. 20460 December 14, 2001 OFFICE OF THE ADMINISTRATOR SCIENCE ADVISORY BOARD EPA-SAB-RSAC-02-003 Honorable Christine Todd Whitman Administrator U.S. Environmental Protection Agency 1200 Pennsylvania Avenue, NW Washington, DC 20460 Subject: Review of the U.S. EPA Office of Research and Development's Water Quality and Pollution Prevention Multiyear Plans: An SAB Report Dear Governor Whitman: On October 16 and 17, 2001 the Research Strategies Advisory Committee (RSAC) of the EPA Science Advisory Board (SAB) met to review the Water Quality and Pollution Prevention Multiyear Plans for the Office of Research and Development in the U.S. Environmental Protection Agency. Multiyear Plans (MYPs) provide a basis and context for developing annual plans. Examination of how annual plans evolve from year to year serves to document the progress that the Office of Research and Development (ORD) has made towards meeting its strategic goals to help the Agency achieve appropriate environmental outcomes. The linkage between the research strategies, prior year outputs and outcomes, and the research plans should clearly show how ORD conducts and integrates its research program to achieve its long-term goals. The objective of this review was to: a) evaluate available illustrative MYPs from the standpoint of ORD"s science planning strategy; b) to understand the extent to which major issues were being addressed, particularly in the context of the overall EPA science and research strategy; c) to identify how annual performance goals were intended to relate to measures of performance; and d) more importantly, to identify topics, themes, emerging issues, and "lessons learned" that could help increase the understanding and usefulness of the other MYPs that are being and will be developed. The review was structured around responses to six specific questions which provided the focus for our evaluation. However, such a focus leads the committee to identify gaps, unclear linkages, and lack of context even though the review is necessarily based on an inadequate understanding of background efforts. As a result, responses tend to focus more on the negatives than the positives with the possible impression that RSAC found the two MYPs to be quite inadequate. On the contrary, RSAC found great value in these MYPs. It recognizes the long and arduous efforts that have been spent in preparing them, and considers them to be a sound and ------- essential part of both EPA and ORD research and budget planning. RSAC strongly encourages ORD to finalize and implement these two MYPs and the remaining fourteen MYPs. RSAC's review of these illustrative MYPs revealed a number of items and points that should be considered as the Pollution Prevention and Water Quality MYPs are completed and as the other MYPs are written and finalized, including: a) Contextual information about how the particular multiyear strategy fits in the broader ORD strategy and complements the other multiyear plans. b) Information about the particular value added by EPA's efforts in each of the broader research areas. The plans should clearly indicate why EPA's research strategy adds value to scientific and policy efforts. c) Discussion of the value of the research strategies in the context of complementary efforts conducted outside the Agency. d) Identification of the areas of research for which they were designed, i.e., core research, problem-driven research or both. For MYPs that include both areas, there should be an explicit formulation of which components of the plan address the regulatory needs of the program offices, and which address the core research needs. e) Information in each MYP that indicates how the efforts and deliverables are to be accomplished including consideration of the specific measurement issues and scientific and engineering advances identified in the strategy. From a strategic view, a MYP should consider whether the implementation process (e.g., in house staff vs. grant, contracts, etc.) substantially affects the attainment of deliverables and outcomes. f) A clear link between completion of the annual performance measures (i.e., outputs) and the achievement of the annual performance goals (i.e., outcomes), for in the absence of such a link it is not clear how the annual goals logically meet the long- term goals. g) Definition of outcomes that reflect concrete and measurable milestones in the path towards achieving the MYPs long-term goals (i.e., clean air, water and soil). The potential for success in reaching the long-term goals cannot be evaluated without the formulation of concrete outcomes and the time frame for their achievement. The Committee also recommends that the annual performance measures should be focused on desired outcomes, not outputs. The Committee found that the collection of annual measures in the plans reviewed were largely outputs. It was difficult to understand how the collection of outputs would eventually combine and contribute to achieving outcomes that ------- further annual performance goals or long-term goals. Developing outcome measures is not an easy task, but it should be pursued. ORD planners may conclude that articulation of the relationship between outputs and outcomes is itself an important research question. We hope that these comments are helpful to the Office of Research and Development as it finalizes it's first round of Multiyear Plans, and to the Agency, as it continues to refine its planning processes to develop means to better plan for obtaining the science needed by EPA and how to use it to best inform decision-making at the Agency. We look forward to your response to our comments. Sincerely, /S/ /S/ Dr. William H. Glaze, Chair Dr. Raymond C. Loehr, Chair EPA Science Advisory Board Research Strategies Advisory Committee EPA Science Advisory Board ------- NOTICE This report has been written as part of the activities of the EPA Science Advisory Board, a public advisory group providing extramural scientific information and advice to the Administrator and other officials of the Environmental Protection Agency. The Board is structured to provide balanced, expert assessment of scientific matters related to problems facing the Agency. This report has not been reviewed for approval by the Agency and, hence, the contents of this report do not necessarily represent the views and policies of the Environmental Protection Agency, nor of other agencies in the Executive Branch of the Federal government, nor does mention of trade names or commercial products constitute a recommendation for use. Distribution and Availability: This EPA Science Advisory Board report is provided to the EPA Administrator, senior Agency management, appropriate program staff, interested members of the public, and is posted on the SAB website (www.epa.gov/sab). Information on its availability is also provided in the SAB's monthly newsletter (Happenings at the Science Advisory Board). Additional copies and further information are available from the SAB Staff [US EPA Science Advisory Board (1400A), 1200 Pennsylvania Avenue, NW, Washington, DC 20460-0001; 202- 564-4533]. ------- ABSTRACT The Research Strategies Advisory Committee (RSAC) of the EPA Science Advisory Board (SAB) met October 16 and 17, 2001 to review the Water Quality and Pollution Prevention Multiyear Plans of the Office of Research and Development. The objective was to: a) evaluate available illustrative MYPs from a strategic standpoint; b) to understand the extent to which major issues were being addressed, particularly in the context of the overall EPA science and research strategy; c) to identify how annual performance goals were intended to relate to measures of performance; and d) more importantly, to identify topics, themes, emerging issues, and "lessons learned" that could help increase the understanding and usefulness of the MYPs that are being and will be developed. The RSAC review of these two illustrative MYPs revealed a number of items and points that should be considered as these MYPs are completed and as the other MYPs are written and finalized. These included contextual information about how the particular multiyear strategy fits in the broader ORD strategy and complements the other multiyear plans, a discussion of specific measurement issues and advances that will be addressed by the strategy, information about the particular advantages of EPA's efforts in each of the broader research areas, and the relationship of the activities to be accomplished to complementary efforts conducted outside of the Agency. The Committee also felt that MYPs should explicitly address the areas of research for which they were designed and that information be included in each MYP that indicates how the efforts and deliverables are to be accomplished (e.g., by grants, contracts, in-house, etc.). Similarly, each plan would benefit from a careful consideration and clear discussion of how the long-term goals can more likely be met by addressing the most important areas of scientific uncertainty. Planners are encouraged to develop long-term goals that are not open-ended, because annual goals cannot logically meet long-term goals where desired outcomes are not clearly articulated. Overall, RSAC found great value in these MYPs, recognizes the thoughtful and dedicated efforts that have been spent in preparing them, and considers them to be a sound and essential part of both EPA and ORD research and budget planning. RSAC strongly encourages ORD to consider seriously the comments in this report and to use them to finalize and implement these two MYPs and the remaining MYPs. Keywords: Multiyear plan, GPRA, budget, research, strategic planning 11 ------- US ENVIRONMENTAL PROTECTION AGENCY EPA SCIENCE ADVISORY BOARD RESEARCH STRATEGIES ADVISORY COMMITTEE (RSAC) CHAIR Dr. Raymond C. Loehr, Professor, University of Texas at Austin, Department of Civil Engineering, Austin, TX MEMBERS Dr. Steven Bartell, Principal, Cadmus Group, Oak Ridge, TN Dr. Richard J. Bull, President, MoBull Consulting, Kennewick, WA Dr. Robin Cantor, Principal and Managing Director, LECG, LLC, Washington, DC Dr. Maria Morandi, Associate Professor of Environmental Science, University of Texas Health Science Center at Houston, School of Public Health, Houston, TX Dr. William Smith, Professor Emeritus of Forest Biology, School of Forestry and Environmental Studies, Yale University New Haven, CT Dr. James E. Watson, Professor, Department of Environmental Sciences and Engineering , University of North Carolina at Chapel Hill, Chapel Hill, NC Liaison Member Dr. Jerald Schnoor, Professor, Department of Civil and Environmental Engineering, University of Iowa, Iowa City, IA. Also a member of the Board of Scientific Counselors. Dr. Thomas Theis, Bayard D. Clarkson Distinguished Professor, Director of the Center for Environmental Management, Clarkson University, Potsdam, NY. Also a member of the Environmental Engineering Committee. DESIGNATED FEDERAL OFFICER Dr. John R. Fowle, III, Designated Federal Officer, US Environmental Protection Agency, EPA Science Advisory Board (1400A), 1200 Pennsylvania Avenue, NW, Washington, DC 20460 Ms. Wanda R. Fields, Management Assistant, US Environmental Protection Agency, EPA Science Advisory Board (1400A), 1200 Pennsylvania, Ave, NW, Washington, DC 20460 in ------- TABLE OF CONTENTS 1. INTRODUCTION 2 1.1 Background 2 1.2 Charge to the Committee 4 1.3 Format of this Report 4 1.4 Process for the Review 4 2. RESPONSE TO THE CHARGE FOR THE WATER QUALITY MYP 6 2.1 Does the multiyear plan convey the Office of Research and Development's strategic research plans for the subject area in an understandable fashion and at an appropriate level of detail? 6 2.2 Do the long-term goals and underlying science questions identified in the plan address the most important areas of scientific uncertainty in the subject area? 7 2.3 Does the proposed scope of work proposed by the Office of Research and Development complement research by others? 8 2.4 Would accomplishing the annual performance goals allow the Office of Research and Development to both achieve the long-term goals and answer the science questions identified in the plan? 8 2.5 Would accomplishing the annual performance measures clearly demonstrate that the associated annual performance goal was attained? 9 2.6 Does the plan clearly describe the outcomes that the Office of Research and Development will achieve through the proposed research? 10 3. RESPONSE TO THE CHARGE FOR THE POLLUTION PREVENTION MYP 11 3.1 Does the multiyear plan convey the Office of Research and Development's strategic research plans for the subject area in an understandable fashion and at an appropriate level of detail? 11 3.2 Do the long-term goals and underlying science questions identified in the plan address the most important areas of scientific uncertainty in the subject area? .... 11 3.3 Does the proposed scope of work proposed by the Office of Research and Development complement research by others? 12 3.4 Would accomplishing the annual performance goals allow the Office of Research and Development to both achieve the long-term goals and answer the science questions identified in the plan? 14 3.5 Would accomplishing the annual performance measures clearly demonstrate that the associated annual performance goal was attained? 15 3.6 Does the plan clearly describe the outcomes that the Office of Research and Development will achieve through the proposed research? 15 4. COMMENTS ON THE MYP PROCESS 17 4.1 Overview 17 4.2 Recommendations 17 APPENDIX A. ACRONYMNS A-l APPENDIXB. ORD MULTI-YEAR PLANNING GUIDANCE B-l iv ------- 1. INTRODUCTION 1.1 Background Effective environmental protection requires extraordinary integration of research efforts. Not only must the research be interdisciplinary, it has to be sensitive to the formulation of policy decisions that are a central role of the EPA and provide knowledge that informs the Agency's decisions. EPA's Office of Research and Development (ORD) developed Multiyear Plans (MYPs) as a tool to plan the direction of their research program and to communicate these plans both within ORD and to others. The MYPs are intended to provide a link between the Agency's and ORD's strategic plans and the ORD annual plans that serve as the basis for ORD's budget request. (See Appendix B, "ORD Multiyear Planning Guidance".) There are several key planning documents that guide ORD's research activities. The EPA and ORD Strategic Plans set the direction for ORD and highlight major areas of emphasis. They provide for both the Agency's Government Performance and Results Act (GPRA) goals and ORD's interpretation of the science needed to support the achievement of these goals. These plans, in conjunction with Agency Science Plans (e.g., the Contaminated Sediment Science Plan currently under development) when available, inform individual topic research strategies and plans that translate the overall direction of the science program into more specific strategic approaches, priorities and outcomes. Research strategies describe the approach ORD will take to address the scientific questions facing EPA, and research plans focus on where a laboratory or center can best use its resources to develop science to inform decision-making. MYPs provide a basis for creating annual plans and the context to understand how decisions made in annual planning impact the ability of ORD to meet future goals and outcomes by linking the research strategies and research plans to show how ORD conducts research in an integrated fashion to achieve long-term goals. The MYP process also improves ORD's ability to understand and document the impact of Agency priorities and budget guidance on its research program. Further, the MYP process allows for a more comprehensive understanding of the changes needed to emphasize a new research direction or accelerate an existing research effort. ORD has briefed the Research Strategies Advisory Committee (RSAC) several times over the past two years about their intent to develop 16 MYPs, and to keep the Committee appraised of the progress made in developing them. RSAC is a Standing Committee of the EPA Science Advisory Board (SAB) of the United States Environmental Protection Agency (USEPA). It was convened to provide broad advice to the Administrator on research planning, management, and budget development for use by the Agency in its long-term budget planning process. RSAC provides a point of focus for the Board to consider the overall directions of the intramural and extramural research programs of EPA. ORD requested that RSAC review the first two MYPs for their utility in budget planning because of the committee's mission, which includes an annual review of the President's budget 1 ------- request for the Science and Technology component of EPA's budget. Clearly, other SAB committees as well as advisory committees outside the SAB have an interest in specific MYPs and will likely provide review on additional MYPs. Because ORD's resources are roughly equally divided between core research and problem-driven research RSAC wished to review both a core and a problem-driven multiyear plan in its evaluation of the Multiyear Planning Process. The Pollution Prevention (core) and Water Quality (problem-driven) MYPs were the first two such MYPs available. Thus, they were selected for review. In its review of these two MYPs RSAC: a) considered them to be illustrative of the entire 16 being prepared. b) attempted to review the MYPs from a strategic overview perspective (i.e. "30,000 foot view") rather than a detailed operational evaluation. c) attempted to identify issues and items that could be helpful in the development and understanding of all 16 MYPs. d) attempted to understand the extent to which major issues of concern were being addressed. e) tried to determine how a MYP built upon knowledge developed by prior research at EPA and by programs and efforts underway at other organizations and agencies. f) tried to identify and understand the "value" of research (i.e., outputs) to existing environmental needs and the nation, that is to say, why is this research necessary and, if it is, why should EPA undertake the proposed research rather than other entities? g) attempted to understand the context for the two MYPs in terms of the overall EPA science and research strategy. h) attempted to identify the measures of performance that were being considered to evaluate the effectiveness and value of the activities included in the MYP, and to consider the appropriateness of such measures to identify environmental outcomes (i.e., clean air, water and soil). i) tried to identify "lessons learned" from the Multiyear Planning Process in this review that could be used by the Agency to increase the understanding and usefulness of the other MYPs that are being and will be developed. Committee members agreed that what is presented in the two MYP's should be understood within the framework of the broader planning activities at EPA and RSAC's review ------- processes. Therefore, many of the comments and suggestions reflect a broader concern about science planning in EPA. This review is based upon documents received prior to the meeting held October 16- 17,2001; however, some issues were clarified or expanded during the presentations by EPA. RSAC members also requested additional information during a telephone conference held on October 11, 2001 in preparation for the meeting of October 16. 1.2 Charge to the Committee EPA's charge to the committee was: a) Does the multiyear plan convey the Office of Research and Development's strategic research plans for the subject area in an understandable fashion and at an appropriate level of detail? b) Do the long-term goals and underlying science questions identified in the plan address the most important areas of scientific uncertainty in the subject area? c) Does the proposed scope of work proposed by the Office of Research and Development complement research by others? d) Would accomplishing the annual performance goals allow the Office of Research and Development to both achieve the long-term goals and answer the science questions identified in the plan? e) Would accomplishing the annual performance measures clearly demonstrate that the associated annual performance goal was attained? f) Does the plan clearly describe the outcomes that the Office of Research and Development will achieve through the proposed research? 1.3 Format of this Report Following this Introduction, the report provides specific responses to the questions in the Charge to the Committee for the Water Quality MYP (Chapter 2) and the Pollution Prevention MYP (Chapter 3). General comments applicable to both MYPs and others under development overall are found in Chapter 4. 1.4 Process for the Review Presentations concerning the MYPs were made by ORD to the Committee on October 16, 2001. The RSAC had an active discussion of the above plans, and other relevant issues were discussed with Agency personnel during the presentations and during discussions the following ------- day. Readers of this report should recognize that the following evaluation of the two MYPs and this report resulted from: a) detailed reading of the two draft MYPs prior to the meeting on October 16 b) interactive discussion following the presentations on October 16, and c) internal discussion by RSAC members immediately following the meeting and during the several intervening weeks prior to releasing this report, including a teleconference held November 8, 2001 to help finalize the report. ------- 2. RESPONSE TO THE CHARGE FOR THE WATER QUALITY MYP 2.1 Does the multiyear plan convey the Office of Research and Development's strategic research plans for the subject area in an understandable fashion and at an appropriate level of detail? The Water Quality Multiyear Plan (MYP) was presented as an example of an important research area that is structured entirely in the context of activities of the program office responsible for administering the provisions of the Clean Water Act. The presentation clarified that this plan included entirely problem-driven research only. The context of this research plan is largely captured in Figure 1 of the MYP that illustrates the steps required to meet the water quality goals which are tightly coupled to regulatory activities within the Office of Water under the Clean Water Act. In addition, Figure 2 of the MYP shows specific links to other multiyear plans. This was a very effective way of providing context for this MYP. The answer to this charge question is, in general, yes. However the presentation and subsequent discussion did indicate topics and issues for which greater exposition may be warranted. Such topics and issues are discussed in the following paragraphs. While this MYP was generally understood by the RSAC committee members who were familiar with the program, the committee was of the opinion that the plan should be readable as a stand-alone document. Thus, it would benefit by including more explicit descriptions of the regulatory constructs of the Clean Water Act. For example, the plan addresses designated uses for watersheds and how a body of water is designated as being impaired for specific uses, but it does not provide an overview of what the designated uses of a water body are or the process that is used for determining if a body of water is impaired. The RSAC suggests that the Water Quality MYP provide some additional descriptions of key concepts (such as designated uses and impaired uses) of the Clean Water Act to provide some perspective for those readers not directly knowledgeable of Clean Water Programs. The Water Quality MYP also makes reference to other MYPs that are of clear importance to the Agency's implementation of the Clean Water Act. Particular MYPs that are identified included the Ecosystem Protection MYP, which includes the Agency's Environmental Monitoring and Assessment Program (EMAP) activities. The Water Quality MYP is critical for developing the metrics necessary for the evaluation of the degree of success of the regulatory programs under the Clean Water Act. The Ecosystem Protection MYP is critical for identifying problems and causes of problems in the aquatic environment. The RSAC feels that the role of other related MYPs needs to be made more explicit in an overarching introduction to the MYPs because of their importance for problem definition and program evaluation. There were also some obvious omissions within the Water Quality MYP and its relationship to other MYPs. For instance, there was no mention of the Drinking Water MYP, nor was a clear attention given as to how the Water Quality MYP works in support of a program that protects drinking water as a significant use. Although elements of some projects related to ------- drinking water issues are included in the Water Quality MYP, there are obvious omissions, perhaps because the issue is being addressed in the Drinking Water MYP. This linkage is simply not discernable in the Water Quality MYP. This and similar connections need to be made explicit. Moreover, efforts should be made to make tighter connections between research in drinking water and water quality. Some of these issues can be quite subtle, but if properly developed could be synergistic across initiatives directed to health and ecosystem protection. For example, it is probable that efforts to protect habitat would also have the effect of reducing disinfection by-products in drinking water by simply reducing the amounts of organic carbon in the water. A second, more particular issue, is that some wastewaters apparently contain traces of secondary amines or secondary amine precursors that give rise to the formation of very potent carcinogens, such as nitrosamines, when drinking water is chlorinated. In the few systems that have been examined, these disinfection by-products may represent a major portion of the health risk attributable to the chlorination of drinking water. Therefore, in order to protect drinking water uses, parameters related to relatively innocuous compounds (i.e. traces of secondary amines) might need to be controlled under the Clean Water Act. 2.2 Do the long-term goals and underlying science questions identified in the plan address the most important areas of scientific uncertainty in the subject area? The Committee was not able to answer this question in the affirmative. The development of the long-term goals is an important construct for laying out the Water Quality MYP. However, the RSAC found the long-term goals to be stated in such broad and non-specific terms that it was virtually impossible to determine if and when a goal would be achieved. ORD should take the initiative to describe more explicitly the body of research that satisfies each stated goal as part of the framework for the MYP. The science questions presented in the MYP generally reflected the priorities identified in the plan and addressed the most important science issues related to the regulatory agenda of the Office of Water (OW). ORD's identification of OW's priorities in the MYP was an effective way of transit!oning to the more specific elements of the MYP that can be linked to individual projects. RSAC understands that the priorities assigned to the research questions and the resulting projects reflected a prioritization process that encapsulated simultaneous consideration of the potential impact of the research results and the scientific uncertainties. Neither the plan nor the presentation of the information in the MYP allowed RSAC to distinguish between the relative importance of these two parameters in the prioritization effort. The discussion of science questions and issues does not address specifically whether they reflected the greatest uncertainty or the greatest impact. It was acknowledged in discussion with the Office of Water staff who attended the RSAC meeting that some states have not discharged their responsibilities of designating uses for particular water bodies or issuing indications of impaired use and that this might be a high impact/high uncertainty science issue. If so, OW should direct some effort to helping ORD understand why some states do not designate uses or impairments of use (i.e., is it due to resource limitations, lack of appropriate guidance, or to other reasons?) as a means to better target the research to client needs. ------- The RSAC was somewhat confused by the fact that the discussion related to establishing cause and effect relationships cited under Long-term Goal 1 did not include discussions of uncertainty. The concern arose relative to the use of biological measures of impairment without rigorous attention to cause and effect issues. In discussions following the presentations it became clear that the term "uncertainty" was being used broadly and without much specificity. The RSAC suggests that future iterations of the plan should clarify some of the ambiguities of the language used in the phrasing of science questions. Some members of the RSAC were of the opinion that consideration of biological impairment will be very important for defining outcome measures for both the research and the regulatory programs. RSAC urges ORD to provide more explicit information related to these issues in subsequent drafts of the Water Quality MYP. 2.3 Does the proposed scope of work proposed by the Office of Research and Development complement research by others? The integration of ORD water quality research with federal, state, academic, and private laboratories starts with an awareness of non-EPA research. Some information related to ORD efforts to integrate non-EPA research in the framing of science questions was described during the oral presentations and some was detailed in the MYP. The Water Quality MYP indicates that attempts were made to review relevant non-EPA research by holding workshops and via several other strategies associated with specific science issues. However, the committee could not answer this charge question in the affirmative, because explicit details regarding how the EPA efforts described in the MYP "complement" the non-EPA research efforts were not provided. RSAC recommends that in subsequent drafts of this MYP ORD address in greater detail and specificity the extent to which the efforts described in the Water Quality MYP complement and do not duplicate past or on-going research by others. 2.4 Would accomplishing the annual performance goals allow the Office of Research and Development to both achieve the long-term goals and answer the science questions identified in the plan? Accomplishing the indicated annual performance goals described in the plan might allow the Office of Research and Development to make progress towards achieving the stated long- term goals and might provide some initial answers to the science questions identified in the plan. However, it should be recognized that the set of stated annual performance goals may not necessarily constitute the entire set of intermediate milestones needed to be met in order to answer the key scientific questions that need to be answered in order to achieve the long-term water quality goals of the MYP. Thus, the committee could not answer this question in the affirmative. The three long-term goals described in the Water Quality MYP are quite open-ended from the perspective of supplying the necessary science to achieve these goals. The Water Quality MYP does not provide the criteria that were used to identify all the necessary and sufficient annual performance goals that, if achieved, would provide definitive answers to the long-term goals. Therefore, it remains difficult to evaluate whether the accomplishment of the annual goals in sum will in fact provide answers to these three longer term performance-driven 7 ------- objectives. Completion of the annual goals provided in Tables 2-4 of the MYP does not necessarily "add up" to providing answers to the longer term goals. Each of the stated annual performance goals should be examined in relation to its technical feasibility, as well as its relevance and contribution towards realizing the long-term goals identified in the MYP. For example, the second question asked in relation to the first long- term goal addresses the "best way" to classify ecosystems, landscapes, and watersheds under the assumption that such a classification will lead to efficient and sound development of water quality criteria. However, there are no stated annual performance goals aimed at evaluating the implicit assumption that developing water quality criteria would in any way be facilitated even by the existence of the "best" ecological classification. Similarly, the questions in the MYP stated in support of the second long-term goal concerning the inverse problem ask "how can. ecosystems be classified", yet there are no performance goals for this problem. Several performance goals outlined in Tables 2 appear very specific and clearly refer to ongoing projects that appear motivated by Agency interests or by prior and on-going projects that are perhaps consistent with the multiyear plan (e.g., habitat alteration and mercury impacts on the Great Lakes loon, effects of habitat degradation and flow alterations on salmon in the Pacific Northwest). However, these goals do not seem to necessarily derive from the MYP. Similar examples can be identified in Tables 3 and 4. In summary, RSAC strongly suggests that the MYP make the linkage between the annual performance goals and the long-term water quality goals much clearer. 2.5 Would accomplishing the annual performance measures clearly demonstrate that the associated annual performance goal was attained? RSAC cannot provide an affirmative to this question. As discussed in relation to annual performance goals (Section 2.4), accomplishing the annual performance measures might allow ORD to make progress towards achieving the long-term performance goals. However, it must be emphasized that the list of annual performance measures provided in the Water Quality MYP does not necessarily constitute the entire set of performance measures that might be required to attain the annual performance goals. This MYP does not present or discuss criteria that will be used to determine if annual performance measures have been met or nor does it discuss how the completion of the annual performance measures contributes to meeting the annual performance goals. Many of the performance measures are based simply on providing some kind of a report. This may become important in evaluating the overall success of implementing the plan, because many of the performance measures appear to involve providing interpretations of supporting science or technical research that has already been largely developed and exists as part of the scientific knowledge base (i.e., peer reviewed technical literature). The existing knowledge base that might contribute substantially to meeting performance measures was not necessarily funded by the Agency or developed to address the three long-term goals stated in the water quality plan. Analogous to evaluating the annual performance goals, the annual performance measures should be individually assessed in terms of their relevance to the stated goals. Examination of the stated annual performance measures in relation to the performance goals suggests that some of the performance measures represent works in progress that have been incorporated as part of the strategic plan, but which do not necessarily derive from the plan. Additionally, the plan could be ------- strengthened by providing some information that describes how the various performance measures were scheduled. It would also be helpful if the MYP were described in relationship to the Quality Plan for these efforts. 2.6 Does the plan clearly describe the outcomes that the Office of Research and Development will achieve through the proposed research? No, although the Water Quality MYP does have sections entitled "Expected Impact and Outcomes" after the summary of each of the three goals and the related research questions. First, the document does not define clearly what is meant by impact (which is a relatively imprecise and general term) and outcome (a term that connotes the idea of measurement). Consequently, the outcomes are described rather briefly and in very general terms. It appears that the Agency has not engaged in the strategic thinking required to develop measures of the effectiveness of the research plan for meeting each of the three long-term goals. RSAC recognizes that there might be a range of outcomes over time and that defining each of them and the parameters to be used as indicators of success is not a trivial exercise. There is obviously a concern that by explicitly describing expected outcomes and related measures, a partial success could be perceived as a failure of the research plan in achieving the stated goals. However, the definition of expected outcomes is not only important as an effectiveness evaluation tool for the MYPs, but it is also helpful because specifically defined outcomes (i.e., clean air, water and soil) provide long-term focal points or targets for the research plan in a far better operational way than the goals themselves. Thus, as projects are developed or new findings are made that may change or refocus the direction of research, the specific outcomes provide permanent guiding posts that are directly linked to the long-term goals and that are concrete milestones. For example, the ultimate outcome of this MYP could be to have a fraction of, or even all, the watersheds in the country classified into a scientifically robust water designation use category by a predefined target year. This is a concretely defined outcome (as compared to the broad statement of the goals) that: 1) would require a certain rate of progress in the classification of watersheds over time (which could be linked to annual performance goals); and 2) would also probably require the Agency to have a hierarchy of science questions related to water use designation so that those questions that require longer term research start to be addressed early- on. A concretely defined outcome would also provide impetus for maximizing the efficiency of the research plan (i.e., what science questions are likely to have the most impact in helping increase the numbers of watersheds classified for use, and how precisely do we need to know the relevant drivers for watershed quality classification before a reasonably robust classification can be made?). In addition, concrete outcomes are very useful tools for involving stakeholders within and outside the Agency in the whole process of multiyear planning. ------- 3. RESPONSE TO THE CHARGE FOR THE POLLUTION PREVENTION MYP 3.1 Does the multiyear plan convey the Office of Research and Development's strategic research plans for the subject area in an understandable fashion and at an appropriate level of detail? The draft Multiyear Plan (MYP) for Pollution Prevention and New Technologies (P2NT) for Environmental Protection (EPA/600/R-DRAFT- August 2001) represents a considerable and positive effort to address important issues relative to this area of activity. The individuals who prepared this MYP have created a comprehensive, and readable, plan of action. However, this MYP is not a strategic plan for this technical area. It also does not appear to address all of the five priority setting criteria that are to drive choices in research emphasis (page 2 of the MYP). These criteria are noted as: 1) address high risk problems; 2) respond to needs of stakeholders; 3) fill important research gaps not being addressed by others; 4) leverage resources with other organizations; and 5) provide potentially effective research. As indicated by material in the MYP, criteria 2) and 5) appear to be met. However, there is little in the MYP that allows a reader to judge that high-risk problems are being addressed. RSAC is not suggesting that this MYP does not address high risk problems, but that there is little information in the MYP to support that it does. There also appears to be no information that indicates that criteria 3 and 4 have been addressed. The Committee found that the strategic value of the P2NT MYP should be clarified by further discussion of EPA's comparative advantage in certain research and technology transfer areas. The document must convey how the P2NT long-term goals, measures, and annual goals add value to on-going efforts in other agencies and the private sector. What is the EPA niche in this area? What factors or information support EPA's conclusion that these projects are "below the radar screen" of other parties capable of conducting the projects? The public policy objectives should be better articulated to convey the potential contribution of the plan. Notwithstanding the concern about justification made above, the "strategic research plans" presented in ORD's the P2NT MYP are well-described and understandable. However, as stated below, both the annual performance goals and the long-term goals are open-ended, except for Goal III (Environmental Technology Verification), and specific outcomes expected from the proposed research are not presented. Thus, it is not clear that these goals could be completely achieved and, as a result, the effectiveness of this MYP can not be evaluated. 3.2 Do the long-term goals and underlying science questions identified in the plan address the most important areas of scientific uncertainty in the subject area? No. An important question would seem to be whether the long-term goals and underlying science questions identified in the plan address the most important scientific issues, not just uncertainty, in the subject area. The matter of uncertainty, which is critical to many research 10 ------- areas, seems applicable only to a subset of items in this subject area. For instance, one of the critical issues for pollution prevention, and the sustainability paradigm generally, is the development of commensurable metrics (i.e., metrics that are comparable and meaningful across human and industrial activities). The science underlying P2 performance metrics is poorly developed, yet it is critical to allocating resources optimally. Investigations into such metrics are especially compatible with the long-term goal of the P2 research program, and in particular those Annual Performance Measures (APM's) associated with the development of Life Cycle Assessment (LCA) evaluative techniques. Also, since LCA analyses are based on appropriate data, a portion of the LCA, known as Life Cycle Inventory or LCI, it is important that the agency take steps to ensure that the quality and availability of databases be peer-reviewed, open and comprehensive. At present, most data bases are proprietary and limited in scope. RSAC also suggests that consideration be given to increased emphasis on pollution indoors. The EPA Science Advisory Board has identified pollution indoors as a problem that "poses relatively high human health risks". ("Reducing Risk: Setting Priorities and Strategies for Environmental Protection", SAB-EC-90-021, September 1990.) Addressing high risk human health or environmental problems is one of the stated priority-setting criteria for driving choices in the research strategy. Thus, it is appropriate for the P2NT MYP to address issues related to pollution indoors. An additional focus of the P2NT MYP program should be full implementation of the Agency's Quality system (QS). The QS is the process by which the Agency establishes, documents and certifies data quality. Although Agency policy (Order 5360.1) requires use of an approved QAPP for all environmental data collection operations in which data are collected for or on behalf of the EPA, in practice this policy is not always followed. (Science Advisory Board Review of the Implementation of the Agency-Wide Quality System, EPA-SAB-EEC-LTR-99- 002.) For example, in its Review of EPA 's Environmental Technology Verification Program (EPA-SAB-EEC-00-012), the SAB noted that, "Not all verification partners are fully aware that. . . all work performed by extramural organizations on behalf of or funded by the EPA that involves the collection or use of environmental data in Agency programs shall be implemented in accordance with an Agency-approved quality assurance project plan (QAPP) developed from a systematic planning process (EPA QA/R-5)." At least for the verification portion of the P2NT MYP, and probably more generally, there is a need for improved QS implementation. By proper integration of data quality performance standards in the generic verification test protocol, the structure of specific Test/Quality Assurance (AQ) plans could be designed to reflect the inherent variability of a particular technology's performance. 3.3 Does the proposed scope of work proposed by the Office of Research and Development complement research by others? The relation of the proposed scope of work to research by others is not adequately presented. Thus, it is not known how or whether this work complements the research of others. 11 ------- It is stated in the plan that it is essential that ORD work more closely with those directly involved in the implementation of pollution prevention. The groups listed are EPA's Program Offices, the industrial community, states, communities, tribes, Federal organizations, and the international community. It is also noted in the plan that EPA collaborates with the National Science Foundation on an extramural grants program. However, the plan does not present a discussion of research conducted by others in the area of pollution prevention and new technology or of the extent that the P2NT MYP does not duplicate that being done by other entities. One of the stated priority-setting criteria for driving choices in the research strategy is to "fill important research gaps not being addressed by others". The Pollution Prevention MYP also does not contain specific reference to efforts to integrate relevant research from other federal, state, academic, or private/corporate laboratories. Relevant research is presently being conducted by the Departments of Defense and Energy among others. Obviously, information on research being conducted by others is needed to implement this criterion. In this regard, it is interesting to note that none of 86 deliverables (page 13 to 23 of the MYP) identified as annual performance measures (APM) indicates any potential interaction with other agencies, organizations, or companies having activity in issues related to this MYP. As a result, this current draft of the MYP does not indicate awareness of the research, development and implementation activities conducted by others, nor does it indicate how future efforts will keep abreast of the multitude of efforts underway in this area. As an example, there are broad approaches being taken by many industries that have the effect of changing production processes, environmental economics, and environmental measures of performance. Industries active in this effort include 3M, BP, Alcoa and Dow. An international organization active in this area is the World Business Council for Sustainable Development. There is no indication in the MYP that EPA researchers are aware of these on- going efforts and have efforts underway to track, understand and link EPA's activities with such efforts. In the verbal presentation of this MYP on October 16, it was noted that the Pollution Prevention Research Strategy Team did consider related research being conducted by others. It is recommended that a summary of this information be incorporated in the multiyear plan. Also, an explanation of how research undertaken by other entities influenced EPA's planned research should be included. The rationale for research planned by EPA is needed in this MYP. Although there is evidence of intent to cooperate with industries on P2 implementation, the APM's suggest a more unidirectional transfer of information from EPA to industry. Such an approach runs the risk of overlooking important P2 advances that are ongoing within the industrial sector, including robust data on the effectiveness of various P2 options. An important related issue is the disposition of proprietary data that contains information on P2 that is embedded within process manufacturing and product control approaches. Overall, a greater emphasis on industrial partnering would enhance the EPA's P2 research effort at all stages. 12 ------- In summary, the P2NT MYP should make it very clear that what EPA intends to do as part of this plan is not being done and can not be done better by industries and industrially- related organizations or by others. Similarly, in spite of the reduction in pollution prevention budgets within federal facilities and agencies, there has been a steady increase in the use of pollution prevention approaches to address environmental compliance issues and reduce costs. Compliance through pollution prevention (CTP2) programs, which focus on those pollutant emissions with the highest regulatory burden, are designed to give budget priority to those compliance methods that include pollution prevention. It would be useful for the Agency to establish liaisons with ongoing efforts at other Federal agencies that have fostered and developed P2 initiatives and practices. 3.4 Would accomplishing the annual performance goals allow the Office of Research and Development to both achieve the long-term goals and answer the science questions identified in the plan? Many of the concerns expressed in evaluating annual performance goals in relation to implementing the multiyear Water Quality plan appear relevant to the Pollution Prevention plan (Section 2.4). The P2 plan does not provide criteria for determining whether any of the long-term goals will be met as a result of accomplishing all of the listed annual performance goals. The P2 long-term goals are stated with sufficient generality that it would be difficult to determine if they had been reached, even if all of the annual performance goals were met. The annual performance goals are similarly very generally stated (e.g., see p. 13 - "Complete first generation of environmental impact assessment tools"). The P2 plan does not articulate what constitutes a "first generation" tool. The generality and selection of topical areas used to develop the annual performance goals suggests that there has not been a comprehensive evaluation of existing P2 approaches and technologies. The plan would benefit from some discussion concerning what has been accomplished in the pollution prevention areas addressed by the long-term goals, and, therefore, why these are the most important P2NT areas in which to invest research funds. It is not easy to find any annual performance measures in the P2 MYP. While there are 86 deliverables identified in the MYP (pages 13 to 23), these are deliverables and outputs. They are not really performance measures. As the MYP notes (pages 23 and 24), the performance and effectiveness of the program should be measured by the actual use of these technologies and tools. On p. 24 it is noted that an effort in outcome measurement must be conducted separately from the actual research. Apparently, although a funding increase is planned for the pollution prevention research program the MYP has no apparent effort related to annual or overall performance measures. In addition, there is no apparent programmatic effort included in the MYP to research, develop, identify or find suitable performance measures from a strategic standpoint. This is an unfortunate void in the MYP. This also would appear to be an area where the opportunity to interact with other organizations and companies could be vital. An interactive effort to learn 13 ------- what industries and global corporations are doing to measure environmental performance, particularly in pollution prevention, could be beneficial for everyone and it could help leverage resources. 3.5 Would accomplishing the annual performance measures clearly demonstrate that the associated annual performance goal was attained? To a large extent the response to this question was provided in Section 2.5. However, there are other issues that need consideration. One concern in evaluating whether completion of the listed annual performance measures will achieve the corresponding annual goals is that serious consideration needs to be given concerning the scientific and technical feasibility of accomplishing many of the measures. One performance measure to meet the long-term ESM goal will in 2003 "deliver principles for ecosystem sustainability based on information theory, ecological models, and field data...." (p. 21). However, there is no consensus among the scientific community that such principles can be developed or that such principles would be based on information theory or that they would derive from ecological models. The time scales relevant to sustainability would make it difficult to derive such principles from field data. This performance measure might not be feasible, particularly by 2003. Similar considerations of feasibility appear relevant to other performance measures outlined to support the ESM long-term goal. Thus, the Agency might consider rewriting the performance measures. 3.6 Does the plan clearly describe the outcomes that the Office of Research and Development will achieve through the proposed research? No. Goals are, of course, presented, but specific outcomes expected from the proposed research are not described in the plan itself. It is stated in an Appendix to the plan that if additional funds were made available "increased efforts on the measurements of outcomes would be made". However, some discussion of expected outcomes is needed in the MYP in order to evaluate the effectiveness of this program. This is especially important in view of the open- ended nature of the goals. In the presentation of the P2NT MYP on October 16, a diagram of a "Logic Model for the 'Tools' Component of the Pollution Prevention and New Technology Multiyear Plan" was included. This diagram included resources, examples of activities, outputs, customers and outcomes. RSAC found this diagram to be very helpful and recommends that it be included in the P2NT and that analogous diagrams be included in the other MYPs. Similar diagrams for the other components (green chemistry and engineering, environmental technology verification, and environmental systems management) would also strengthen the P2NT MYP. With respect to outcomes, the plan should not overly emphasize the pursuit of successful technology development or decision tools. A potentially important contribution of the supported research is that it can focus on investments with lower probabilities of commercial success. By 14 ------- doing so, the research may help clarify whether particular pollution prevention strategies should be continued or eliminated from further consideration by other parties. This could be an important contribution made by the P2NT efforts that would not be duplicated by P2NT activities by industries. Although the issue of outcomes was more directly addressed during the presentation of the P2 plan, there is still a lack of definition of what is meant by "outcomes" related to achieving a long-term goal. This is a similar problem to that found in the Water Quality MYP, and probably in other MYPs as well. There is a need for having clear definitions of what outcomes are. For example, one of the long-term outcomes presented under this MYP (in the presentation, not the document) is "reduced incidence of disease (e.g., cancer, asthma) from industrial and commercial toxic chemicals, especially organic-based solvents". It would be difficult to use this statement as an evaluation tool for determining the effectiveness of the Pollution Prevention MYP for obvious reasons: there are no associated measures that could be directly linked to the P2NT MYP as any reduction, however minor, would be considered a success, etc. This statement is actually a goal of multiple EPA research and regulatory efforts and of its overall strategic plan. An outcome should be more concretely stated and it should be relevant to the specific MYP. It should also include specific measures that can be used to compare the status of specific issues after implementation of the MYP with the status before its implementation (i.e., before the research is done) so that the effectiveness of the research can be determined. For example, an outcome relevant to the Pollution Prevention Program might be an specific reduction in emissions of certain pollutants by a particular date because of adoption of certain control technology. If this is not what EPA has meant by the term "outcome", then an appropriate definition should be provided in the introduction to the MYPs. It might be useful for the Agency to consider the definition of "outcome" in the human health field as a model, where there is a separate discipline of "outcomes research" that is dedicated to developing and investigating methods and measures for determining the effectiveness of disease treatment or prevention approaches. 15 ------- 4. COMMENTS ON THE MYP PROCESS 4.1 Overview As indicated earlier (Section 2.1), the intent of this RSAC effort was to review two draft ORD MYPs. The objective was to: a) evaluate available illustrative MYPs from a strategic standpoint; b) to understand the extent to which major issues were being addressed, particularly in the context of the overall EPA science and research strategy; c) to identify how annual performance goals were intended to relate to measures of performance; and, d) more importantly, to identify topics, themes, unclear issues, and "lessons learned" that could help increase the understanding and usefulness of the MYPs that are being and will be developed. What is presented in the two MYPs should be understood within the framework of the broader planning activities at EPA and RSAC's review processes. The review was structured around responses to six specific questions (Section 1.2). However, such a focus necessarily identifies gaps, unclear linkages, lack of context, and is based on an inadequate understanding of background efforts. As a result, responses tend to focus more on the negatives than the positives. This results in a possible impression that RSAC found the two MYPs to be quite inadequate. On the contrary, RSAC found great value in these MYPs, recognizes the thoughtful and dedicated efforts that have been spent in preparing these plans, and considers the MYPs to be a sound and essential part of both EPA and ORD research and budget planning. RSAC strongly encourages ORD to consider seriously the comments in this report and use them to finalize and implement these two MYPs and the remaining fourteen MYPs. As it has done in the past, RSAC commends ORD on the development and implementation of its planning structure for research, and the Committee once again observes that in the past five years ORD, in coordination with Program Offices and regional offices, has made considerable progress in its planning process, in its focus on strategies and goals, in the development of multiyear planning, in the use of National Program Directors, and in transitioning some portions of the R&D program to the states (e.g., Coastal Monitoring Program). Program staff interviewed by the committee during this and the last several meetings support RSAC's perception that there have been tremendous strides in communication between ORD and Program Offices. RSAC strongly recommends that ORD "stay the course" and continue these efforts. 4.2 Recommendations The RSAC review of these two illustrative MYPs revealed a number of items and points that should be considered as these MYPs are completed and as the other MYPs are written and finalized. Items and points that would be helpful in all MYPs include: 16 ------- a) Contextual information about how the particular multiyear strategy fits in the broader ORD strategy and complements the other multiyear plans. b) Information about the particular advantages of EPA's efforts in each of the broader research areas. The plans should clearly indicate why EPA's research strategy adds value to scientific and policy efforts. c) Discussion of the value of the research strategies also would benefit from placing the proposed work in context of complementary efforts conducted outside the Agency. d) Identification of the areas of research for which they were designed, i.e., core research, problem-driven research or both. For MYPs that include both areas, there should be an explicit formulation of information to indicate which components of the plan address the regulatory needs of the program offices, and which address the core research needs. e) Information in each MYP that indicates how the efforts and deliverables are to be accomplished, including consideration of the specific measurement issues and advances that will be addressed by the strategy. From a strategic view, a MYP should consider whether the implementation process (e.g., in-house staff vs. grant, contracts, etc.) substantially affects the attainment of deliverables and outcomes. For instance, the P2NT MYP should provide some information about the staging of the efforts, number of deliverables expected per year, and how the budget is to be spread over the four identified topics (Tools, GC&E, ETV and ESM). Without such information, it is difficult to identify the adequacy and strategic nature of any MYP. f) A careful consideration and clear discussion of the relationships between the long- term goals and the most important areas of scientific uncertainty. The more explicit these ties, the easier it will be for the reader to see how specific targets can actually make a significant contribution to expanding our knowledge and creating the desired outcomes for each planning area. In addition, planners are encouraged to develop long-term goals that are not open-ended. Annual goals cannot logically meet long-term goals where desired outcomes are not clearly articulated. g) Explicit and unambiguous definition of outcomes that are reflective of concrete and measurable milestones in the path towards achieving the MYPs long-term goals. The potential for success in reaching the long-term goals cannot be evaluated without the formulation of concrete outcomes and the time frame for their achievement. While EPA has grappled with the question of outcome definition to some extent, it is not apparent that it has engaged in the kind of strategic thinking required for this effort. As a start, it might be useful to derive parallels from the human health model of disease eradication, prevention, or treatment. In this model, outcomes have very clear and quantitative definitions, and are used for 17 ------- monitoring the progress of research or treatment strategies towards the eventual long-term goal of a cure or elimination of disease. Overall the Committee was particularly concerned about how well the documents and the presentations conveyed information that could be used to answer the fourth and fifth questions. Based on the information presented, the committee found that there was not a clear link between completion of the annual performance measures and achievement of the annual performance goals. There was no clear indication of how annual measures contributed to meeting annual goals. Moreover, some of the long-term goals were open-ended; therefore, the Committee remained concerned that annual goals could not logically lead to meeting the long-term goals. In addition, the Committee found that there is an apparent missing connection between the inventory of annual targets and the long-term goals. Consequently, the underlying research strategy and its potential are not transparent. A way of addressing this missing connection might be to collect measures and annual goals under target science issues or questions. These target issues or questions should be directly relevant to the key science questions that each of the long- term goals seeks to address. As an example, consider the long-term goal from the Water Quality MYP to improve approaches and methods to develop criteria to support designated water body uses. A related key science question is to understand the causal relationships between the varying levels of stressors and the biological response of aquatic ecosystems. The linkage between the annual goals and this key science question can be clarified by indicating which goals are intended to contribute to further understanding of the causal relationships. The Committee also recommends that the annual performance measures should be focused on desired outcomes, not outputs. The Committee found that the collection of annual measures in the plans reviewed were largely outputs. It was difficult to understand how the collection of outputs would eventually combine and contribute to achieving outcomes that further annual performance goals or long-term goals. Understandably, this is not an easy task, and ORD planners may conclude that articulation of the relationship between outputs and outcomes is itself an important research question. RSAC understands that developing and formulating concrete outcomes is not a short- term exercise and may require a significant effort on the part of EPA as well as more integration across related MYPs. In some sense this exercise is the reverse of the MYP process which focuses on meeting goals and which is "forward looking" in that EPA formulates a series of steps in the future that could lead eventually to achieving each goal. The definition of outcomes, instead, starts with the end in mind. By formulating specific, clear, and very concrete outcomes that are amenable to some sort of measurement the status after the control technologies, pollution prevention actions, etc. are put into place can be compared with the prior status (i.e., before the research was undertaken that led to the control technologies, pollution prevention strategies, etc.). Once the outcomes are formulated, the strategic thinking process "moves backward" to the present in order to formulate the series of steps necessary to achieve them. 18 ------- Neither of the two plans (and probably also those yet to be developed) provides clearly defined outcomes. MYPs have not considered carefully the operational differences between goals, objectives, performance measures, impact, and outcomes, so that statements under each of these rubrics could be frequently interchanged. RSAC recommends that the Agency define clearly in each MYP what is meant by each of these terms. These definitions and their specific use should be applied consistently across the Agency's documents. RSAC perceives that EPA has not engaged in the strategic thinking process that would allow it to state the expected outcomes of their MYPs in a manner that would allow the success or failure of the plans in achieving the stated goals to be evaluated. The Agency needs to undertake this effort as part of the development of the multiyear plans. 19 ------- APPENDIX A. ACRONYMNS APMs Annual Performance Measures CTP2 Compliance Through Pollution Prevention EMAP Environmental Monitoring and Assessment Program ESM Ecosystem Management LCA Life Cycle Assessment LCI Life Cycle Inventory MYP Multi-Year Plan P2 Pollution Prevention P2NT Pollution Prevention/New Technology QA Quality Assurance QAPP Quality Assurance Project Plan WQ Water Quality A-l ------- 5/24/01 APPENDIX B. ORD MULTI-YEAR PLANNING GUIDANCE Introduction The Office of Research and Development (ORD) multi-year plans (MYPs) serve as a tool to plan the direction of our research program and to communicate these plans both within ORD and with others. This MYP guidance describes the process for developing a MYP and presents a format to assist the lead authors in developing their assigned MYP. The use of a common format and approach will aid ORD in comparing different plans, identifying areas for integration across plans, and effectively communicating our ideas. The lead authors should use their discretion in tailoring this guidance, based on their knowledge and the input they receive, to best fit their topic. A list of current MYP topics and lead authors is included in Attachment 1, Lead Authors and Executive Guidance Groups. The calendar for revision of the draft MYPs is in Attachment 2. This guidance includes: »• Purpose of MYPs * Relationship of MYPs to other plans and strategies *• Overview of the MYP process *• MYP development *• MYP review and approval Purpose of MYPs a) ORD internal planning and communication b) Communication with EPA programs and regions c) Communication outside EPA Purpose of the Multi-Year Plans The purpose of the MYPs is to aid ORD as a planning and communication tool. Although the information contained in the MYPs is of interest to a wide variety of audiences, their primary purpose is planning and communication within ORD. The information in the MYP will also be useful for communicating our vision within the Agency, with other research organizations, and with anyone interested in the direction of ORD research. Planning MYPs are a planning tool intended to provide a link between the Agency and ORD's strategic plans and our annual plans that serve as the basis for ORD's budget request (the relationship between MYPs and other planning documents is described in the next section). MYPs must clearly describe what research ORD wants to accomplish to reach the long term goals and objectives we set for ourselves. By defining the direction and timing of our research program, the MYPs will assist ORD to define our unique role within the research community, focus resources on key problems and risk reduction, leverage our work with others, and demonstrate B-l ------- MYP Uses *• Provide the link between the Agency and ORD Strategic Plans and Research Strategies to Annual Plans and Laboratory Implementation Plans *• Use long term goals to develop annual performance goals and measures •• Explain the context and impact of annual planning decisions *• Provide a framework for integration across labs/centers, Agency goals, and with other research organizations *• Communicate the direction of ORD's research program internally and externally performance. progress. MYPs will help ORD managers better anticipate future needs for facilities and scientific expertise by illustrating how our research program is expected to evolve and where resources are needed in the future. MYPs provide a framework to conduct annual planning. MYPs will allow for a thorough understanding of the context for annual planning decisions and inform ORD managers of the potential impacts on ORD's ability to accomplish its goals on time. MYPs will also improve accountability by projecting work outcomes (annual performance goals) and outputs (annual performance measures) in advance, and developing ways for ORD to measure its Communication Since MYPs are intended to describe the logic and sequencing of ORD's research program, they can help to foster integration through increased awareness and understanding. Integration and collaboration across disciplines and goals should be increased by providing a mechanism to understand where similar work is needed in multiple areas, and broadly communicating possibilities for collaboration at all levels within ORD. MYPs should also improve the understanding of how individual work efforts play a role in both our overall research effort and accomplishing our organizational goals. MYPs will support external communication with our customers and stakeholders in the Agency and beyond. The stated outputs and outcomes, and the direction and sequencing of ORD's research efforts will improve ORD's ability to coordinate with EPA program and regional offices and better support the Agency's science needs. MYPs will help ORD communicate the direction and purpose of our research program to promote understanding and build support from others, including EPA's Science Advisory Board and Congress, and help ORD find opportunities to collaborate with other research organizations. Relationship of Multi-Year Plans to Other Plans and Strategies There are several important planning documents that guide ORD's research activities. Figure 1, Interaction Between Planning Documents, portrays the interaction of several of these key documents in relation to the MYP. The EPA and ORD Strategic Plans set the direction for ORD and highlight major areas of emphasis. These documents provide both the Agency's Government B-2 ------- National Env. Research Plan Fmcy Science Plans Research S trategy (by to pic) ti-Year Planning Annual Planning (RCTs) NSTC/EPA Priorities 8 Budget Guidance Research Efforts FTE, $, and Timelines 0 utpu ts Lab/Center Research Plans Figure 1, Interaction Between Planning Documents Performance and Results Act (GPRA) goals and ORD's interpretation of the science needed to support the achievement of these goals. These plans, in conjunction with Agency Science Plans where available, inform individual topic research strategies and plans that translate this overall direction into more specific strategic approaches, priorities, and outcomes. These documents do not generally address resources and are shown outside of the dashed box in Figure 1. While research strategies describe the approach to addressing scientific questions, research plans focus on where a lab oratory/center can best use its resources to develop science results. MYPs link the research strategies and research plans to show how ORD conducts research in an integrated fashion to achieve our long term goals, annual performance goals and annual performance measures. The phasing of goals and measures in the MYP reflects available resources and the sequencing of research activities. The MYP provides a basis for creating annual plans and a context to understand how decisions made in annual planning impact the ability of ORD to meet future goals and outcomes. They also improve ORD's understanding of the impact of Agency priorities and budget guidance, and allow for a more comprehensive understanding of changes needed to emphasize a new research direction or accelerate an existing research effort. ------- Multi-Year Plan Process Overview Multi-year planning allows ORD to consider the future strategic direction of the Agency and determine where scientific discovery can contribute. ORD will use strategic planning documents and partner with others in the Agency to define the major outcomes that ORD research will support. MYPs are assumed to cover a window of at least five to ten years. The GPRA structure of goals and objectives provides a useful starting point to organize this effort. After evaluating the major outcomes set by the Agency, ORD applies its scientific expertise to identify major areas of uncertainty and key science questions that must be answered. ORD will evaluate the key science questions and narrow the wide range of potential choices to those where ORD chooses to invest resources. The MYPs should describe why ORD chose the areas of research selected and identify the expected customer. The science questions form the basis for ORD's long term goals (LTGs). The MYPs will use a flow diagram to depict how annual performance goals (APGs) are logically sequenced toward accomplishing the LTG. The APGs define major outcomes needed to progress toward achieving the LTG. APGs are further defined in supporting tables by annual performance measures (APMs). APMs are lab oratory/center specific activities (or at least designated to a single responsible laboratory or center) defined as outputs that ORD will produce to accomplish the APGs. MYPs consist of three parts: > A narrative description that * Introduces multi-year planning and the purpose of the MYP and its relationship to the Agency's GPRA structure *• Describes the long term goals and underlying science questions * Explains the rationale behind the sequence of the annual performance goals and measures *• Addresses capacity/capability issues when the proposed work requires a shift in facilities or expertise *• A graphic depiction of the logical sequencing and interrelationship of APGs over time to achieve each LTG (flow diagram) * Tables that further describe the APGs and supporting APMs MYP Assumptions *• Annual resources will not exceed the most recent President's Budget *• Plans will cover five to ten years *• MYPs will be updated annually In order to develop a realistic limit to the amount of work that can be completed in a given year, MYPs will assume an annual budget no greater than the current President's Budget. Although the total resources will be constant, the distribution of resources across ORD laboratories and centers and the relative emphasis in any particular area is likely to change over the years in the planning window and are not expected to be constant. B-4 ------- It is important that the MYPs demonstrate how research results will be transferred and communicated to ORD customers. Specific activities should be considered as intermediate points to consolidate and communicate information. This type of activity may be best included in the form of an APM. Once the logic and sequence of work are written in the flow diagrams and supporting tables, a narrative is completed to explain the work. The narrative is intended to provide an overall context and describe the thought that went into the MYP. The description in the narrative is primarily intended to serve as a communication tool to inform others of the intent of the writing team. MYPs are intended to be living documents. They will be updated annually and are expected to change as the state-of-the-science advances to incorporate new thinking and priorities. Updates will be completed by October 1 of each year in order to have the MYPs available for the annual planning cycle. Writing the MYP The ORD Executive Council (EC) selects MYP topics, assigns a lead author to be responsible for developing each MYP, and will approve the completed product (discussed later under review and approval). Although the lead author has the responsibility to develop the assigned MYP, he or she is not expected to do the job alone. The lead author should assemble a writing team to help throughout the MYP development process. The writing team will provide the information needed to complete the document, organize the presentation of material, and write/prepare assigned materials. The lead author should consider knowledgeable staff from across EPA including ORD staff (at all levels), and staff from EPA program and regional offices as appropriate. Writing team members should be knowledgeable in various aspects of the MYP topic (e.g., needed and ongoing research within and outside of EPA, and existing and proposed regulations and Agency policy). One of the first tasks of the writing team is to review existing materials related to the MYP topic. These include, but are not limited to, GPRA goals and objectives; existing EPA and ORD strategic plans; prior annual MYP Roles and Responsibilities > ORD Executive Council (EC): Approves MYP guidance, assigns topics and lead authors, approves completed MYPs » ORD Science Council (SC): Provides scientific analysis of MYPs and recommends approval to the Executive Council > Lead Author: Selected by EC to develop an assigned MYP *• Writing Team: Representatives from ORD and across EPA selected by the lead author to assist in preparation of the MYP *• Executive Guidance Group (EGG): Consists of EC/SC members and provides senior ORD guidance to the writing team > Office of Science Policy (OSP): Develops MYP guidance and coordinates development and approval processes B-5 ------- performance plans; relevant existing research strategies and research plans; lab oratory/center specific annual plans; and strategic plans and direction from program and regional offices. The MYP does not need to show the same level of detail for new or emerging areas, but the MYP should give the best estimate of how work is envisioned to proceed given the available knowledge. Executive Guidance Groups Executive Guidance Groups (EGGs) will be assigned to provide senior level guidance to the writing teams. The EGGs will each consist of approximately three to five volunteers from the EC and Science Council (SC) with expertise in the MYP subject. The EGGs will meet periodically with the writing team throughout writing and revision of the MYPs to ensure the plan is focused on high priority science within the subject area, provide guidance on resource shifts, and assess ORD capacity and capabilities. A list of current EGG members is in Attachment 1. Steps to Complete the Flow Diagram *• Review existing material *• Develop science questions and long term goals * Develop and sequence annual performance goals Flow Diagram and Tables The majority of time and effort required to complete the MYP will be spent determining research priorities and describing the logical progression of work to accomplish them. The lead author and writing team must narrow the focus of the plan to reduce redundancy with research in other MYPs and describe why the work is necessary and unique. The MYPs should give compelling reasons why the areas of research were selected instead of other possible choices. A flow diagram (see Figure 2, Flow Diagram, and the example in Attachment 3) will be created to display the projected outcomes and actions needed to reach each Long Term Goal (LTG). The flow diagram presents, in a concise format, the annual performance goals (APGs) and their interrelationship across time. Since the flow diagram will illustrate how the APGs relate to the LTGs, a separate flow diagram will likely be needed for each LTG. The APGs should be connected to highlight the importance of the APG sequence and show if there is a critical path to achieve the LTG. APGs from other MYPs needed to accomplish the LTG should also be included in the flow diagram, requiring coordination between writing teams. The diagram should clearly reference the other MYP as the source of the APG. It should also be noted where APGs in the MYP are included in another MYP as critical to an LTG outside the plan. The cross linkage of APGs in the MYPs will aid in integration of research across ORD and explicitly identify dependency on other work. B-6 ------- FY03 FY04 FY05 FY06 FY07 FY08 I 1 I APGX 1 From XXXMYPr Long Term Goal 1 Figure 2, Flow Diagram The logic flow should include periodic assessment of the state-of-the-science at intermediate points to draw work together and provide information to the Program and Regional Offices (and ourselves). APGs and APMs should demonstrate where information will be brought together for communication to ORD customers. Development of Long-Term Goals The MYP describes the steps needed to achieve the LTGs and how they support the Agency's GPRA goals and objectives. As such, the foundation of the flow diagram is the development of the LTGs. The writing team must first consider where science can contribute to achieving an Agency GPRA goal or objective. The writing team will then identify the key science questions to be addressed. Writing teams should consider the questions of scientific merit, accountability, and capability and capacity presented under "Review and Approval" on page 14 of this guidance when narrowing the science questions and developing the LTG. The science questions will likely cover a broad Long-Term Goals * Include input from a wide range of sources *• Narrow in scope, not overly broad *• Consider three to five LTGs per MYP *• APGs show a logical progression toward LTG > Only LTGs that will be accomplished within planning window should have a date B-7 ------- range of activities, and the writing team will need to consider what are the key questions that ORD science will need to address. Once the science questions are focused, the writing team will create LTGs that define the research goals to be achieved from the proposed research and how the results of the research will support the mission of ORD and the Agency. The LTGs should be well thought out and include input from a wide range of sources. They should be narrow in scope to show the specific contribution of the research, and not overly broad such that any related work is applicable (i.e., "improve the science" in a particular area). The LTGs should be written so the APGs show a logical progression toward the stated research goal. The process of developing the flow diagram is likely to be an iterative process, and the LTG may be clarified as the writing team begins to define the APGs and examine how they support the LTG. Between three and five LTGs per MYP should be considered as a starting point, but may vary between plans. The LTGs that will be accomplished within the planning window should have a date. However, it is not necessary to force a date into the LTG if it falls outside the planning window. Development of Annual Performance Goals Once the LTGs are established, the steps needed to reach each LTG are determined and formulated into APGs. Developing and sequencing the APGs is one of the most difficult and time-consuming aspects of creating the MYP. Key outcomes required to achieve the LTG and the research needed to answer the science questions must be described. Available resources have to be considered and activities phased to project a schedule that will not require annual resources greater than the latest President's budget request. Annual Performance Goals *• Identify key outcomes to achieve the LTG *• Consider resources and phased activities to project a schedule > Align in a logical sequence to address knowledge gaps in early stages *• Design flow diagram to show the relationship between APGs and how they flow toward achieving the LTG The APGs should be aligned in a logical sequence so that knowledge gaps are addressed in the early stages for use in future research. The flow diagram should be designed to show the relationship between APGs and how they flow toward achieving the LTG. The flow diagram should include APGs from other MYPs, or equivalent level outcomes from other organizations, if the APG is critical to completing the LTG. If the LTG does not depend on the APG from another source, or if the APG is relevant but not on the critical path, they should not be included. Work in other MYPs or from other organizations should be included at the highest level possible and referenced (i.e., if an APG from another MYP is included, the supporting APMs should not be cited as well). It should be noted in the narrative that work of other research organizations on the critical path is a vulnerability, since ORD does not control the level of effort devoted to these activities. ------- Each APG should be further described through a table (see Figure 3, APG/APM Table and an example for one LTG in Attachment 4) showing the APMs that will contribute to achieving the APG. The APGs should be written so that the APMs show a logical progression toward the stated outcome in the APG. Development of Annual Performance Measures The next step is to determine how ORD will measure the accomplishment of the APGs and develop APMs. The APMs may be outputs or outcomes, and the completion of all the proposed APMs should demonstrate that the APG is achieved. APMs define what specific actions will be completed to accomplish the APG. Each APM should be assigned to an APG, be completed before the APG is considered complete, be assigned a fiscal year for completion, and have a designated laboratory or center responsible for the APM. APMs should be included that identify specific activities or products intended to communicate results at key points. Annual Performance Measures *• May be outputs or outcomes *• Assigned to an APG * Include the fiscal year of completion * Must be completed before the APG is completed * Are assigned to a single lead laboratory or center who is responsible for the APM ANNUAL PERFORMANCE GOALS AND MEASURES APG 1 - TITLE APM APM APM APM APM TITLE APM TITLE APM TITLE - XXX MYP, APG X, 2005 APM TITLE APG 2 - TITLE APM APM APM APM TITLE APM TITLE APM TITLE YEAR 2003 2001 2002 2002 2003 2004 2002 2002 2003 LAB/ CENTER ORD LAB/CENTER LAB/CENTER LAB/CENTER LAB/CENTER ORD LAB/CENTER LAB/CENTER LAB/CENTER Figures, APG/APM Table APMs from other MYPs or organizations can be included on the table if they are critical to accomplishing the APG. Again, all work from other sources should be included at the highest level possible and the source referenced for additional detail. APMs from other sources should be indicated in italics. B-9 ------- Narrative Development Although the majority of the work needed to create the MYP is spent developing the flow diagram and its supporting tables, a narrative description is needed to orient and inform readers of the writing team's intent. The narrative will provide background, and describe both why the research was selected and the logic behind the sequence of the work. The narrative will also include some quantitative information to give the reader an idea of the magnitude of the research effort described, as well as some trend information to inform the reader of where the relative effort will be as the program matures. The narrative should be concise. Approximately fifteen pages should be used as a guide. Background *• Describes Agency priorities and regulatory program *• Introduces science questions *• Identifies focus of ORD contribution * Describes state-of-the-science and work of others includes limited quantitative ana trend information Introduction *• Describes MYP Process *• Describes scope of MYP *• Identifies primary customers for work * Provides size of annual resources to nearest $1 million Introduction The introduction provides the reader with an overview of the scope of the MYP. The introduction should briefly describe what an MYP is, and reference any key documents, such as research strategies, that the MYP builds upon. The introduction should include a broad description of the scope covered by the MYP and the primary customer for the research described. The introduction should also state the size of the total annual program rounded to the nearest million dollars to inform the reader of the magnitude of the proposed program (e.g., "the research described in this MYP assumes annual resources of approximately $20 million.") Background The background section should familiarize the reader with the goals and objectives of the Agency and the resulting science questions that the ORD research will address. This should include a description of the major areas of uncertainty and opportunities for risk reduction. The background should note key regulatory programs and any cross-agency workgroups, agreements, or significant work by other organizations that influence ORD's decision to focus work in a particular area. The background should also briefly inform the reader of the current state of the science and ORD's position in the research community relative to the MYP topic. Lastly, the background should inform the reader of the role of the MYP relative to other ORD planning documents, such as research strategies and plans, similar to the discussion of Figure 1 on pages 2 and 3 of this guidance. B-10 ------- Narrative Description of Long Term Goals *• Introduces LTGs and rationale for selecting them *• Introduces flow diagram and tables *• Identifies changes in emphasis between LTGs over time I t Present the Long Term Goals After evaluating the GPRA goals and objectives set by the Agency, ORD will narrow the wide range of potential choices to those where ORD chooses to invest resources. The narrative should provide an overview to introduce the LTGs that will be addressed by the MYP and the science questions that form their basis. The narrative should convey why ORD chose the areas of research selected and why ORD can make the greatest contribution in these particular areas. The narrative should also introduce the flow diagram and supporting tables and describe how they support the discussion of the specific LTGs in the next section. A table should be included to show the relative emphasis changes over time among LTGs. The level of effort for each LTG should be described in general terms on the table. Simple descriptions should be used such as increasing, level, or decreasing along with a supporting rationale in the narrative text to explain the change in emphasis (see Figure 4, Table of Evolving LTG Emphases). The narrative should discuss potential changes needed in ORD capability and/or capacity to accomplish the LTGs. The plan should describe how potential shortfalls in capability or capacity will prevent ORD from accomplishing the work envisioned by the plan. Examples of capability and capacity issues include lack of technical expertise or lack of appropriate facilities. Area LTG1 LTG 2 LTGS LTG 4 LTGS Emphasis in MYP Planning Window Decreasing Level Level and then decreasing Increasing Increasing and then level Figure 4, Table of Evolving LTG Emphases B-ll ------- Description of the Flow Diagrams After describing the LTGs that will be addressed in the plan and the rationale for selecting the LTGs, each LTG should be presented with the supporting logic used to develop the flow diagram. The narrative should discuss the rationale used to sequence the APGs and discuss time dependent APGs that define a critical path (i.e., successful completion of the LTG is dependent on timely completion of the APG). Important APMs may also be discussed if they substantially add to the information presented. The discussion should include work from other MYPs or other research organizations (addressing only those included in the flow diagram) on the critical path and potential vulnerabilities. The narrative should also discuss how accomplishing the LTG relates to the science questions and the Agency's goals and objectives. The narrative should highlight plans to communicate research results to ORE) customers. This description should include interim products, evaluations of the state-of-the-science, and technology transfer. The writing team should consider adding a separate section or attachment that further describes the overall communication plan. Flow Diagram Description * Describes logic used to sequence APGs *• Describes important APMs where needed *• Discusses communication of research results Appendix Describing Potential Additional Work The writing teams may include an appendix to describe how the plan would be modified if additional resources are available. The proposed work in the appendix should be limited to a maximum of 20 percent of the current resource base. Proposed work should be considered to address (1) research gaps, (2) acceleration of the current program if necessary, (3) the next logical piece of work, and (4) work that was not considered in the original scope due to resource constraints. The appendix should clearly describe new or modified LTGs and the related science questions. The appendix is not anticipated to be at the same level of detail as the MYP, but should convey an outline of how the additional or accelerated work would affect the flow diagram. Review and Approval It is expected that writing teams will coordinate with the EGGs throughout the development/revision process to receive input from senior ORD leadership and streamline subsequent reviews. Completed/updated MYPs will be submitted to OSP who will coordinate B-12 ------- review by the full SC. Although individual SC members will participate throughout the writing process as EGG members, the full SC will have enough members who did not participate on an individual MYP EGG to give each MYP a critical review prior to referral to the EC for approval. The SC will review the plan for (1) the scientific merit of the MYP, (2) accountability, and (3) ORD capability/capacity to accomplish the proposed work within resource assumptions. Specific review questions for review by the SC include: Scientific Merit: 1. Do the long term goals support the priority needs of the program and regional offices? 2. Do the science questions address the most important areas of scientific uncertainty in the subject area? 3. Is the proposed scope of work consistent with the current state-of-the-science? Accountability: 1. Would accomplishing the long term goals enable ORD to accomplish the GPRA objective? 2. Would accomplishing the APGs allow ORD to achieve the long term goals and answer the science questions? 3. Would accomplishing the APMs clearly demonstrate the APG was attained? 4. Are the APGs and APMs measurable outcomes and outputs, respectfully (APMs may be outcomes or outputs, but generally represent outputs)? Capability and Capacity: 1. Does ORD have the capacity and capability (e.g., skill mix, facilities) to accomplish the proposed scope of work? If not, can ORD realistically obtain the capacity or capability within the projected time frame? 2. Is the proposed scope of work reasonable within the resource assumption (i.e., no greater than the current President's Budget or an agreed upon alternative)? The SC will make a recommendation to the EC regarding final approval of the MYPs, and the EC will approve final versions of the MYPs. ORD is considering options and requirements for external peer review of the MYPs after approval by the EC. This guidance will be appended when further information is available regarding potential external peer review. B-13 ------- |