United States EPA Science Advisory EPA-SAB-RSAC-02-003
Environmental Board (1400A) December 2001
Protection Agency Washington, DC iviviv.epa.gov/sab
&EPA WATER QUALITY AND
POLLUTION PREVENTION
MULTIYEAR PLANS: AN
SAB REVIEW
A REVIEW BY THE RESEARCH
STRATEGIES ADVISORY
COMMITTEE (RSAC) OF THE EPA
SCIENCE ADVISORY BOARD (SAB)
-------
I UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
/ WASHINGTON, D.C. 20460
December 14, 2001
OFFICE OF THE ADMINISTRATOR
SCIENCE ADVISORY BOARD
EPA-SAB-RSAC-02-003
Honorable Christine Todd Whitman
Administrator
U.S. Environmental Protection Agency
1200 Pennsylvania Avenue, NW
Washington, DC 20460
Subject: Review of the U.S. EPA Office of Research and Development's Water Quality and
Pollution Prevention Multiyear Plans: An SAB Report
Dear Governor Whitman:
On October 16 and 17, 2001 the Research Strategies Advisory Committee (RSAC) of the
EPA Science Advisory Board (SAB) met to review the Water Quality and Pollution Prevention
Multiyear Plans for the Office of Research and Development in the U.S. Environmental
Protection Agency.
Multiyear Plans (MYPs) provide a basis and context for developing annual plans.
Examination of how annual plans evolve from year to year serves to document the progress that
the Office of Research and Development (ORD) has made towards meeting its strategic goals to
help the Agency achieve appropriate environmental outcomes. The linkage between the research
strategies, prior year outputs and outcomes, and the research plans should clearly show how
ORD conducts and integrates its research program to achieve its long-term goals. The objective
of this review was to: a) evaluate available illustrative MYPs from the standpoint of ORD"s
science planning strategy; b) to understand the extent to which major issues were being
addressed, particularly in the context of the overall EPA science and research strategy; c) to
identify how annual performance goals were intended to relate to measures of performance; and
d) more importantly, to identify topics, themes, emerging issues, and "lessons learned" that
could help increase the understanding and usefulness of the other MYPs that are being and will
be developed.
The review was structured around responses to six specific questions which provided the
focus for our evaluation. However, such a focus leads the committee to identify gaps, unclear
linkages, and lack of context even though the review is necessarily based on an inadequate
understanding of background efforts. As a result, responses tend to focus more on the negatives
than the positives with the possible impression that RSAC found the two MYPs to be quite
inadequate. On the contrary, RSAC found great value in these MYPs. It recognizes the long and
arduous efforts that have been spent in preparing them, and considers them to be a sound and
-------
essential part of both EPA and ORD research and budget planning. RSAC strongly encourages
ORD to finalize and implement these two MYPs and the remaining fourteen MYPs.
RSAC's review of these illustrative MYPs revealed a number of items and points that
should be considered as the Pollution Prevention and Water Quality MYPs are completed and as
the other MYPs are written and finalized, including:
a) Contextual information about how the particular multiyear strategy fits in the broader
ORD strategy and complements the other multiyear plans.
b) Information about the particular value added by EPA's efforts in each of the broader
research areas. The plans should clearly indicate why EPA's research strategy
adds value to scientific and policy efforts.
c) Discussion of the value of the research strategies in the context of complementary
efforts conducted outside the Agency.
d) Identification of the areas of research for which they were designed, i.e., core
research, problem-driven research or both. For MYPs that include both areas,
there should be an explicit formulation of which components of the plan address
the regulatory needs of the program offices, and which address the core research
needs.
e) Information in each MYP that indicates how the efforts and deliverables are to be
accomplished including consideration of the specific measurement issues and
scientific and engineering advances identified in the strategy. From a strategic
view, a MYP should consider whether the implementation process (e.g., in house
staff vs. grant, contracts, etc.) substantially affects the attainment of deliverables
and outcomes.
f) A clear link between completion of the annual performance measures (i.e., outputs)
and the achievement of the annual performance goals (i.e., outcomes), for in the
absence of such a link it is not clear how the annual goals logically meet the long-
term goals.
g) Definition of outcomes that reflect concrete and measurable milestones in the path
towards achieving the MYPs long-term goals (i.e., clean air, water and soil). The
potential for success in reaching the long-term goals cannot be evaluated without
the formulation of concrete outcomes and the time frame for their achievement.
The Committee also recommends that the annual performance measures should be
focused on desired outcomes, not outputs. The Committee found that the collection of annual
measures in the plans reviewed were largely outputs. It was difficult to understand how the
collection of outputs would eventually combine and contribute to achieving outcomes that
-------
further annual performance goals or long-term goals. Developing outcome measures is not an
easy task, but it should be pursued. ORD planners may conclude that articulation of the
relationship between outputs and outcomes is itself an important research question.
We hope that these comments are helpful to the Office of Research and Development as
it finalizes it's first round of Multiyear Plans, and to the Agency, as it continues to refine its
planning processes to develop means to better plan for obtaining the science needed by EPA and
how to use it to best inform decision-making at the Agency. We look forward to your response to
our comments.
Sincerely,
/S/ /S/
Dr. William H. Glaze, Chair Dr. Raymond C. Loehr, Chair
EPA Science Advisory Board Research Strategies Advisory Committee
EPA Science Advisory Board
-------
NOTICE
This report has been written as part of the activities of the EPA Science Advisory Board,
a public advisory group providing extramural scientific information and advice to the
Administrator and other officials of the Environmental Protection Agency. The Board is
structured to provide balanced, expert assessment of scientific matters related to problems facing
the Agency. This report has not been reviewed for approval by the Agency and, hence, the
contents of this report do not necessarily represent the views and policies of the Environmental
Protection Agency, nor of other agencies in the Executive Branch of the Federal government, nor
does mention of trade names or commercial products constitute a recommendation for use.
Distribution and Availability: This EPA Science Advisory Board report is provided to the EPA
Administrator, senior Agency management, appropriate program staff, interested members of the
public, and is posted on the SAB website (www.epa.gov/sab). Information on its availability is
also provided in the SAB's monthly newsletter (Happenings at the Science Advisory Board).
Additional copies and further information are available from the SAB Staff [US EPA Science
Advisory Board (1400A), 1200 Pennsylvania Avenue, NW, Washington, DC 20460-0001; 202-
564-4533].
-------
ABSTRACT
The Research Strategies Advisory Committee (RSAC) of the EPA Science Advisory
Board (SAB) met October 16 and 17, 2001 to review the Water Quality and Pollution Prevention
Multiyear Plans of the Office of Research and Development. The objective was to: a) evaluate
available illustrative MYPs from a strategic standpoint; b) to understand the extent to which
major issues were being addressed, particularly in the context of the overall EPA science and
research strategy; c) to identify how annual performance goals were intended to relate to
measures of performance; and d) more importantly, to identify topics, themes, emerging issues,
and "lessons learned" that could help increase the understanding and usefulness of the MYPs
that are being and will be developed. The RSAC review of these two illustrative MYPs revealed
a number of items and points that should be considered as these MYPs are completed and as the
other MYPs are written and finalized. These included contextual information about how the
particular multiyear strategy fits in the broader ORD strategy and complements the other
multiyear plans, a discussion of specific measurement issues and advances that will be addressed
by the strategy, information about the particular advantages of EPA's efforts in each of the
broader research areas, and the relationship of the activities to be accomplished to
complementary efforts conducted outside of the Agency. The Committee also felt that MYPs
should explicitly address the areas of research for which they were designed and that information
be included in each MYP that indicates how the efforts and deliverables are to be accomplished
(e.g., by grants, contracts, in-house, etc.). Similarly, each plan would benefit from a careful
consideration and clear discussion of how the long-term goals can more likely be met by
addressing the most important areas of scientific uncertainty. Planners are encouraged to
develop long-term goals that are not open-ended, because annual goals cannot logically meet
long-term goals where desired outcomes are not clearly articulated.
Overall, RSAC found great value in these MYPs, recognizes the thoughtful and dedicated
efforts that have been spent in preparing them, and considers them to be a sound and essential
part of both EPA and ORD research and budget planning. RSAC strongly encourages ORD to
consider seriously the comments in this report and to use them to finalize and implement these
two MYPs and the remaining MYPs.
Keywords: Multiyear plan, GPRA, budget, research, strategic planning
11
-------
US ENVIRONMENTAL PROTECTION AGENCY
EPA SCIENCE ADVISORY BOARD
RESEARCH STRATEGIES ADVISORY COMMITTEE (RSAC)
CHAIR
Dr. Raymond C. Loehr, Professor, University of Texas at Austin, Department of Civil
Engineering, Austin, TX
MEMBERS
Dr. Steven Bartell, Principal, Cadmus Group, Oak Ridge, TN
Dr. Richard J. Bull, President, MoBull Consulting, Kennewick, WA
Dr. Robin Cantor, Principal and Managing Director, LECG, LLC, Washington, DC
Dr. Maria Morandi, Associate Professor of Environmental Science, University of Texas Health
Science Center at Houston, School of Public Health, Houston, TX
Dr. William Smith, Professor Emeritus of Forest Biology, School of Forestry and
Environmental Studies, Yale University New Haven, CT
Dr. James E. Watson, Professor, Department of Environmental Sciences and Engineering ,
University of North Carolina at Chapel Hill, Chapel Hill, NC
Liaison Member
Dr. Jerald Schnoor, Professor, Department of Civil and Environmental Engineering, University
of Iowa, Iowa City, IA. Also a member of the Board of Scientific Counselors.
Dr. Thomas Theis, Bayard D. Clarkson Distinguished Professor, Director of the Center for
Environmental Management, Clarkson University, Potsdam, NY. Also a member of the
Environmental Engineering Committee.
DESIGNATED FEDERAL OFFICER
Dr. John R. Fowle, III, Designated Federal Officer, US Environmental Protection Agency, EPA
Science Advisory Board (1400A), 1200 Pennsylvania Avenue, NW, Washington, DC
20460
Ms. Wanda R. Fields, Management Assistant, US Environmental Protection Agency, EPA
Science Advisory Board (1400A), 1200 Pennsylvania, Ave, NW, Washington, DC 20460
in
-------
TABLE OF CONTENTS
1. INTRODUCTION 2
1.1 Background 2
1.2 Charge to the Committee 4
1.3 Format of this Report 4
1.4 Process for the Review 4
2. RESPONSE TO THE CHARGE FOR THE WATER QUALITY MYP 6
2.1 Does the multiyear plan convey the Office of Research and Development's
strategic research plans for the subject area in an understandable fashion
and at an appropriate level of detail? 6
2.2 Do the long-term goals and underlying science questions identified in the plan
address the most important areas of scientific uncertainty in the subject area? 7
2.3 Does the proposed scope of work proposed by the Office of Research and
Development complement research by others? 8
2.4 Would accomplishing the annual performance goals allow the Office of Research
and Development to both achieve the long-term goals and answer the science
questions identified in the plan? 8
2.5 Would accomplishing the annual performance measures clearly demonstrate
that the associated annual performance goal was attained? 9
2.6 Does the plan clearly describe the outcomes that the Office of Research and
Development will achieve through the proposed research? 10
3. RESPONSE TO THE CHARGE FOR THE
POLLUTION PREVENTION MYP 11
3.1 Does the multiyear plan convey the Office of Research and Development's
strategic research plans for the subject area in an understandable fashion
and at an appropriate level of detail? 11
3.2 Do the long-term goals and underlying science questions identified in the plan
address the most important areas of scientific uncertainty in the subject area? .... 11
3.3 Does the proposed scope of work proposed by the Office of Research and
Development complement research by others? 12
3.4 Would accomplishing the annual performance goals allow the Office of Research
and Development to both achieve the long-term goals and answer the science
questions identified in the plan? 14
3.5 Would accomplishing the annual performance measures clearly demonstrate
that the associated annual performance goal was attained? 15
3.6 Does the plan clearly describe the outcomes that the Office of Research and
Development will achieve through the proposed research? 15
4. COMMENTS ON THE MYP PROCESS 17
4.1 Overview 17
4.2 Recommendations 17
APPENDIX A. ACRONYMNS A-l
APPENDIXB. ORD MULTI-YEAR PLANNING GUIDANCE B-l
iv
-------
1. INTRODUCTION
1.1 Background
Effective environmental protection requires extraordinary integration of research efforts.
Not only must the research be interdisciplinary, it has to be sensitive to the formulation of policy
decisions that are a central role of the EPA and provide knowledge that informs the Agency's
decisions. EPA's Office of Research and Development (ORD) developed Multiyear Plans
(MYPs) as a tool to plan the direction of their research program and to communicate these plans
both within ORD and to others. The MYPs are intended to provide a link between the Agency's
and ORD's strategic plans and the ORD annual plans that serve as the basis for ORD's budget
request. (See Appendix B, "ORD Multiyear Planning Guidance".)
There are several key planning documents that guide ORD's research activities. The
EPA and ORD Strategic Plans set the direction for ORD and highlight major areas of emphasis.
They provide for both the Agency's Government Performance and Results Act (GPRA) goals
and ORD's interpretation of the science needed to support the achievement of these goals. These
plans, in conjunction with Agency Science Plans (e.g., the Contaminated Sediment Science Plan
currently under development) when available, inform individual topic research strategies and
plans that translate the overall direction of the science program into more specific strategic
approaches, priorities and outcomes.
Research strategies describe the approach ORD will take to address the scientific
questions facing EPA, and research plans focus on where a laboratory or center can best use its
resources to develop science to inform decision-making. MYPs provide a basis for creating
annual plans and the context to understand how decisions made in annual planning impact the
ability of ORD to meet future goals and outcomes by linking the research strategies and research
plans to show how ORD conducts research in an integrated fashion to achieve long-term goals.
The MYP process also improves ORD's ability to understand and document the impact of
Agency priorities and budget guidance on its research program. Further, the MYP process
allows for a more comprehensive understanding of the changes needed to emphasize a new
research direction or accelerate an existing research effort.
ORD has briefed the Research Strategies Advisory Committee (RSAC) several times
over the past two years about their intent to develop 16 MYPs, and to keep the Committee
appraised of the progress made in developing them. RSAC is a Standing Committee of the EPA
Science Advisory Board (SAB) of the United States Environmental Protection Agency
(USEPA). It was convened to provide broad advice to the Administrator on research planning,
management, and budget development for use by the Agency in its long-term budget planning
process. RSAC provides a point of focus for the Board to consider the overall directions of the
intramural and extramural research programs of EPA.
ORD requested that RSAC review the first two MYPs for their utility in budget planning
because of the committee's mission, which includes an annual review of the President's budget
1
-------
request for the Science and Technology component of EPA's budget. Clearly, other SAB
committees as well as advisory committees outside the SAB have an interest in specific MYPs
and will likely provide review on additional MYPs.
Because ORD's resources are roughly equally divided between core research and
problem-driven research RSAC wished to review both a core and a problem-driven multiyear
plan in its evaluation of the Multiyear Planning Process. The Pollution Prevention (core) and
Water Quality (problem-driven) MYPs were the first two such MYPs available. Thus, they
were selected for review.
In its review of these two MYPs RSAC:
a) considered them to be illustrative of the entire 16 being prepared.
b) attempted to review the MYPs from a strategic overview perspective (i.e. "30,000 foot
view") rather than a detailed operational evaluation.
c) attempted to identify issues and items that could be helpful in the development and
understanding of all 16 MYPs.
d) attempted to understand the extent to which major issues of concern were being
addressed.
e) tried to determine how a MYP built upon knowledge developed by prior research at
EPA and by programs and efforts underway at other organizations and agencies.
f) tried to identify and understand the "value" of research (i.e., outputs) to existing
environmental needs and the nation, that is to say, why is this research necessary
and, if it is, why should EPA undertake the proposed research rather than other
entities?
g) attempted to understand the context for the two MYPs in terms of the overall EPA
science and research strategy.
h) attempted to identify the measures of performance that were being considered to
evaluate the effectiveness and value of the activities included in the MYP, and to
consider the appropriateness of such measures to identify environmental
outcomes (i.e., clean air, water and soil).
i) tried to identify "lessons learned" from the Multiyear Planning Process in this review
that could be used by the Agency to increase the understanding and usefulness of
the other MYPs that are being and will be developed.
Committee members agreed that what is presented in the two MYP's should be
understood within the framework of the broader planning activities at EPA and RSAC's review
-------
processes. Therefore, many of the comments and suggestions reflect a broader concern about
science planning in EPA.
This review is based upon documents received prior to the meeting held October 16-
17,2001; however, some issues were clarified or expanded during the presentations by EPA.
RSAC members also requested additional information during a telephone conference held on
October 11, 2001 in preparation for the meeting of October 16.
1.2 Charge to the Committee
EPA's charge to the committee was:
a) Does the multiyear plan convey the Office of Research and Development's strategic
research plans for the subject area in an understandable fashion and at an
appropriate level of detail?
b) Do the long-term goals and underlying science questions identified in the plan address
the most important areas of scientific uncertainty in the subject area?
c) Does the proposed scope of work proposed by the Office of Research and
Development complement research by others?
d) Would accomplishing the annual performance goals allow the Office of Research and
Development to both achieve the long-term goals and answer the science
questions identified in the plan?
e) Would accomplishing the annual performance measures clearly demonstrate that the
associated annual performance goal was attained?
f) Does the plan clearly describe the outcomes that the Office of Research and
Development will achieve through the proposed research?
1.3 Format of this Report
Following this Introduction, the report provides specific responses to the questions in the
Charge to the Committee for the Water Quality MYP (Chapter 2) and the Pollution Prevention
MYP (Chapter 3). General comments applicable to both MYPs and others under development
overall are found in Chapter 4.
1.4 Process for the Review
Presentations concerning the MYPs were made by ORD to the Committee on October
16, 2001. The RSAC had an active discussion of the above plans, and other relevant issues were
discussed with Agency personnel during the presentations and during discussions the following
-------
day. Readers of this report should recognize that the following evaluation of the two MYPs and
this report resulted from:
a) detailed reading of the two draft MYPs prior to the meeting on October 16
b) interactive discussion following the presentations on October 16, and
c) internal discussion by RSAC members immediately following the meeting and during
the several intervening weeks prior to releasing this report, including a
teleconference held November 8, 2001 to help finalize the report.
-------
2. RESPONSE TO THE CHARGE FOR THE WATER QUALITY MYP
2.1 Does the multiyear plan convey the Office of Research and Development's strategic
research plans for the subject area in an understandable fashion and at an
appropriate level of detail?
The Water Quality Multiyear Plan (MYP) was presented as an example of an important
research area that is structured entirely in the context of activities of the program office
responsible for administering the provisions of the Clean Water Act. The presentation clarified
that this plan included entirely problem-driven research only. The context of this research plan
is largely captured in Figure 1 of the MYP that illustrates the steps required to meet the water
quality goals which are tightly coupled to regulatory activities within the Office of Water under
the Clean Water Act. In addition, Figure 2 of the MYP shows specific links to other multiyear
plans. This was a very effective way of providing context for this MYP.
The answer to this charge question is, in general, yes. However the presentation and
subsequent discussion did indicate topics and issues for which greater exposition may be
warranted. Such topics and issues are discussed in the following paragraphs.
While this MYP was generally understood by the RSAC committee members who were
familiar with the program, the committee was of the opinion that the plan should be readable as a
stand-alone document. Thus, it would benefit by including more explicit descriptions of the
regulatory constructs of the Clean Water Act. For example, the plan addresses designated uses
for watersheds and how a body of water is designated as being impaired for specific uses, but it
does not provide an overview of what the designated uses of a water body are or the process that
is used for determining if a body of water is impaired. The RSAC suggests that the Water
Quality MYP provide some additional descriptions of key concepts (such as designated uses and
impaired uses) of the Clean Water Act to provide some perspective for those readers not directly
knowledgeable of Clean Water Programs.
The Water Quality MYP also makes reference to other MYPs that are of clear importance
to the Agency's implementation of the Clean Water Act. Particular MYPs that are identified
included the Ecosystem Protection MYP, which includes the Agency's Environmental
Monitoring and Assessment Program (EMAP) activities. The Water Quality MYP is critical for
developing the metrics necessary for the evaluation of the degree of success of the regulatory
programs under the Clean Water Act. The Ecosystem Protection MYP is critical for identifying
problems and causes of problems in the aquatic environment. The RSAC feels that the role of
other related MYPs needs to be made more explicit in an overarching introduction to the MYPs
because of their importance for problem definition and program evaluation.
There were also some obvious omissions within the Water Quality MYP and its
relationship to other MYPs. For instance, there was no mention of the Drinking Water MYP, nor
was a clear attention given as to how the Water Quality MYP works in support of a program that
protects drinking water as a significant use. Although elements of some projects related to
-------
drinking water issues are included in the Water Quality MYP, there are obvious omissions,
perhaps because the issue is being addressed in the Drinking Water MYP. This linkage is simply
not discernable in the Water Quality MYP. This and similar connections need to be made
explicit. Moreover, efforts should be made to make tighter connections between research in
drinking water and water quality. Some of these issues can be quite subtle, but if properly
developed could be synergistic across initiatives directed to health and ecosystem protection.
For example, it is probable that efforts to protect habitat would also have the effect of reducing
disinfection by-products in drinking water by simply reducing the amounts of organic carbon in
the water. A second, more particular issue, is that some wastewaters apparently contain traces of
secondary amines or secondary amine precursors that give rise to the formation of very potent
carcinogens, such as nitrosamines, when drinking water is chlorinated. In the few systems that
have been examined, these disinfection by-products may represent a major portion of the health
risk attributable to the chlorination of drinking water. Therefore, in order to protect drinking
water uses, parameters related to relatively innocuous compounds (i.e. traces of secondary
amines) might need to be controlled under the Clean Water Act.
2.2 Do the long-term goals and underlying science questions identified in the plan address
the most important areas of scientific uncertainty in the subject area?
The Committee was not able to answer this question in the affirmative. The development
of the long-term goals is an important construct for laying out the Water Quality MYP.
However, the RSAC found the long-term goals to be stated in such broad and non-specific terms
that it was virtually impossible to determine if and when a goal would be achieved. ORD should
take the initiative to describe more explicitly the body of research that satisfies each stated goal
as part of the framework for the MYP.
The science questions presented in the MYP generally reflected the priorities identified
in the plan and addressed the most important science issues related to the regulatory agenda of
the Office of Water (OW). ORD's identification of OW's priorities in the MYP was an effective
way of transit!oning to the more specific elements of the MYP that can be linked to individual
projects. RSAC understands that the priorities assigned to the research questions and the
resulting projects reflected a prioritization process that encapsulated simultaneous consideration
of the potential impact of the research results and the scientific uncertainties. Neither the plan
nor the presentation of the information in the MYP allowed RSAC to distinguish between the
relative importance of these two parameters in the prioritization effort. The discussion of
science questions and issues does not address specifically whether they reflected the greatest
uncertainty or the greatest impact.
It was acknowledged in discussion with the Office of Water staff who attended the RSAC
meeting that some states have not discharged their responsibilities of designating uses for
particular water bodies or issuing indications of impaired use and that this might be a high
impact/high uncertainty science issue. If so, OW should direct some effort to helping ORD
understand why some states do not designate uses or impairments of use (i.e., is it due to
resource limitations, lack of appropriate guidance, or to other reasons?) as a means to better
target the research to client needs.
-------
The RSAC was somewhat confused by the fact that the discussion related to establishing
cause and effect relationships cited under Long-term Goal 1 did not include discussions of
uncertainty. The concern arose relative to the use of biological measures of impairment without
rigorous attention to cause and effect issues. In discussions following the presentations it
became clear that the term "uncertainty" was being used broadly and without much specificity.
The RSAC suggests that future iterations of the plan should clarify some of the ambiguities of
the language used in the phrasing of science questions. Some members of the RSAC were of the
opinion that consideration of biological impairment will be very important for defining outcome
measures for both the research and the regulatory programs. RSAC urges ORD to provide more
explicit information related to these issues in subsequent drafts of the Water Quality MYP.
2.3 Does the proposed scope of work proposed by the Office of Research and Development
complement research by others?
The integration of ORD water quality research with federal, state, academic, and private
laboratories starts with an awareness of non-EPA research. Some information related to ORD
efforts to integrate non-EPA research in the framing of science questions was described during
the oral presentations and some was detailed in the MYP. The Water Quality MYP indicates
that attempts were made to review relevant non-EPA research by holding workshops and via
several other strategies associated with specific science issues. However, the committee could
not answer this charge question in the affirmative, because explicit details regarding how the
EPA efforts described in the MYP "complement" the non-EPA research efforts were not
provided. RSAC recommends that in subsequent drafts of this MYP ORD address in greater
detail and specificity the extent to which the efforts described in the Water Quality MYP
complement and do not duplicate past or on-going research by others.
2.4 Would accomplishing the annual performance goals allow the Office of Research and
Development to both achieve the long-term goals and answer the science questions
identified in the plan?
Accomplishing the indicated annual performance goals described in the plan might allow
the Office of Research and Development to make progress towards achieving the stated long-
term goals and might provide some initial answers to the science questions identified in the plan.
However, it should be recognized that the set of stated annual performance goals may not
necessarily constitute the entire set of intermediate milestones needed to be met in order to
answer the key scientific questions that need to be answered in order to achieve the long-term
water quality goals of the MYP. Thus, the committee could not answer this question in the
affirmative.
The three long-term goals described in the Water Quality MYP are quite open-ended
from the perspective of supplying the necessary science to achieve these goals. The Water
Quality MYP does not provide the criteria that were used to identify all the necessary and
sufficient annual performance goals that, if achieved, would provide definitive answers to the
long-term goals. Therefore, it remains difficult to evaluate whether the accomplishment of the
annual goals in sum will in fact provide answers to these three longer term performance-driven
7
-------
objectives. Completion of the annual goals provided in Tables 2-4 of the MYP does not
necessarily "add up" to providing answers to the longer term goals.
Each of the stated annual performance goals should be examined in relation to its
technical feasibility, as well as its relevance and contribution towards realizing the long-term
goals identified in the MYP. For example, the second question asked in relation to the first long-
term goal addresses the "best way" to classify ecosystems, landscapes, and watersheds under the
assumption that such a classification will lead to efficient and sound development of water
quality criteria. However, there are no stated annual performance goals aimed at evaluating the
implicit assumption that developing water quality criteria would in any way be facilitated even
by the existence of the "best" ecological classification. Similarly, the questions in the MYP
stated in support of the second long-term goal concerning the inverse problem ask "how can.
ecosystems be classified", yet there are no performance goals for this problem.
Several performance goals outlined in Tables 2 appear very specific and clearly refer to
ongoing projects that appear motivated by Agency interests or by prior and on-going projects
that are perhaps consistent with the multiyear plan (e.g., habitat alteration and mercury impacts
on the Great Lakes loon, effects of habitat degradation and flow alterations on salmon in the
Pacific Northwest). However, these goals do not seem to necessarily derive from the MYP.
Similar examples can be identified in Tables 3 and 4.
In summary, RSAC strongly suggests that the MYP make the linkage between the annual
performance goals and the long-term water quality goals much clearer.
2.5 Would accomplishing the annual performance measures clearly demonstrate that the
associated annual performance goal was attained?
RSAC cannot provide an affirmative to this question. As discussed in relation to annual
performance goals (Section 2.4), accomplishing the annual performance measures might allow
ORD to make progress towards achieving the long-term performance goals. However, it must be
emphasized that the list of annual performance measures provided in the Water Quality MYP
does not necessarily constitute the entire set of performance measures that might be required to
attain the annual performance goals. This MYP does not present or discuss criteria that will be
used to determine if annual performance measures have been met or nor does it discuss how the
completion of the annual performance measures contributes to meeting the annual performance
goals. Many of the performance measures are based simply on providing some kind of a report.
This may become important in evaluating the overall success of implementing the plan, because
many of the performance measures appear to involve providing interpretations of supporting
science or technical research that has already been largely developed and exists as part of the
scientific knowledge base (i.e., peer reviewed technical literature). The existing knowledge base
that might contribute substantially to meeting performance measures was not necessarily funded
by the Agency or developed to address the three long-term goals stated in the water quality plan.
Analogous to evaluating the annual performance goals, the annual performance measures
should be individually assessed in terms of their relevance to the stated goals. Examination of the
stated annual performance measures in relation to the performance goals suggests that some of
the performance measures represent works in progress that have been incorporated as part of the
strategic plan, but which do not necessarily derive from the plan. Additionally, the plan could be
-------
strengthened by providing some information that describes how the various performance
measures were scheduled. It would also be helpful if the MYP were described in relationship to
the Quality Plan for these efforts.
2.6 Does the plan clearly describe the outcomes that the Office of Research and
Development will achieve through the proposed research?
No, although the Water Quality MYP does have sections entitled "Expected Impact and
Outcomes" after the summary of each of the three goals and the related research questions. First,
the document does not define clearly what is meant by impact (which is a relatively imprecise
and general term) and outcome (a term that connotes the idea of measurement). Consequently,
the outcomes are described rather briefly and in very general terms. It appears that the Agency
has not engaged in the strategic thinking required to develop measures of the effectiveness of the
research plan for meeting each of the three long-term goals. RSAC recognizes that there might
be a range of outcomes over time and that defining each of them and the parameters to be used as
indicators of success is not a trivial exercise.
There is obviously a concern that by explicitly describing expected outcomes and related
measures, a partial success could be perceived as a failure of the research plan in achieving the
stated goals. However, the definition of expected outcomes is not only important as an
effectiveness evaluation tool for the MYPs, but it is also helpful because specifically defined
outcomes (i.e., clean air, water and soil) provide long-term focal points or targets for the research
plan in a far better operational way than the goals themselves. Thus, as projects are developed or
new findings are made that may change or refocus the direction of research, the specific
outcomes provide permanent guiding posts that are directly linked to the long-term goals and
that are concrete milestones.
For example, the ultimate outcome of this MYP could be to have a fraction of, or even
all, the watersheds in the country classified into a scientifically robust water designation use
category by a predefined target year. This is a concretely defined outcome (as compared to the
broad statement of the goals) that: 1) would require a certain rate of progress in the classification
of watersheds over time (which could be linked to annual performance goals); and 2) would also
probably require the Agency to have a hierarchy of science questions related to water use
designation so that those questions that require longer term research start to be addressed early-
on.
A concretely defined outcome would also provide impetus for maximizing the efficiency
of the research plan (i.e., what science questions are likely to have the most impact in helping
increase the numbers of watersheds classified for use, and how precisely do we need to know the
relevant drivers for watershed quality classification before a reasonably robust classification can
be made?). In addition, concrete outcomes are very useful tools for involving stakeholders
within and outside the Agency in the whole process of multiyear planning.
-------
3. RESPONSE TO THE CHARGE FOR THE
POLLUTION PREVENTION MYP
3.1 Does the multiyear plan convey the Office of Research and Development's strategic
research plans for the subject area in an understandable fashion and at an
appropriate level of detail?
The draft Multiyear Plan (MYP) for Pollution Prevention and New Technologies (P2NT)
for Environmental Protection (EPA/600/R-DRAFT- August 2001) represents a considerable and
positive effort to address important issues relative to this area of activity. The individuals who
prepared this MYP have created a comprehensive, and readable, plan of action.
However, this MYP is not a strategic plan for this technical area. It also does not appear
to address all of the five priority setting criteria that are to drive choices in research emphasis
(page 2 of the MYP). These criteria are noted as: 1) address high risk problems; 2) respond to
needs of stakeholders; 3) fill important research gaps not being addressed by others; 4) leverage
resources with other organizations; and 5) provide potentially effective research. As indicated
by material in the MYP, criteria 2) and 5) appear to be met. However, there is little in the MYP
that allows a reader to judge that high-risk problems are being addressed. RSAC is not
suggesting that this MYP does not address high risk problems, but that there is little information
in the MYP to support that it does. There also appears to be no information that indicates that
criteria 3 and 4 have been addressed.
The Committee found that the strategic value of the P2NT MYP should be clarified by
further discussion of EPA's comparative advantage in certain research and technology transfer
areas. The document must convey how the P2NT long-term goals, measures, and annual goals
add value to on-going efforts in other agencies and the private sector. What is the EPA niche in
this area? What factors or information support EPA's conclusion that these projects are "below
the radar screen" of other parties capable of conducting the projects? The public policy
objectives should be better articulated to convey the potential contribution of the plan.
Notwithstanding the concern about justification made above, the "strategic research
plans" presented in ORD's the P2NT MYP are well-described and understandable. However, as
stated below, both the annual performance goals and the long-term goals are open-ended, except
for Goal III (Environmental Technology Verification), and specific outcomes expected from the
proposed research are not presented. Thus, it is not clear that these goals could be completely
achieved and, as a result, the effectiveness of this MYP can not be evaluated.
3.2 Do the long-term goals and underlying science questions identified in the plan address
the most important areas of scientific uncertainty in the subject area?
No. An important question would seem to be whether the long-term goals and underlying
science questions identified in the plan address the most important scientific issues, not just
uncertainty, in the subject area. The matter of uncertainty, which is critical to many research
10
-------
areas, seems applicable only to a subset of items in this subject area. For instance, one of the
critical issues for pollution prevention, and the sustainability paradigm generally, is the
development of commensurable metrics (i.e., metrics that are comparable and meaningful across
human and industrial activities).
The science underlying P2 performance metrics is poorly developed, yet it is critical to
allocating resources optimally. Investigations into such metrics are especially compatible with
the long-term goal of the P2 research program, and in particular those Annual Performance
Measures (APM's) associated with the development of Life Cycle Assessment (LCA) evaluative
techniques. Also, since LCA analyses are based on appropriate data, a portion of the LCA,
known as Life Cycle Inventory or LCI, it is important that the agency take steps to ensure that
the quality and availability of databases be peer-reviewed, open and comprehensive. At present,
most data bases are proprietary and limited in scope.
RSAC also suggests that consideration be given to increased emphasis on pollution
indoors. The EPA Science Advisory Board has identified pollution indoors as a problem that
"poses relatively high human health risks". ("Reducing Risk: Setting Priorities and Strategies for
Environmental Protection", SAB-EC-90-021, September 1990.) Addressing high risk human
health or environmental problems is one of the stated priority-setting criteria for driving choices
in the research strategy. Thus, it is appropriate for the P2NT MYP to address issues related to
pollution indoors.
An additional focus of the P2NT MYP program should be full implementation of the
Agency's Quality system (QS). The QS is the process by which the Agency establishes,
documents and certifies data quality. Although Agency policy (Order 5360.1) requires use of an
approved QAPP for all environmental data collection operations in which data are collected for
or on behalf of the EPA, in practice this policy is not always followed. (Science Advisory Board
Review of the Implementation of the Agency-Wide Quality System, EPA-SAB-EEC-LTR-99-
002.)
For example, in its Review of EPA 's Environmental Technology Verification Program
(EPA-SAB-EEC-00-012), the SAB noted that, "Not all verification partners are fully aware that.
. . all work performed by extramural organizations on behalf of or funded by the EPA that
involves the collection or use of environmental data in Agency programs shall be implemented
in accordance with an Agency-approved quality assurance project plan (QAPP) developed from
a systematic planning process (EPA QA/R-5)." At least for the verification portion of the P2NT
MYP, and probably more generally, there is a need for improved QS implementation. By proper
integration of data quality performance standards in the generic verification test protocol, the
structure of specific Test/Quality Assurance (AQ) plans could be designed to reflect the inherent
variability of a particular technology's performance.
3.3 Does the proposed scope of work proposed by the Office of Research and Development
complement research by others?
The relation of the proposed scope of work to research by others is not adequately
presented. Thus, it is not known how or whether this work complements the research of others.
11
-------
It is stated in the plan that it is essential that ORD work more closely with those directly
involved in the implementation of pollution prevention. The groups listed are EPA's Program
Offices, the industrial community, states, communities, tribes, Federal organizations, and the
international community. It is also noted in the plan that EPA collaborates with the National
Science Foundation on an extramural grants program. However, the plan does not present a
discussion of research conducted by others in the area of pollution prevention and new
technology or of the extent that the P2NT MYP does not duplicate that being done by other
entities. One of the stated priority-setting criteria for driving choices in the research strategy is
to "fill important research gaps not being addressed by others".
The Pollution Prevention MYP also does not contain specific reference to efforts to
integrate relevant research from other federal, state, academic, or private/corporate laboratories.
Relevant research is presently being conducted by the Departments of Defense and Energy
among others. Obviously, information on research being conducted by others is needed to
implement this criterion.
In this regard, it is interesting to note that none of 86 deliverables (page 13 to 23 of the
MYP) identified as annual performance measures (APM) indicates any potential interaction with
other agencies, organizations, or companies having activity in issues related to this MYP. As a
result, this current draft of the MYP does not indicate awareness of the research, development
and implementation activities conducted by others, nor does it indicate how future efforts will
keep abreast of the multitude of efforts underway in this area.
As an example, there are broad approaches being taken by many industries that have the
effect of changing production processes, environmental economics, and environmental measures
of performance. Industries active in this effort include 3M, BP, Alcoa and Dow. An
international organization active in this area is the World Business Council for Sustainable
Development. There is no indication in the MYP that EPA researchers are aware of these on-
going efforts and have efforts underway to track, understand and link EPA's activities with such
efforts.
In the verbal presentation of this MYP on October 16, it was noted that the Pollution
Prevention Research Strategy Team did consider related research being conducted by others. It
is recommended that a summary of this information be incorporated in the multiyear plan. Also,
an explanation of how research undertaken by other entities influenced EPA's planned research
should be included. The rationale for research planned by EPA is needed in this MYP.
Although there is evidence of intent to cooperate with industries on P2 implementation,
the APM's suggest a more unidirectional transfer of information from EPA to industry. Such an
approach runs the risk of overlooking important P2 advances that are ongoing within the
industrial sector, including robust data on the effectiveness of various P2 options. An important
related issue is the disposition of proprietary data that contains information on P2 that is
embedded within process manufacturing and product control approaches. Overall, a greater
emphasis on industrial partnering would enhance the EPA's P2 research effort at all stages.
12
-------
In summary, the P2NT MYP should make it very clear that what EPA intends to do as
part of this plan is not being done and can not be done better by industries and industrially-
related organizations or by others.
Similarly, in spite of the reduction in pollution prevention budgets within federal
facilities and agencies, there has been a steady increase in the use of pollution prevention
approaches to address environmental compliance issues and reduce costs. Compliance through
pollution prevention (CTP2) programs, which focus on those pollutant emissions with the
highest regulatory burden, are designed to give budget priority to those compliance methods that
include pollution prevention. It would be useful for the Agency to establish liaisons with
ongoing efforts at other Federal agencies that have fostered and developed P2 initiatives and
practices.
3.4 Would accomplishing the annual performance goals allow the Office of Research and
Development to both achieve the long-term goals and answer the science questions
identified in the plan?
Many of the concerns expressed in evaluating annual performance goals in relation to
implementing the multiyear Water Quality plan appear relevant to the Pollution Prevention plan
(Section 2.4). The P2 plan does not provide criteria for determining whether any of the long-term
goals will be met as a result of accomplishing all of the listed annual performance goals. The P2
long-term goals are stated with sufficient generality that it would be difficult to determine if they
had been reached, even if all of the annual performance goals were met. The annual performance
goals are similarly very generally stated (e.g., see p. 13 - "Complete first generation of
environmental impact assessment tools"). The P2 plan does not articulate what constitutes a
"first generation" tool.
The generality and selection of topical areas used to develop the annual performance
goals suggests that there has not been a comprehensive evaluation of existing P2 approaches and
technologies. The plan would benefit from some discussion concerning what has been
accomplished in the pollution prevention areas addressed by the long-term goals, and, therefore,
why these are the most important P2NT areas in which to invest research funds.
It is not easy to find any annual performance measures in the P2 MYP. While there are
86 deliverables identified in the MYP (pages 13 to 23), these are deliverables and outputs. They
are not really performance measures. As the MYP notes (pages 23 and 24), the performance and
effectiveness of the program should be measured by the actual use of these technologies and
tools. On p. 24 it is noted that an effort in outcome measurement must be conducted separately
from the actual research. Apparently, although a funding increase is planned for the pollution
prevention research program the MYP has no apparent effort related to annual or overall
performance measures.
In addition, there is no apparent programmatic effort included in the MYP to research,
develop, identify or find suitable performance measures from a strategic standpoint. This is an
unfortunate void in the MYP. This also would appear to be an area where the opportunity to
interact with other organizations and companies could be vital. An interactive effort to learn
13
-------
what industries and global corporations are doing to measure environmental performance,
particularly in pollution prevention, could be beneficial for everyone and it could help leverage
resources.
3.5 Would accomplishing the annual performance measures clearly demonstrate that the
associated annual performance goal was attained?
To a large extent the response to this question was provided in Section 2.5. However,
there are other issues that need consideration. One concern in evaluating whether completion of
the listed annual performance measures will achieve the corresponding annual goals is that
serious consideration needs to be given concerning the scientific and technical feasibility of
accomplishing many of the measures.
One performance measure to meet the long-term ESM goal will in 2003 "deliver
principles for ecosystem sustainability based on information theory, ecological models, and field
data...." (p. 21). However, there is no consensus among the scientific community that such
principles can be developed or that such principles would be based on information theory or that
they would derive from ecological models. The time scales relevant to sustainability would make
it difficult to derive such principles from field data. This performance measure might not be
feasible, particularly by 2003. Similar considerations of feasibility appear relevant to other
performance measures outlined to support the ESM long-term goal. Thus, the Agency might
consider rewriting the performance measures.
3.6 Does the plan clearly describe the outcomes that the Office of Research and
Development will achieve through the proposed research?
No. Goals are, of course, presented, but specific outcomes expected from the proposed
research are not described in the plan itself. It is stated in an Appendix to the plan that if
additional funds were made available "increased efforts on the measurements of outcomes would
be made".
However, some discussion of expected outcomes is needed in the MYP in order to
evaluate the effectiveness of this program. This is especially important in view of the open-
ended nature of the goals.
In the presentation of the P2NT MYP on October 16, a diagram of a "Logic Model for
the 'Tools' Component of the Pollution Prevention and New Technology Multiyear Plan" was
included. This diagram included resources, examples of activities, outputs, customers and
outcomes. RSAC found this diagram to be very helpful and recommends that it be included in
the P2NT and that analogous diagrams be included in the other MYPs. Similar diagrams for the
other components (green chemistry and engineering, environmental technology verification, and
environmental systems management) would also strengthen the P2NT MYP.
With respect to outcomes, the plan should not overly emphasize the pursuit of successful
technology development or decision tools. A potentially important contribution of the supported
research is that it can focus on investments with lower probabilities of commercial success. By
14
-------
doing so, the research may help clarify whether particular pollution prevention strategies should
be continued or eliminated from further consideration by other parties. This could be an
important contribution made by the P2NT efforts that would not be duplicated by P2NT
activities by industries.
Although the issue of outcomes was more directly addressed during the presentation of
the P2 plan, there is still a lack of definition of what is meant by "outcomes" related to achieving
a long-term goal. This is a similar problem to that found in the Water Quality MYP, and
probably in other MYPs as well. There is a need for having clear definitions of what outcomes
are. For example, one of the long-term outcomes presented under this MYP (in the presentation,
not the document) is "reduced incidence of disease (e.g., cancer, asthma) from industrial and
commercial toxic chemicals, especially organic-based solvents".
It would be difficult to use this statement as an evaluation tool for determining the
effectiveness of the Pollution Prevention MYP for obvious reasons: there are no associated
measures that could be directly linked to the P2NT MYP as any reduction, however minor,
would be considered a success, etc. This statement is actually a goal of multiple EPA research
and regulatory efforts and of its overall strategic plan.
An outcome should be more concretely stated and it should be relevant to the specific
MYP. It should also include specific measures that can be used to compare the status of specific
issues after implementation of the MYP with the status before its implementation (i.e., before the
research is done) so that the effectiveness of the research can be determined. For example, an
outcome relevant to the Pollution Prevention Program might be an specific reduction in
emissions of certain pollutants by a particular date because of adoption of certain control
technology. If this is not what EPA has meant by the term "outcome", then an appropriate
definition should be provided in the introduction to the MYPs. It might be useful for the Agency
to consider the definition of "outcome" in the human health field as a model, where there is a
separate discipline of "outcomes research" that is dedicated to developing and investigating
methods and measures for determining the effectiveness of disease treatment or prevention
approaches.
15
-------
4. COMMENTS ON THE MYP PROCESS
4.1 Overview
As indicated earlier (Section 2.1), the intent of this RSAC effort was to review two draft
ORD MYPs. The objective was to: a) evaluate available illustrative MYPs from a strategic
standpoint; b) to understand the extent to which major issues were being addressed, particularly
in the context of the overall EPA science and research strategy; c) to identify how annual
performance goals were intended to relate to measures of performance; and, d) more importantly,
to identify topics, themes, unclear issues, and "lessons learned" that could help increase the
understanding and usefulness of the MYPs that are being and will be developed. What is
presented in the two MYPs should be understood within the framework of the broader planning
activities at EPA and RSAC's review processes.
The review was structured around responses to six specific questions (Section 1.2).
However, such a focus necessarily identifies gaps, unclear linkages, lack of context, and is based
on an inadequate understanding of background efforts. As a result, responses tend to focus more
on the negatives than the positives.
This results in a possible impression that RSAC found the two MYPs to be quite
inadequate. On the contrary, RSAC found great value in these MYPs, recognizes the thoughtful
and dedicated efforts that have been spent in preparing these plans, and considers the MYPs to
be a sound and essential part of both EPA and ORD research and budget planning. RSAC
strongly encourages ORD to consider seriously the comments in this report and use them to
finalize and implement these two MYPs and the remaining fourteen MYPs.
As it has done in the past, RSAC commends ORD on the development and
implementation of its planning structure for research, and the Committee once again observes
that in the past five years ORD, in coordination with Program Offices and regional offices, has
made considerable progress in its planning process, in its focus on strategies and goals, in the
development of multiyear planning, in the use of National Program Directors, and in
transitioning some portions of the R&D program to the states (e.g., Coastal Monitoring
Program). Program staff interviewed by the committee during this and the last several meetings
support RSAC's perception that there have been tremendous strides in communication between
ORD and Program Offices. RSAC strongly recommends that ORD "stay the course" and
continue these efforts.
4.2 Recommendations
The RSAC review of these two illustrative MYPs revealed a number of items and points
that should be considered as these MYPs are completed and as the other MYPs are written and
finalized.
Items and points that would be helpful in all MYPs include:
16
-------
a) Contextual information about how the particular multiyear strategy fits in the broader
ORD strategy and complements the other multiyear plans.
b) Information about the particular advantages of EPA's efforts in each of the broader
research areas. The plans should clearly indicate why EPA's research strategy
adds value to scientific and policy efforts.
c) Discussion of the value of the research strategies also would benefit from placing the
proposed work in context of complementary efforts conducted outside the
Agency.
d) Identification of the areas of research for which they were designed, i.e., core
research, problem-driven research or both. For MYPs that include both areas,
there should be an explicit formulation of information to indicate which
components of the plan address the regulatory needs of the program offices, and
which address the core research needs.
e) Information in each MYP that indicates how the efforts and deliverables are to be
accomplished, including consideration of the specific measurement issues and
advances that will be addressed by the strategy. From a strategic view, a MYP
should consider whether the implementation process (e.g., in-house staff vs. grant,
contracts, etc.) substantially affects the attainment of deliverables and outcomes.
For instance, the P2NT MYP should provide some information about the staging
of the efforts, number of deliverables expected per year, and how the budget is to
be spread over the four identified topics (Tools, GC&E, ETV and ESM). Without
such information, it is difficult to identify the adequacy and strategic nature of
any MYP.
f) A careful consideration and clear discussion of the relationships between the long-
term goals and the most important areas of scientific uncertainty. The more
explicit these ties, the easier it will be for the reader to see how specific targets
can actually make a significant contribution to expanding our knowledge and
creating the desired outcomes for each planning area. In addition, planners are
encouraged to develop long-term goals that are not open-ended. Annual goals
cannot logically meet long-term goals where desired outcomes are not clearly
articulated.
g) Explicit and unambiguous definition of outcomes that are reflective of concrete and
measurable milestones in the path towards achieving the MYPs long-term goals.
The potential for success in reaching the long-term goals cannot be evaluated
without the formulation of concrete outcomes and the time frame for their
achievement. While EPA has grappled with the question of outcome definition to
some extent, it is not apparent that it has engaged in the kind of strategic thinking
required for this effort. As a start, it might be useful to derive parallels from the
human health model of disease eradication, prevention, or treatment. In this
model, outcomes have very clear and quantitative definitions, and are used for
17
-------
monitoring the progress of research or treatment strategies towards the eventual
long-term goal of a cure or elimination of disease.
Overall the Committee was particularly concerned about how well the documents and the
presentations conveyed information that could be used to answer the fourth and fifth questions.
Based on the information presented, the committee found that there was not a clear link between
completion of the annual performance measures and achievement of the annual performance
goals. There was no clear indication of how annual measures contributed to meeting annual
goals.
Moreover, some of the long-term goals were open-ended; therefore, the Committee
remained concerned that annual goals could not logically lead to meeting the long-term goals. In
addition, the Committee found that there is an apparent missing connection between the
inventory of annual targets and the long-term goals. Consequently, the underlying research
strategy and its potential are not transparent. A way of addressing this missing connection might
be to collect measures and annual goals under target science issues or questions. These target
issues or questions should be directly relevant to the key science questions that each of the long-
term goals seeks to address.
As an example, consider the long-term goal from the Water Quality MYP to improve
approaches and methods to develop criteria to support designated water body uses. A related
key science question is to understand the causal relationships between the varying levels of
stressors and the biological response of aquatic ecosystems. The linkage between the annual
goals and this key science question can be clarified by indicating which goals are intended to
contribute to further understanding of the causal relationships.
The Committee also recommends that the annual performance measures should be
focused on desired outcomes, not outputs. The Committee found that the collection of annual
measures in the plans reviewed were largely outputs. It was difficult to understand how the
collection of outputs would eventually combine and contribute to achieving outcomes that
further annual performance goals or long-term goals. Understandably, this is not an easy task,
and ORD planners may conclude that articulation of the relationship between outputs and
outcomes is itself an important research question.
RSAC understands that developing and formulating concrete outcomes is not a short-
term exercise and may require a significant effort on the part of EPA as well as more integration
across related MYPs. In some sense this exercise is the reverse of the MYP process which
focuses on meeting goals and which is "forward looking" in that EPA formulates a series of
steps in the future that could lead eventually to achieving each goal. The definition of outcomes,
instead, starts with the end in mind. By formulating specific, clear, and very concrete outcomes
that are amenable to some sort of measurement the status after the control technologies, pollution
prevention actions, etc. are put into place can be compared with the prior status (i.e., before the
research was undertaken that led to the control technologies, pollution prevention strategies,
etc.). Once the outcomes are formulated, the strategic thinking process "moves backward" to the
present in order to formulate the series of steps necessary to achieve them.
18
-------
Neither of the two plans (and probably also those yet to be developed) provides clearly
defined outcomes. MYPs have not considered carefully the operational differences between
goals, objectives, performance measures, impact, and outcomes, so that statements under each of
these rubrics could be frequently interchanged. RSAC recommends that the Agency define
clearly in each MYP what is meant by each of these terms. These definitions and their specific
use should be applied consistently across the Agency's documents. RSAC perceives that EPA
has not engaged in the strategic thinking process that would allow it to state the expected
outcomes of their MYPs in a manner that would allow the success or failure of the plans in
achieving the stated goals to be evaluated. The Agency needs to undertake this effort as part of
the development of the multiyear plans.
19
-------
APPENDIX A. ACRONYMNS
APMs Annual Performance Measures
CTP2 Compliance Through Pollution Prevention
EMAP Environmental Monitoring and Assessment Program
ESM Ecosystem Management
LCA Life Cycle Assessment
LCI Life Cycle Inventory
MYP Multi-Year Plan
P2 Pollution Prevention
P2NT Pollution Prevention/New Technology
QA Quality Assurance
QAPP Quality Assurance Project Plan
WQ Water Quality
A-l
-------
5/24/01
APPENDIX B. ORD MULTI-YEAR PLANNING GUIDANCE
Introduction
The Office of Research and Development (ORD) multi-year plans (MYPs) serve as a tool to plan
the direction of our research program and to communicate these plans both within ORD and with
others. This MYP guidance describes the process for developing a MYP and presents a format
to assist the lead authors in developing their assigned MYP. The use of a common format and
approach will aid ORD in comparing different plans, identifying areas for integration across
plans, and effectively communicating our ideas. The lead authors should use their discretion in
tailoring this guidance, based on their knowledge and the input they receive, to best fit their
topic. A list of current MYP topics and lead authors is included in Attachment 1, Lead Authors
and Executive Guidance Groups. The calendar for revision of the draft MYPs is in
Attachment 2.
This guidance includes:
»• Purpose of MYPs
* Relationship of MYPs to other plans and strategies
*• Overview of the MYP process
*• MYP development
*• MYP review and approval
Purpose of MYPs
a) ORD internal planning and
communication
b) Communication with EPA programs
and regions
c) Communication outside EPA
Purpose of the Multi-Year Plans
The purpose of the MYPs is to aid ORD as a
planning and communication tool. Although
the information contained in the MYPs is of
interest to a wide variety of audiences, their
primary purpose is planning and
communication within ORD. The
information in the MYP will also be useful
for communicating our vision within the
Agency, with other research organizations,
and with anyone interested in the direction of ORD research.
Planning
MYPs are a planning tool intended to provide a link between the Agency and ORD's strategic
plans and our annual plans that serve as the basis for ORD's budget request (the relationship
between MYPs and other planning documents is described in the next section). MYPs must
clearly describe what research ORD wants to accomplish to reach the long term goals and
objectives we set for ourselves. By defining the direction and timing of our research program,
the MYPs will assist ORD to define our unique role within the research community, focus
resources on key problems and risk reduction, leverage our work with others, and demonstrate
B-l
-------
MYP Uses
*• Provide the link between the Agency and
ORD Strategic Plans and Research
Strategies to Annual Plans and Laboratory
Implementation Plans
*• Use long term goals to develop annual
performance goals and measures
•• Explain the context and impact of annual
planning decisions
*• Provide a framework for integration across
labs/centers, Agency goals, and with other
research organizations
*• Communicate the direction of ORD's
research program internally and externally
performance.
progress. MYPs will help ORD
managers better anticipate future needs
for facilities and scientific expertise by
illustrating how our research program is
expected to evolve and where resources
are needed in the future.
MYPs provide a framework to conduct
annual planning. MYPs will allow for a
thorough understanding of the context
for annual planning decisions and inform
ORD managers of the potential impacts
on ORD's ability to accomplish its goals
on time. MYPs will also improve
accountability by projecting work
outcomes (annual performance goals)
and outputs (annual performance
measures) in advance, and developing
ways for ORD to measure its
Communication
Since MYPs are intended to describe the logic and sequencing of ORD's research program, they
can help to foster integration through increased awareness and understanding. Integration and
collaboration across disciplines and goals should be increased by providing a mechanism to
understand where similar work is needed in multiple areas, and broadly communicating
possibilities for collaboration at all levels within ORD. MYPs should also improve the
understanding of how individual work efforts play a role in both our overall research effort and
accomplishing our organizational goals.
MYPs will support external communication with our customers and stakeholders in the Agency
and beyond. The stated outputs and outcomes, and the direction and sequencing of ORD's
research efforts will improve ORD's ability to coordinate with EPA program and regional offices
and better support the Agency's science needs. MYPs will help ORD communicate the direction
and purpose of our research program to promote understanding and build support from others,
including EPA's Science Advisory Board and Congress, and help ORD find opportunities to
collaborate with other research organizations.
Relationship of Multi-Year Plans to Other Plans and Strategies
There are several important planning documents that guide ORD's research activities. Figure 1,
Interaction Between Planning Documents, portrays the interaction of several of these key
documents in relation to the MYP. The EPA and ORD Strategic Plans set the direction for ORD
and highlight major areas of emphasis. These documents provide both the Agency's
Government
B-2
-------
National Env.
Research Plan
Fmcy Science
Plans
Research
S trategy
(by to pic)
ti-Year
Planning
Annual
Planning
(RCTs)
NSTC/EPA
Priorities 8
Budget Guidance
Research
Efforts
FTE, $, and
Timelines
0 utpu ts
Lab/Center
Research Plans
Figure 1, Interaction Between Planning Documents
Performance and Results Act (GPRA) goals and ORD's interpretation of the science needed to
support the achievement of these goals. These plans, in conjunction with Agency Science Plans
where available, inform individual topic research strategies and plans that translate this overall
direction into more specific strategic approaches, priorities, and outcomes. These documents do
not generally address resources and are shown outside of the dashed box in Figure 1.
While research strategies describe the approach to addressing scientific questions, research plans
focus on where a lab oratory/center can best use its resources to develop science results. MYPs
link the research strategies and research plans to show how ORD conducts research in an
integrated fashion to achieve our long term goals, annual performance goals and annual
performance measures. The phasing of goals and measures in the MYP reflects available
resources and the sequencing of research activities. The MYP provides a basis for creating
annual plans and a context to understand how decisions made in annual planning impact the
ability of ORD to meet future goals and outcomes. They also improve ORD's understanding of
the impact of Agency priorities and budget guidance, and allow for a more comprehensive
understanding of changes needed to emphasize a new research direction or accelerate an existing
research effort.
-------
Multi-Year Plan Process Overview
Multi-year planning allows ORD to consider
the future strategic direction of the Agency
and determine where scientific discovery
can contribute. ORD will use strategic
planning documents and partner with others
in the Agency to define the major outcomes
that ORD research will support. MYPs are
assumed to cover a window of at least five
to ten years. The GPRA structure of goals
and objectives provides a useful starting
point to organize this effort.
After evaluating the major outcomes set by
the Agency, ORD applies its scientific
expertise to identify major areas of
uncertainty and key science questions that
must be answered. ORD will evaluate the
key science questions and narrow the wide
range of potential choices to those where
ORD chooses to invest resources. The
MYPs should describe why ORD chose the
areas of research selected and identify the
expected customer. The science questions
form the basis for ORD's long term goals (LTGs).
The MYPs will use a flow diagram to depict how annual performance goals (APGs) are logically
sequenced toward accomplishing the LTG. The APGs define major outcomes needed to
progress toward achieving the LTG. APGs are further defined in supporting tables by annual
performance measures (APMs). APMs are lab oratory/center specific activities (or at least
designated to a single responsible laboratory or center) defined as outputs that ORD will produce
to accomplish the APGs.
MYPs consist of three parts:
> A narrative description that
* Introduces multi-year planning and
the purpose of the MYP and its
relationship to the Agency's GPRA
structure
*• Describes the long term goals and
underlying science questions
* Explains the rationale behind the
sequence of the annual performance
goals and measures
*• Addresses capacity/capability issues
when the proposed work requires a
shift in facilities or expertise
*• A graphic depiction of the logical
sequencing and interrelationship of
APGs over time to achieve each LTG
(flow diagram)
* Tables that further describe the APGs
and supporting APMs
MYP Assumptions
*• Annual resources will not exceed the
most recent President's Budget
*• Plans will cover five to ten years
*• MYPs will be updated annually
In order to develop a realistic limit to the
amount of work that can be completed in a
given year, MYPs will assume an annual
budget no greater than the current President's
Budget. Although the total resources will be
constant, the distribution of resources across
ORD laboratories and centers and the relative
emphasis in any particular area is likely to
change over the years in the planning window
and are not expected to be constant.
B-4
-------
It is important that the MYPs demonstrate how research results will be transferred and
communicated to ORD customers. Specific activities should be considered as intermediate
points to consolidate and communicate information. This type of activity may be best included
in the form of an APM.
Once the logic and sequence of work are written in the flow diagrams and supporting tables, a
narrative is completed to explain the work. The narrative is intended to provide an overall
context and describe the thought that went into the MYP. The description in the narrative is
primarily intended to serve as a communication tool to inform others of the intent of the writing
team.
MYPs are intended to be living documents. They will be updated annually and are expected to
change as the state-of-the-science advances to incorporate new thinking and priorities. Updates
will be completed by October 1 of each year in order to have the MYPs available for the annual
planning cycle.
Writing the MYP
The ORD Executive Council (EC) selects
MYP topics, assigns a lead author to be
responsible for developing each MYP, and
will approve the completed product (discussed
later under review and approval). Although
the lead author has the responsibility to
develop the assigned MYP, he or she is not
expected to do the job alone. The lead author
should assemble a writing team to help
throughout the MYP development process.
The writing team will provide the information
needed to complete the document, organize
the presentation of material, and write/prepare
assigned materials. The lead author should
consider knowledgeable staff from across
EPA including ORD staff (at all levels), and
staff from EPA program and regional offices
as appropriate. Writing team members should
be knowledgeable in various aspects of the
MYP topic (e.g., needed and ongoing research
within and outside of EPA, and existing and
proposed regulations and Agency policy).
One of the first tasks of the writing team is to
review existing materials related to the MYP
topic. These include, but are not limited to,
GPRA goals and objectives; existing EPA and
ORD strategic plans; prior annual
MYP Roles and Responsibilities
> ORD Executive Council (EC):
Approves MYP guidance, assigns
topics and lead authors, approves
completed MYPs
» ORD Science Council (SC):
Provides scientific analysis of
MYPs and recommends approval to
the Executive Council
> Lead Author: Selected by EC to
develop an assigned MYP
*• Writing Team: Representatives from
ORD and across EPA selected by
the lead author to assist in
preparation of the MYP
*• Executive Guidance Group (EGG):
Consists of EC/SC members and
provides senior ORD guidance to
the writing team
> Office of Science Policy (OSP):
Develops MYP guidance and
coordinates development and
approval processes
B-5
-------
performance plans; relevant existing research strategies and research plans; lab oratory/center
specific annual plans; and strategic plans and direction from program and regional offices.
The MYP does not need to show the same level of detail for new or emerging areas, but the
MYP should give the best estimate of how work is envisioned to proceed given the available
knowledge.
Executive Guidance Groups
Executive Guidance Groups (EGGs) will be assigned to provide senior level guidance to the
writing teams. The EGGs will each consist of approximately three to five volunteers from the
EC and Science Council (SC) with expertise in the MYP subject. The EGGs will meet
periodically with the writing team throughout writing and revision of the MYPs to ensure the
plan is focused on high priority science within the subject area, provide guidance on resource
shifts, and assess ORD capacity and capabilities. A list of current EGG members is in
Attachment 1.
Steps to Complete the Flow Diagram
*• Review existing material
*• Develop science questions and long
term goals
* Develop and sequence annual
performance goals
Flow Diagram and Tables
The majority of time and effort required to
complete the MYP will be spent determining
research priorities and describing the logical
progression of work to accomplish them. The
lead author and writing team must narrow the
focus of the plan to reduce redundancy with
research in other MYPs and describe why the
work is necessary and unique. The MYPs
should give compelling reasons why the areas
of research were selected instead of other possible choices. A flow diagram (see Figure 2, Flow
Diagram, and the example in Attachment 3) will be created to display the projected outcomes
and actions needed to reach each Long Term Goal (LTG). The flow diagram presents, in a
concise format, the annual performance goals (APGs) and their interrelationship across time.
Since the flow diagram will illustrate how the APGs relate to the LTGs, a separate flow diagram
will likely be needed for each LTG. The APGs should be connected to highlight the importance
of the APG sequence and show if there is a critical path to achieve the LTG.
APGs from other MYPs needed to accomplish the LTG should also be included in the flow
diagram, requiring coordination between writing teams. The diagram should clearly reference
the other MYP as the source of the APG. It should also be noted where APGs in the MYP are
included in another MYP as critical to an LTG outside the plan. The cross linkage of APGs in
the MYPs will aid in integration of research across ORD and explicitly identify dependency on
other work.
B-6
-------
FY03
FY04
FY05
FY06
FY07
FY08
I 1
I APGX
1 From XXXMYPr
Long Term Goal 1
Figure 2, Flow Diagram
The logic flow should include periodic assessment of the state-of-the-science at intermediate
points to draw work together and provide information to the Program and Regional Offices (and
ourselves). APGs and APMs should demonstrate where information will be brought together for
communication to ORD customers.
Development of Long-Term Goals
The MYP describes the steps needed to
achieve the LTGs and how they support the
Agency's GPRA goals and objectives. As
such, the foundation of the flow diagram is
the development of the LTGs. The writing
team must first consider where science can
contribute to achieving an Agency GPRA
goal or objective. The writing team will
then identify the key science questions to be
addressed. Writing teams should consider
the questions of scientific merit,
accountability, and capability and capacity
presented under "Review and Approval" on page 14 of this guidance when narrowing the
science questions and developing the LTG. The science questions will likely cover a broad
Long-Term Goals
* Include input from a wide range of
sources
*• Narrow in scope, not overly broad
*• Consider three to five LTGs per MYP
*• APGs show a logical progression
toward LTG
> Only LTGs that will be accomplished
within planning window should have a
date
B-7
-------
range of activities, and the writing team will need to consider what are the key questions that
ORD science will need to address.
Once the science questions are focused, the writing team will create LTGs that define the
research goals to be achieved from the proposed research and how the results of the research will
support the mission of ORD and the Agency. The LTGs should be well thought out and include
input from a wide range of sources. They should be narrow in scope to show the specific
contribution of the research, and not overly broad such that any related work is applicable (i.e.,
"improve the science" in a particular area). The LTGs should be written so the APGs show a
logical progression toward the stated research goal.
The process of developing the flow diagram is likely to be an iterative process, and the LTG may
be clarified as the writing team begins to define the APGs and examine how they support the
LTG. Between three and five LTGs per MYP should be considered as a starting point, but may
vary between plans. The LTGs that will be accomplished within the planning window should
have a date. However, it is not necessary to force a date into the LTG if it falls outside the
planning window.
Development of Annual Performance Goals
Once the LTGs are established, the steps needed to reach each LTG are determined and
formulated into APGs. Developing and
sequencing the APGs is one of the most
difficult and time-consuming aspects of
creating the MYP. Key outcomes
required to achieve the LTG and the
research needed to answer the science
questions must be described. Available
resources have to be considered and
activities phased to project a schedule that
will not require annual resources greater
than the latest President's budget request.
Annual Performance Goals
*• Identify key outcomes to achieve the LTG
*• Consider resources and phased activities
to project a schedule
> Align in a logical sequence to address
knowledge gaps in early stages
*• Design flow diagram to show the
relationship between APGs and how they
flow toward achieving the LTG
The APGs should be aligned in a logical
sequence so that knowledge gaps are
addressed in the early stages for use in
future research. The flow diagram should be designed to show the relationship between APGs
and how they flow toward achieving the LTG.
The flow diagram should include APGs from other MYPs, or equivalent level outcomes from
other organizations, if the APG is critical to completing the LTG. If the LTG does not depend
on the APG from another source, or if the APG is relevant but not on the critical path, they
should not be included. Work in other MYPs or from other organizations should be included at
the highest level possible and referenced (i.e., if an APG from another MYP is included, the
supporting APMs should not be cited as well). It should be noted in the narrative that work of
other research organizations on the critical path is a vulnerability, since ORD does not control
the level of effort devoted to these activities.
-------
Each APG should be further described through a table (see Figure 3, APG/APM Table and an
example for one LTG in Attachment 4) showing the APMs that will contribute to achieving the
APG. The APGs should be written so that the APMs show a logical progression toward the
stated outcome in the APG.
Development of Annual Performance
Measures
The next step is to determine how ORD will
measure the accomplishment of the APGs and
develop APMs. The APMs may be outputs or
outcomes, and the completion of all the
proposed APMs should demonstrate that the
APG is achieved. APMs define what specific
actions will be completed to accomplish the
APG. Each APM should be assigned to an
APG, be completed before the APG is
considered complete, be assigned a fiscal year
for completion, and have a designated
laboratory or center responsible for the APM. APMs should be included that identify specific
activities or products intended to communicate results at key points.
Annual Performance Measures
*• May be outputs or outcomes
*• Assigned to an APG
* Include the fiscal year of completion
* Must be completed before the APG
is completed
* Are assigned to a single lead
laboratory or center who is
responsible for the APM
ANNUAL PERFORMANCE GOALS AND MEASURES
APG 1 - TITLE
APM
APM
APM
APM
APM TITLE
APM TITLE
APM TITLE - XXX MYP, APG X, 2005
APM TITLE
APG 2 - TITLE
APM
APM
APM
APM TITLE
APM TITLE
APM TITLE
YEAR
2003
2001
2002
2002
2003
2004
2002
2002
2003
LAB/
CENTER
ORD
LAB/CENTER
LAB/CENTER
LAB/CENTER
LAB/CENTER
ORD
LAB/CENTER
LAB/CENTER
LAB/CENTER
Figures, APG/APM Table
APMs from other MYPs or organizations can be included on the table if they are critical to
accomplishing the APG. Again, all work from other sources should be included at the highest
level possible and the source referenced for additional detail. APMs from other sources should
be indicated in italics.
B-9
-------
Narrative Development
Although the majority of the work needed to
create the MYP is spent developing the flow
diagram and its supporting tables, a narrative
description is needed to orient and inform
readers of the writing team's intent. The
narrative will provide background, and describe
both why the research was selected and the logic
behind the sequence of the work. The narrative
will also include some quantitative information
to give the reader an idea of the magnitude of
the research effort described, as well as some
trend information to inform the reader of where
the relative effort will be as the program matures.
The narrative should be concise. Approximately
fifteen pages should be used as a guide.
Background
*• Describes Agency priorities and
regulatory program
*• Introduces science questions
*• Identifies focus of ORD contribution
* Describes state-of-the-science and
work of others
includes limited quantitative ana
trend information
Introduction
*• Describes MYP Process
*• Describes scope of MYP
*• Identifies primary customers for
work
* Provides size of annual resources to
nearest $1 million
Introduction
The introduction provides the reader with an
overview of the scope of the MYP. The
introduction should briefly describe what an
MYP is, and reference any key documents, such
as research strategies, that the MYP builds upon.
The introduction should include a broad
description of the scope covered by the MYP
and the primary customer for the research
described. The introduction should also state
the size of the total annual program rounded to
the nearest million dollars to inform the reader of the magnitude of the proposed program (e.g.,
"the research described in this MYP assumes annual resources of approximately $20 million.")
Background
The background section should familiarize the reader with the goals and objectives of the
Agency and the resulting science questions that the ORD research will address. This should
include a description of the major areas of uncertainty and opportunities for risk reduction. The
background should note key regulatory programs and any cross-agency workgroups, agreements,
or significant work by other organizations that influence ORD's decision to focus work in a
particular area. The background should also briefly inform the reader of the current state of the
science and ORD's position in the research community relative to the MYP topic. Lastly, the
background should inform the reader of the role of the MYP relative to other ORD planning
documents, such as research strategies and plans, similar to the discussion of Figure 1 on pages
2 and 3 of this guidance.
B-10
-------
Narrative Description of Long Term
Goals
*• Introduces LTGs and rationale for
selecting them
*• Introduces flow diagram and tables
*• Identifies changes in emphasis
between LTGs over time
I
t
Present the Long Term Goals
After evaluating the GPRA goals and objectives
set by the Agency, ORD will narrow the wide
range of potential choices to those where ORD
chooses to invest resources. The narrative should
provide an overview to introduce the LTGs that
will be addressed by the MYP and the science
questions that form their basis. The narrative
should convey why ORD chose the areas of
research selected and why ORD can make the
greatest contribution in these particular areas.
The narrative should also introduce the flow
diagram and supporting tables and describe how they support the discussion of the specific LTGs
in the next section. A table should be included to show the relative emphasis changes over time
among LTGs. The level of effort for each LTG should be described in general terms on the
table. Simple descriptions should be used such as increasing, level, or decreasing along with a
supporting rationale in the narrative text to explain the change in emphasis (see Figure 4, Table
of Evolving LTG Emphases).
The narrative should discuss potential changes needed in ORD capability and/or capacity to
accomplish the LTGs. The plan should describe how potential shortfalls in capability or
capacity will prevent ORD from accomplishing the work envisioned by the plan. Examples of
capability and capacity issues include lack of technical expertise or lack of appropriate facilities.
Area
LTG1
LTG 2
LTGS
LTG 4
LTGS
Emphasis in MYP Planning
Window
Decreasing
Level
Level and then decreasing
Increasing
Increasing and then level
Figure 4, Table of Evolving LTG Emphases
B-ll
-------
Description of the Flow Diagrams
After describing the LTGs that will be addressed in the plan and the rationale for selecting the
LTGs, each LTG should be presented with the supporting logic used to develop the flow
diagram. The narrative should discuss the rationale used to sequence the APGs and discuss time
dependent APGs that define a critical path (i.e., successful completion of the LTG is dependent
on timely completion of the APG). Important APMs may also be discussed if they substantially
add to the information presented. The discussion should include work from other MYPs or other
research organizations (addressing only those included in the flow diagram) on the critical path
and potential vulnerabilities. The narrative should also discuss how accomplishing the LTG
relates to the science questions and the Agency's goals and objectives.
The narrative should highlight plans to communicate research results to ORE) customers. This
description should include interim products, evaluations of the state-of-the-science, and
technology transfer. The writing team should consider adding a separate section or attachment
that further describes the overall communication plan.
Flow Diagram Description
* Describes logic used to
sequence APGs
*• Describes important APMs
where needed
*• Discusses communication of
research results
Appendix Describing Potential Additional Work
The writing teams may include an appendix to describe how the plan would be modified if
additional resources are available. The proposed work in the appendix should be limited to a
maximum of 20 percent of the current resource base. Proposed work should be considered to
address (1) research gaps, (2) acceleration of the current program if necessary, (3) the next
logical piece of work, and (4) work that was not considered in the original scope due to resource
constraints.
The appendix should clearly describe new or modified LTGs and the related science questions.
The appendix is not anticipated to be at the same level of detail as the MYP, but should convey
an outline of how the additional or accelerated work would affect the flow diagram.
Review and Approval
It is expected that writing teams will coordinate with the EGGs throughout the
development/revision process to receive input from senior ORD leadership and streamline
subsequent reviews. Completed/updated MYPs will be submitted to OSP who will coordinate
B-12
-------
review by the full SC. Although individual SC members will participate throughout the writing
process as EGG members, the full SC will have enough members who did not participate on an
individual MYP EGG to give each MYP a critical review prior to referral to the EC for approval.
The SC will review the plan for (1) the scientific merit of the MYP, (2) accountability, and (3)
ORD capability/capacity to accomplish the proposed work within resource assumptions.
Specific review questions for review by the SC include:
Scientific Merit:
1. Do the long term goals support the priority needs of the program and regional offices?
2. Do the science questions address the most important areas of scientific uncertainty in the
subject area?
3. Is the proposed scope of work consistent with the current state-of-the-science?
Accountability:
1. Would accomplishing the long term goals enable ORD to accomplish the GPRA objective?
2. Would accomplishing the APGs allow ORD to achieve the long term goals and answer the
science questions?
3. Would accomplishing the APMs clearly demonstrate the APG was attained?
4. Are the APGs and APMs measurable outcomes and outputs, respectfully (APMs may be
outcomes or outputs, but generally represent outputs)?
Capability and Capacity:
1. Does ORD have the capacity and capability (e.g., skill mix, facilities) to accomplish the
proposed scope of work? If not, can ORD realistically obtain the capacity or capability
within the projected time frame?
2. Is the proposed scope of work reasonable within the resource assumption (i.e., no greater
than the current President's Budget or an agreed upon alternative)?
The SC will make a recommendation to the EC regarding final approval of the MYPs, and the
EC will approve final versions of the MYPs. ORD is considering options and requirements for
external peer review of the MYPs after approval by the EC. This guidance will be appended
when further information is available regarding potential external peer review.
B-13
------- |