September 28, 1995
EPA-SAB-RSAC-COM-95-003
Honorable Carol Browner
Administrator
Environmental Protection Agency
401 M Street, SW
Washington, DC 20460
RE: An SAB Commentary on Research Planning and Budgets
Dear Ms. Browner:
The Research Strategies Advisory Committee (RSAC) of the Science Advisory
Board (SAB) met with representatives of the Office of Research and Development
(ORD) on March 17, 1995 and June 15-16, 1995 in Washington, DC as part of its
ongoing review of the process for planning and developing budget priorities for
research. During these meetings, RSAC received briefings from Agency managers on
the status of the research planning process, projections for the FY 1997 budget, and
ORD's research priorities. From these briefings and discussions with Dr. Robert
Huggett, Assistant Administrator for the Office of Research and Development and his
Deputies for Science and Management (Mr. Joseph Alexander and Mr. Henry Longest,
respectively), RSAC learned that ORD has not yet completed a strategic plan or written
guidance on the research planning process, but rather these are works in progress.
Therefore, the SAB was not given a charge and did not receive specific documents for
a formal review.
The purpose of this commentary is to offer interim advice to ORD and to the
Agency programs on the concepts that were presented for developing: a) a strategic
plan for ORD research, b) research budget priorities, and c) a strategy for using sound
science within the Agency. We also propose an agenda for on-going future RSAC
reviews on science strategy for the Agency including ORD.
1. STRATEGIC PLANNING FOR ORD RESEARCH
The new directions that ORD is taking are very appropriate and incorporate the
continuing recommendations from the SAB.1 Although there have been few concrete
results, and we have received little information about the operation of this process,
The following are examples of such advice: SAB. 1988. Future Risk: Research Strategies for the 1990s. SAB-EC-88-040; SAB.
1990. Reducing Risk: Setting Priorities and Strategies for Environmental Protection. SAB-EC-90-021; SAB. 1994. Review of the FY
1995 Presidential Budget Request for the Office of Research and Development. EPA-SAB-RSAC-LTR-94-008; and SAB. 1994. Review
of Mitre Corp. draft report on the EPA Laboratory Study. EPA-SAB-RSAC-94-015.
-------
there are several positive signs of improvement. The on-going strategic planning is
based on a risk assessment paradigm; the grants program (both extramural and
intramural) will be peer reviewed; a significant portion of the research effort will be
long-term; and the reorganization of the laboratories is intended to allow management
of the research to be directed within risk assessment clusters. ORD's realization of the
need for a budget strategy and their efforts to develop one are also positive. The
Committee appreciates the opportunity to examine the development of the research
planning and budgeting processes early in their evolution. ORD seems to be making
good and appropriate progress on those projects.
Given the many changes occurring within ORD and the austere projections for
research budgets, the Agency will be forced to make difficult choices. Therefore, it is
especially important that ORD develop criteria to help prioritize the risks and
uncertainties associated with research topics for the strategic planning process.
The Agency must make a long-term commitment to establish, maintain, and
refine a planning process for developing research priorities and budgets. Beginning
with the development of a strategic plan to address the needs of the Agency, the
process should drive ORD toward a vision of its continuing and future mission. The
planning process must be supported by: a) guidance that describes the roles and
responsibilities of all of the participants; b) a management information system that is
accessible to stakeholders and includes appropriate levels of aggregation and detail to
track performance and cost data over the course of the project; c) a mechanism for
feedback and adjustment of the process and the research activities; and d) a
communication plan to inform the Agency and the broader scientific community about
research findings, their use and limitations, and research needs of the Agency. In the
future the Agency needs to link the strategic plan for research to the budget request to
justify how it will implement the strategic plan.
1.1 Considerations for the Strategic Plan for ORD
As part of the development of a strategic plan, ORD should undertake a self-
assessment with respect to its areas of research strengths and weaknesses and critical
gaps in expertise. Based on this, plans should be made to build strengths in key areas
and phase out areas that are tangential to the strategic plan in an orderly fashion.
Consideration should be given to bringing into ORD programs that are being cut from
other federal Programs (e.g., Ocean research from the National Oceanic and
Atmospheric Administration). A diverse and balanced portfolio of research
mechanisms, including intramural and extramural grants, is an important goal to include
in the strategy.
ORD should develop criteria to prioritize issues in the strategic planning
process. As an example, the criteria for the ecological risk assessments are fairly well
-------
developed and can be found in the several documents produced by the EPA Risk
Assessment Forum. Criteria for ranking the human health risks from a variety of
stressors, including chemicals and microbial organisms, should also be developed and
refined.
The RSAC commends ORD's efforts to shift from media-oriented categories to
risk oriented strategies for setting research priorities. RSAC cautions that categories
such as multimedia" are too broad to be useful. While ORD may not yet have identified
categories which RSAC feels are the most appropriate, this shift represents an
important step toward articulating rational goals for Agency research. Additional
considerations for the strategic planning process are provided in Appendix A.
1.2 Agency Resource Allocations to Research and Risk Assessment
It is increasingly apparent that risk-based decision-making is going to be an
important part of the fabric of the Agency during the next decade of operation. Risk-
based decision making requires creation and utilization of sound risk assessment
processes that make use of the best available scientific information. Both process and
information are critical to success.
The RSAC is concerned that EPA may not be appropriately positioned to meet
this important challenge. For example, the Agency's staffing of the National Center for
Environmental Assessment may not be adequate in number or breadth and depth of
experience in critical areas.2 If it has not already been done, the RSAC recommends
that the Agency conduct a critical assessment of its current staff competencies in risk
assessment and the projected needs for this center over the next decade. The RSAC
would be pleased to review and comment on such an assessment.
The RSAC is also concerned with the level of current resources (staff and funds
available for intramural and extramural use) available to the Agency for developing the
scientific information base needed to reduce uncertainty in assessing human and
ecological risks. It is important to remember that the quality of the risk assessments
cannot be any better than the scientific data used.
Based on the foregoing concerns, the RSAC urges the Agency to evaluate
critically the extent to which an appropriate portion of its total resources is being
allocated to its research and risk assessment activities. It appears that in recent years
the Agency has been using about 15% of its total budget to carry out its ORD
operations (See also SAB review of the 95 Budget in EPA-SAB-RSAC-LTR-94-008). In
view of the importance of good science to good environmental decisions, the complex
Cancer biology, respiratory disease, aquatic and terrestrial ecology, environmental chemistry, biostatistics, forensic research,
teratology, and social behavior/toxicology.
-------
environmental concerns we face (e.g., habitat loss, global change, human behavior and
exposure patterns, and a shift from compliance to pollution prevention), and the
increased costs of conducting research, the Agency should consider whether the split
between research and program offices interests is appropriate.
2. RESEARCH BUDGET PRIORITIES
2.1 Research Projects
The Committee supports ORD's intent to apply explicit criteria to select research
projects for funding. When faced with budget decreases, ORD should not routinely
make cuts across the board but should consider whether entire projects can be deleted
to maintain excellence in more critical programs. These criteria should be related to
the nature of the risks involved and may resemble those in Appendix B, which this
Committee is currently developing.
However, we are not convinced that the deletion of indoor air quality research
from Congressional mark-up of the FY 1996 ORD budget is wise. Most of the
population spends 90% (on average) of its time indoors. Thus, indoor sources and the
penetration of outdoor pollutants into indoor environments are major determinants of
human exposures to many pollutants. A sound scientific understanding of indoor air
quality and indoor air exposures is needed if we are to fully understand the health risks
of environmental pollutants and develop effective control strategies and policies for
reducing unacceptable risks. ORD and Congress should include these considerations
in their eventual decision.
We are also concerned that research at National Institute for Environmental
Health Science and Agency for Toxic Substances and Disease Registry that is funded
by pass through of EPA's research budget is not considered in the Agency planning
and priority setting process. EPA should avoid duplicating these efforts.
2.2 Grant Process
The RSAC applauds the Agency efforts to use grants as one approach to reach
its objectives. However, the RSAC is concerned that the grant process imposes a
substantial burden on the applicants relative to the likelihood of funding. This can be
illustrated by considering the recent "Reducing Uncertainty" Requests for Applications
(RFA). The RFA attracted 669 applications. If it is assumed that preparation of each
proposal took approximately 0.1 of a person year @$100,000/year, the cost per
application was $10,000 or a total aggregate cost of $6,690,000. This is very close to
the anticipated amount of the 54 first year awards-$8,500,000. Consideration of these
numbers suggests that EPA should carefully consider less costly approaches to
-------
soliciting grant applications. For example, it might consider a two-stage process,
modeled after the National Science Foundation (NSF). The first stage might consist of
an abbreviated proposal of perhaps three or four pages to be used to identify
objectively and openly, applicants who would be invited or encouraged to submit a
comprehensive proposal.
The RSAC members are also concerned about the size of grant awards. Some
of the RFAs have caps on the awards of $150,000 for the first year. In today's
environment, that amount represents only about one person year of effort, which may
be inadequate to tackle large issues requiring multi-disciplinary approaches. This
constraint may jeopardize the likelihood of the Grant program producing substantial
research results that will be useful to the Agency in fulfilling its mission. The size of
grants should be adequate to cover the subject. ORD should maintain some flexibility
in funding grants of different sizes (including levels up to $300,000 in special cases)
based on their scope, goals, and degree of difficulty. Some researchers can
accomplish valuable work with less than $100,000, whereas other complex projects will
require multidisciplinary approaches and larger sums, but clearly one size does not fit
all.
The RSAC understands that the Agency is considering alternative approaches to
grants and in-house research to achieve its mission. The RSAC encourages the EPA
to continue to explore the use of a range of approaches-grants, cooperative
agreements, centers, consortia, scientific exchanges, and contracts. A mixed approach
increases the importance of having well-documented policies and procedures that spell
out how each approach is used and the roles of EPA staff, federal employees, and
other personnel associated with the grant, contract, or cooperative agreement.
2.3 Distribution of Research Funding
While the Committee supports ORD's decision to expand its program of grants to
academic and other non-profit institutions, it recommends that ORD be more explicit
about the rationale and evaluation process that governs the split of its total budget
among intramural activities, contract research, grants, and cooperative agreements.
Presumably different goals drive preferences for one type of funding arrangement over
the others, and changes in EPA's research strategy could lead to changes in the
distribution of research funding. For example, intramural funding may optimize
continuity and Agency control, contracts may provide the most targeted research,
grants may encourage the most innovative approaches, and cooperative agreements
may provide the best opportunity for cross-fertilization between Agency and non-
Agency scientists. ORD should have a clear rationale for its funding distribution in
order to minimize criticism from Congress, the scientific community at large, regulated
parties, and EPA's own staff.
-------
It is essential for ORD to develop a balanced portfolio of research "investments"
including: a) Long, intermediate, and short term projects as well as a certain
percentage (5-10%) of high risk but high benefit (if successful) projects. The high risk
topics might be derived from suggestions by foresight activities suggested by the SAB's
Environmental Futures Project3; b) Single investigator and multi-investigator projects as
required by the research topic; c) Intramural and extramural consortiums, cooperative
agreements, interagency agreements, scientific exchanges, etc.; d) Research that
ranges from more basic to applied as needed to meet EPA needs; e) Integrated
assessments of human and ecological exposures, effects, and risk reduction options.
2.4 Other Issues
The recent congressional proposal to eliminate the funding of ORD's fellowship
education program is troublesome, because the enormous response to the fellowship
program testifies to the need for such support. However, EPA should be careful to
target its fellowships toward specialities with known deficiencies in students that the
Agency is likely to need in the future.
2.5 Budget Information
Budget presentations must be organized by task so that the total effort in dollars
and FTE's, can be evaluated and associated with research priorities. Budget
information should be available to meet the needs of all stakeholders. We understand
that some of the complexity of the accounts and classification of monies is the result of
the Congressional appropriations requirements, and that such requirements must be
accommodated in a budget management system. However, all costs should be
included and the information should reflect the total budget of ORD. Anything less is
both misleading and frustrating to any meaningful review.
3. AGENCY STRATEGIC PLANNING FOR SCIENCE
The research planning and prioritization process for the Agency being initiated
by ORD is a very good start, and ORD is the most knowledgeable unit for science
within the Agency. OPPE appears to be attempting to perform a similar function
through the National Goals Project with a staff of predominantly economists and
planners. There should be one integrated strategic planning responsibility into which
all programs and regions transmit their inputs. ORD should manage that integrated
strategic planning process for science because this is the group in the best position to
evaluate scientific quality and changing research needs.
-------
3.1 Consider Opportunities for Synergy and Recombination with other Federal
Agencies
The "reinventing" of government to make it more responsive, efficient, and of
higher quality should not be limited to the boundaries of EPA. The National Oceanic
and Atmospheric Administration (NOAA) has several research programs focused
entirely on environmental research in the coastal and marine ecosystems: the National
Undersea Research Program (NURP) and the Sea Grant Program are two very good
examples. RSAC urges ORD to examine these programs and consider how they may
complement the ORD "place-based" research design in order to better integrate
activities. The current redundancy in coastal marine research between EPA and NOAA
supports this recommendation.
3.2 Research Program Evaluation Criteria
Independently, the Committee has been discussing criteria that might be used
by ORD when evaluating specific proposals for research projects.3 Some of the same
ideas apply to broader research decision units4 (called here "research programs")
which support a variety of projects related to a particular issue and are therefore
offered below for ORD's consideration and more detailed discussion is appended to
this letter.
Possible Evaluation Criteria (in no particular order)
Some criteria that might be used include:
a) The research program addresses a significant issue in environmental
science or technology, independent of its specific relevance to an Agency
program need;
b) The issue addressed entails substantial known or suspected risks to
human health or the environment;
c) Substantial uncertainties remain about the magnitude or character of the
risks;
3
These criteria are still under development. The Committee hopes to finalize our discussions and transmit the results to ORD in
the near future.
4
Decision unit is a term -used by EPA to describe the level in the research program organization where decisions are made about
the scope, funding, and staffing of a research projects.
-------
d) The issue is relevant to EPA regulatory program or regional office needs,
more broadly to EPA's mission to protect human health and the
environment;
e) The research program addresses a potentially significant future
environmental problem that may not yet have attracted legislative or
regulatory attention;
f) The results of the research might address several Agency needs;
g) The research program is cost-effective in terms of the utility of its results
relative to its cost; and
h) The research program can produce useable results in a reasonable time.
Note that these criteria should be quantified or weighted, and combined in a
manner designed to achieve the overall goals of EPA's research strategy.
The members of RSAC encourage ORD to focus on long term goals. Credible
science is built on long term commitment. As a Committee, RSAC wants to focus on
the long-term research needs and the changes in the research budget to address
emerging issues.
We look forward to reviewing the ORD research priorities and budgets for
addressing the research. In addition, we look forward to reviewing the agency-wide
strategy that you have asked Dr. Huggett to develop. In reviewing these documents
and in your response to our commentary, we will be particularly interested in how
research will be related to the Agency's strategic plan and how Program Offices
establish their priorities for research and scientific information. Thank you for the
opportunity to offer these suggestions.
Sincerely yours.
Dr. Genevieve Matanoski, Chair
Executive Committee
Science Advisory Board
Dr. Margaret Kripke, Chair
Research Strategies Advisory Committee
8
-------
Appendix A. Principles for Strategic Planning
In Total Quality Management the customer is defined as the recipient of the
product or service. When this concept is applied within an organization such as the
EPA, the internal customer is the person (or program, etc.) which receives an output
from another internal group and is expected to use that output to produce and deliver
goods, which may in turn be handed to someone internal or external to the
organization.
In this case, the Strategic Plan is an output produced by EPA senior planning
executives to be used by all persons within the organization as direction in making
program decisions and conducting appropriate research. It is at the research level
where the technical information in the Strategic Plan is implemented. Therefore, in
addition to the EPA Program Offices and Regional Offices, another direct customer of
the plan is the researcher at the bench who must carry out the research and have
access to the plan.
The researcher must understand the intent of the Strategic Plan in order to use it
efficiently. A full understanding (and buy-in) can only be gained if researchers are
allowed to be an integral part of the planning process. All too often strategic plans sit
on shelves and gather dust because the group of intended users were not involved in
the process of developing it. This applies particularly to the researchers, primarily
because they are the true experts of their field, and their willingness to simply follow an
executive decision can be less than that of less highly, trained workers.
According to TQM principles, all customers should be given the opportunity to
participate in the developing requirements with their suppliers. The researchers most
likely have input, questions, and generally valuable insight into the details of the plan
and should be asked to discuss them. Very likely their "requirements' will fall into the
following categories:
a) plan information (e.g., specific areas of research needed to understand a
particular field; areas that are lacking the expertise needed to be
successful; information they have gotten directly from their customers;
etc.)
b) plan content (e.g., objectives that are clear and measurable; a vision that
continues beyond the year 2000; identify those competencies that you
believe will be in greater demand in the future; identify areas that will be
phased out in the next ten years; etc.)
A-1
-------
c) plan logistics (e.g., set a due date so implementation plans can be
developed by the required date; provide electronic copy so appropriate
sections can be e-mailed to associates; etc.)
The RSAC recommends that the strategic planning process fully involve the
researchers in the process. Some suggestions on ways to include the researcher in
the process including the following:
a) Hold a requirement gathering meeting at the beginning of the process.
Develop a goal for the strategic planning process. Invite a random
sample of researchers, or open the invitation to anyone interested.
Depending on the number of responses, more than one meeting may be
required.
b) Identify areas of future growth (and areas to potentially diminish) and ask
the researchers to provide the planning team with white papers, small
presentations, or general information about their opinions in these areas
and benchmarks for evaluating the research direction.
c) Hold open forums every few weeks to constantly keep information flowing
both ways on the progress of the plan.
d) Ask one or two researchers to become members of the core planning
team.
e) Ask several researchers to become technical advisors to the core
planning team.
A-2
-------
Appendix B. Criteria for Selecting High Priority Research Programs
a) Agency relevance. The Committee agrees that a high-priority research
decision unit should be relevant to EPA program offices and regional
needs, but cautions that decision units relevant to EPA's broader mission
to protect human health and the environment should also be highly rated.
b) Scientific relevance. Generally speaking, high-priority research decision
units should address a significant problem in environmental science or
technology (Agency relevance is ranked under d below). The hypotheses
to be tested or problems solved should be clearly articulated and the
research plan should be well designed to answer the research questions
posed.
c) Degree of Risk. The Committee agrees with ORD that the known or
suspected magnitude of risk associated with the issue should be
important in determining its priority for research, recognizing that "risk" is
a broad term for the probability and magnitude of consequences on
health, ecological resources, economic resources, and amenities. The
degree of reversibility of the impacts, the time frame of their likely
occurrence, and the uniqueness of the resources affected should also be
considered.
d) Uncertainty of the Risks. The Committee also agrees that uncertainties
and lack of knowledge regarding the issue are important criteria for
selecting decision units, particularly when attempting rule making.
Understanding and managing the uncertainties may be as important or
more important than reducing the uncertainties but all are important for a
high priority decision units. The sensitivity of Agency decisions to the
potential reductions of uncertainties offered by the decision unit should be
considered.
e) The Committee also supports ORD's intent to rate highly decision units
that address potentially significant future environmental problems that
may not yet have attracted legislative or regulatory attention. Such
exploratory research may not meet all the criteria for program-oriented
research yet may contribute importantly to the Agency's overall ability to
protect health and the environment.
f) Additional credit should be given to decision units that might impact on
several Agency needs. These decision units will generally fall in the
B-1
-------
category of "multimedia", meaning programs not specific to one of the
major EPA program areas.
g) Attention should be given to the cost-effectiveness of the decision units,
i.e., the relationship of anticipated improvements in knowledge about
important environmental issues in comparison with the costs of the
research and of implementing the resulting recommendations.
h) Some attention should also be given to the rapidity with which a decision
unit can produce usable results. This criterion should not be used,
however, to reward quick and dirty projects or to reject longer-term
research that receives high scores on the other criteria discussed above.
ORD should be careful how it applies the criteria. In some cases, two or more
criteria must all be met for the decision unit to deserve high priority, while in other
cases, a high score on any of several criteria might be sufficient. For example, a
program that addresses a problem that is already understood with a high degree of
certainty should not be a strong candidate for funding of research, no matter how high
its related risks. On the other hand, a program does not need to be both future-
oriented and widely applicable across regulatory program areas to be suitable for
funding.
B-2
-------
US ENVIRONMENTAL PROTECTION AGENCY
SCIENCE ADVISORY BOARD
RESEARCH STRATEGIES ADVISORY COMMITTEE
CHAIR
Dr. Margaret Kripke, Chair, Department of Immunology, M.D. Anderson Cancer Center, Houston, TX
MEMBERS
Dr. Judy A. Bean, Professor and Director of Biostatistics, University of Miami, Miami, FL
Dr. Stephen L. Brown, R2C2 Risks of Radiation and Chemical Compounds, Oakland, CA
Dr. Edwin L. Cooper, Department of Anatomy and Cell Biology, School of Medicine, UCLA, Los Angeles,
CA
Dr. Theo Colborn, Director, Wildlife and Contaminants Project, World Wildlife Fund, Washington, DC
Dr. Kenny S. Crump, Ruston, LA
Dr. Joan M. Daisey, Indoor Environment Program, Lawrence Berkeley Laboratory, Berkeley, CA
Dr. Kenneth Dickson, Director, Institute of Applied Science, University of North Texas, Denton, TX
Dr. Roger O. McClellan, President, Chemical Industry Institute of Toxicology, Research Triangle Park, NC
Dr. Paulette Middleton, Director, Science & Policy Associates, Inc., Boulder, CO
Dr. W. Randall Seeker, Senior Vice President, Energy & Environmental Research Corp., Irvine, CA
CONSULTANTS
Dr. William E. Cooper, Director, Institute Environmental Toxicology, Michigan State University, East
Lansing, MI
Dr. Virginia Dale, Environmental Sciences Division, Oak Ridge National Laboratory, Oak Ridge, TN
Dr. Richard Kimerle, Monsanto Corporation, St. Louis, MO
SCIENCE ADVISORY BOARD STAFF
Dr. Edward S. Bender, Environmental Protection Agency, Science Advisory Board (1400F), 401 M Street,
SW, Washington, DC 20460
Mrs. Mary Winston, Staff Secretary, Environmental Protection Agency, Science Advisory Board (1400F),
401 M Street, SW, Washington, DC 20460
-------
DISTRIBUTION LIST
The Administrator
Deputy Administrator
AA Office of Research and Development
Deputy AA, Science
Deputy AA, Management
Office of Science Policy
Directors of the Mega Laboratories and Centers
Research Laboratory Directors
AA Office of Policy Planning and Evaluation
Office of Policy Analysis
Office of Strategic Planning
AA Office of Administration and Resource Management
Regional Administrators
EPA Headquarters Library
EPA Regional Libraries
National Technical Information System
Library of Congress
------- |