March 27, 2000
EPA-SAB-EC-00-008
Honorable Carol M. Browner
Administrator
U.S. Environmental Protection Agency
1200 Pennsylvania Ave., NW
Washington, DC 20460
Subject: An SAB/BOSC Report: Review of the Science to Achieve Results
(STAR) Program of the Environmental Protection Agency
Dear Ms. Browner:
On January 12-13, 2000 a joint subcommittee of the Science Advisory Board (SAB) and the
Board of Scientific Counselors (BOSC) of the Office of Research and Development (ORD) met to
review the Agency's Science to Achieve Results (STAR) Program. The STAR Program, which was
established in Fiscal Year 1995, has the mission to include this country's universities and non-profit
centers in EPA's research program and to ensure the best possible quality of science in areas of highest
risk and greatest importance to the Agency. The STAR Program is administered by ORD's National
Center for Environmental Research (NCER). The joint STAR Review Subcommittee (the
Subcommittee) was charged to evaluate whether or not the STAR Program is structured appropriately
to achieve the stated purpose, to evaluate whether the program is integrated effectively with Agency
strategic plans and programs, and to examine the adequacy of efforts to communicate with the external
scientific and regulatory communities regarding STAR research opportunities and outputs. Although the
Agency and the Subcommittee agreed that the STAR Program has not been in operation for long
enough to allow evaluation of its impact on the Agency and the broader research community, the
Subcommittee was asked to recommend measures and systems that should be used to monitor the
STAR Program's impacts, costs, credibility, and effectiveness in later program reviews.
The SAB's Research Strategies Advisory Committee has provided advice to the Agency on the
Science and Technology Budget, implementation of peer review, and other issues that are relevant to
the STAR Program. The BOSC, which was established to advise the Assistant Administrator for
Research and Development on the management and operation of ORD's research programs, has
conducted reviews of the ORD centers, including the National Center for Environmental Research that
administers the STAR Program. Thus, the leadership of SAB and BOSC agreed to form a joint
SAB/BOSC subcommittee that would tap the knowledge and expertise of both advisory bodies for this
-------
review.
The Subcommittee's overall assessment is that the STAR Program is structured and managed
so as to generate high-quality science, conducted by well-qualified scientists, on topics that are relevant
to the environmental problems identified in the EPA Strategic Plan. Research Coordination Teams are
an excellent mechanism for planning solicitations, and there has been significant and beneficial
coordination with other agencies. Outreach to potential STAR applicants is strong, and the peer review
of proposals is rigorous. NCER has been exploring some new management procedures such as multi-
year program planning, web site key word search capabilities, and state-of-the-science reports on
selected topics. The Subcommittee encourages the use of these pilot processes to strengthen planning
and communication. In sum, the Subcommittee believes that the STAR Program is well planned, well
organized, and well managed.
It is in this spirit that we provide constructive suggestions for program management
improvements. The report describe a series of recommendations from the Subcommittee that are
designed to make improvements to the STAR Program in the areas raised in the charge questions. The
Subcommittee also recommends measures and systems that should be used to monitor the STAR
Program's impacts, costs, and effectiveness. Data of this sort should form the basis of a subsequent
review of the Program's impacts, both within and outside of the Agency.
We emphasize two overarching and inter-related issues relative to staff resources and
information transfer. Regarding information transfer, the Agency must develop a comprehensive
approach for effective transfer of STAR results to Agency users. While it is too early to measure the
impacts of the grants awarded to date, it is not too early to focus on better processes, mechanisms, and
tools to transfer STAR results and information to Agency users. The Subcommittee applauds NCER's
ongoing initiatives to improve communication of results, such as State of the Science Reports,
workshops, program office liaisons and Internet tools. Even more emphasis is needed, however, on
developing additional tools, management processes, and procedures for the information transfer aspects
of the program in order to achieve the intended results. The Subcommittee recommends that the
Agency select several STAR research grants as case examples and evaluate the effectiveness of the
coordination with the relevant client offices and the degree to which the awards are supporting the
Agency's strategic goals. In addition, NCER should be looking for ways to shorten the time frames for
getting peer reviewed information to users and making the information easily accessible by potential
users.
The second over-arching issue relates to staff resources. The workload of the project officers
has increased significantly since 1995, as the STAR Program has grown. The high workload threatens
to hinder the ability of the STAR Program to promote communication among researchers and EPA
offices regarding the nature of the funded research and likely applications of the results. In addition to
information transfer activities, project officers' responsibilities include planning, preparation of
solicitations, review of proposals, and monitoring and oversight of awarded grants. As noted above, a
-------
more thorough and systematic approach to the information transfer task is critical to the STAR
Program's success. This clearly will be a challenge for NCER given the limitations on staff resources
but will likely make a difference in the overall success of the program.
We appreciate the opportunity to review and provide advice on the Agency's Science to
Achieve Results Program. The STAR Review Subcommittee would be pleased to expand on any of
the findings in the attached report, and we look forward to your response.
Sincerely,
/signed/ /signed/
Dr. Costel D. Denson, Chair Dr. Morton Lippmann, Interim Chair
Board of Scientific Counselors Science Advisory Board
/signed/ /signed/
Dr. Marilyn A. Brown, Co-Chair Dr. William Randall Seeker, Co-Chair
STAR Review Subcommittee STAR Review Subcommittee
-------
March 27, 2000
Norine E. Noonan
Assistant Administrator for
Research and Development
U.S. Environmental Protection Agency
1200 Pennsylvania Ave., NW
Washington, DC 20460
Subject: An SAB/BOSC Report: Review of the Science to Achieve Results
(STAR) Program of the Environmental Protection Agency
Dear Dr. Noonan:
On January 12-13, 2000 a joint subcommittee of the Science Advisory Board (SAB) and the
Board of Scientific Counselors (BOSC) of the Office of Research and Development (ORD) met to
review the Agency's Science to Achieve Results (STAR) Program. The STAR Program, which was
established in Fiscal Year 1995, has the mission to include this country's universities and non-profit
centers in EPA's research program and to ensure the best possible quality of science in areas of highest
risk and greatest importance to the Agency. The STAR Program is administered by ORD's National
Center for Environmental Research (NCER). The joint STAR Review Subcommittee (the
Subcommittee) was charged to evaluate whether or not the STAR Program is structured appropriately
to achieve the stated purpose, to evaluate whether the program is integrated effectively with Agency
strategic plans and programs, and to examine the adequacy of efforts to communicate with the external
scientific and regulatory communities regarding STAR research opportunities and outputs. Although the
Agency and the Subcommittee agreed that the STAR Program has not been in operation for long
enough to allow evaluation of its impact on the Agency and the broader research community, the
Subcommittee was asked to recommend measures and systems that should be used to monitor the
STAR Program's impacts, costs, credibility, and effectiveness in later program reviews.
The SAB's Research Strategies Advisory Committee has provided advice to the Agency on the
Science and Technology Budget, implementation of peer review, and other issues that are relevant to
the STAR Program. The BOSC, which was established to advise the Assistant Administrator for
Research and Development on the management and operation of ORD's research programs, has
conducted reviews of the ORD centers, including the National Center for Environmental Research that
administers the STAR Program. Thus, the leadership of SAB and BOSC agreed to form a joint
SAB/BOSC subcommittee that would tap the knowledge and expertise of both advisory bodies for this
-------
review.
The Subcommittee's overall assessment is that the STAR Program is structured and managed
so as to generate high-quality science, conducted by well-qualified scientists, on topics that are relevant
to the environmental problems identified in the EPA Strategic Plan. Research Coordination Teams are
an excellent mechanism for planning solicitations, and there has been significant and beneficial
coordination with other agencies. Outreach to potential STAR applicants is strong, and the peer review
of proposals is rigorous. NCER has been exploring some new management procedures such as multi-
year program planning, web site key word search capabilities, and state-of-the-science reports on
selected topics. The Subcommittee encourages the use of these pilot processes to strengthen planning
and communication. In sum, the Subcommittee believes that the STAR Program is well planned, well
organized, and well managed.
It is in this spirit that we provide constructive suggestions for program management
improvements. The report describe a series of recommendations from the Subcommittee that are
designed to make improvements to the STAR Program in the areas raised in the charge questions. The
Subcommittee also recommends measures and systems that should be used to monitor the STAR
Program's impacts, costs, and effectiveness. Data of this sort should form the basis of a subsequent
review of the Program's impacts, both within and outside of the Agency.
We emphasize two overarching and inter-related issues relative to staff resources and
information transfer. Regarding information transfer, the Agency must develop a comprehensive
approach for effective transfer of STAR results to Agency users. While it is too early to measure the
impacts of the grants awarded to date, it is not too early to focus on better processes, mechanisms, and
tools to transfer STAR results and information to Agency users. The Subcommittee applauds NCER's
ongoing initiatives to improve communication of results, such as State of the Science Reports,
workshops, program office liaisons and Internet tools. Even more emphasis is needed, however, on
developing additional tools, management processes, and procedures for the information transfer aspects
of the program in order to achieve the intended results. The Subcommittee recommends that the
Agency select several STAR research grants as case examples and evaluate the effectiveness of the
coordination with the relevant client offices and the degree to which the awards are supporting the
Agency's strategic goals. In addition, NCER should be looking for ways to shorten the time frames for
getting peer reviewed information to users and making the information easily accessible by potential
users.
The second over-arching issue relates to staff resources. The workload of the project officers
has increased significantly since 1995, as the STAR Program has grown. The high workload threatens
to hinder the ability of the STAR Program to promote communication among researchers and EPA
offices regarding the nature of the funded research and likely applications of the results. In addition to
information transfer activities, project officers' responsibilities include planning, preparation of
solicitations, review of proposals, and monitoring and oversight of awarded grants. As noted above, a
-------
more thorough and systematic approach to the information transfer task is critical to the STAR
Program's success. This clearly will be a challenge for NCER given the limitations on staff resources
but will likely make a difference in the overall success of the program.
We appreciate the opportunity to review and provide advice on the Agency's Science to
Achieve Results Program. The STAR Review Subcommittee would be pleased to expand on any of
the findings in the attached report, and we look forward to your response.
Sincerely,
Dr. Costel D. Denson, Chair
Board of Scientific Counselors
lortcfi'Li
Science Advisory Board
Dr. Marilyn A. Brown, Co-Chair
STAR Review Subcommittee
Dr. William Randall Seeker, Co-Chair
STAR Review Subcommittee
-------
NOTICE
This report has been written as part of the activities of the Board of Scientific Counselors and
the Science Advisory Board, public advisory groups that provide extramural scientific information and
advice to the Administrator, the Assistant Administrator for Research and Development, and other
officials of the Environmental Protection Agency. Both boards are structured to provide balanced,
expert assessment of scientific matters related to problems facing the Agency. This report has not been
reviewed for approval by the Agency and, hence, the contents of this report do not necessarily
represent the views and policies of the Environmental Protection Agency, nor of other agencies in the
Executive Branch of the Federal government, nor does mention of trade names or commercial products
constitute a recommendation for use.
Distribution and Availability: This joint Science Advisory Board/Board of Scientific Counselors
report is provided to the EPA Administrator, Assistant Administrator for Research and Development,
senior Agency management, appropriate program staff, and interested members of the public. The
report is posted on the SAB website (www.epa.gov/sab). Information on its availability is also
provided in the SAB's monthly newsletter (Happenings at the Science Advisory Board). Additional
copies and further information are available from the SAB Staff.
-------
U.S. Environmental Protection Agency
Science Advisory Board ancT
Board of Scientific Counselors
Joint STAR Review Subcommittee
CO-CHAIRS
Dr. Marilyn A. Brown, Deputy Director, Energy Efficiency & Renewable Energy Program, Oak
Ridge National Laboratory, Oak Ridge, TN
Dr. W. Randall Seeker, Senior Vice President, General Electric Energy & Environmental Research
Corp., Irvine, CA
MEMBERS
Dr. Ann Bostrom, School of Public Policy, Georgia Institute of Technology, Atlanta, GA
Dr. Stephen L. Brown, Director, R2C2 Risks of Radiation and Chemical Compounds, Oakland, CA
Dr. William E. Cooper, Professor Emeritus, Dept. of Zoology, Michigan State University, East
Lansing, MI
Dr. Ishwar P. Murarka, Chief Scientist and President, Ish Inc., Sunnyvale, CA
STAFF
Ms. Stephanie Sanzone, Designated Federal Officer, Environmental Protection Agency, Science
Advisory Board (1400A), 1200 Pennsylvania, Ave., NW, Washington, DC 20460
Ms. Shirley Hamilton, Designated Federal Officer, Environmental Protection Agency, Board of
Scientific Counselors, Office of Research and Development (8701R), 1200 Pennsylvania, NW,
Washington, DC 20460
Ms. Betty Jo Overton, Alternate Designated Federal Officer, Environmental Protection Agency,
Board of Scientific Counselors, Office of Research and Development (8701R), 1200 Pennsylvania,
-------
NW, Washington, DC 20460
Ms. Betty Fortune, Office Assistant, U.S. Environmental Protection Agency, Science Advisory
Board (1400A), Washington, DC 20460
111
-------
TABLE OF CONTENTS
1. EXECUTIVE SUMMARY
2. INTRODUCTION
3. OVERVIEW OBSERVATIONS AND COMMENTS
4. RESPONSE TO THE CHARGE 10
4.1 Supporting High Quality Science 10
4.2 Supporting the Agency's Strategic Goals 13
4.3 Communication and Coordination 17
4.4 Metrics of Success 19
5. PRINCIPAL RECOMMENDATIONS 23
6. REFERENCES CITED 25
APPENDIX A: NCERQA Web Site Statistics for December 1999 A-l
IV
-------
1. EXECUTIVE SUMMARY
This report presents the results of a review of the U.S. Environmental Protection Agency's
Science to Achieve Results (STAR) Program. The review was conducted by the STAR Review
Subcommittee, a joint subcommittee of the Science Advisory Board (SAB) and the Board of Scientific
Counselors (BOSC) of the Office of Research and Development (ORD).
Background and Purpose
The STAR Program was established in Fiscal Year 1995 for the purpose of including this
country's universities and non-profit centers in EPA's research program and to ensure the best possible
quality of science in areas of highest risk and greatest importance to the Agency. It is managed by
EPA's National Center for Environmental Research (NCER). Because the Program has operated for
only 5 years, the STAR Review Subcommittee concluded that it was too soon to evaluate the quality
and impact of the STAR Program's research results. Instead, the Subcommittee focused on the
Program's structure, managerial approaches, and the processes used to request, select, and
communicate research projects.
Specifically, EPA's Charge to the STAR Review Subcommittee was to evaluate whether or not
the STAR Program is structured appropriately to achieve its goals, to evaluate whether the Program is
integrated effectively with other Agency programs, and to examine the adequacy of efforts to
communicate with the external scientific and regulatory communities regarding STAR research
opportunities. The Subcommittee was also asked to recommend measures and systems that should be
used to monitor the Program's impacts, costs, credibility, and effectiveness in later program reviews.
The Subcommittee's evaluation is based on several sources of information. These include:
a) the NCER responses to a set of questions posed by the Subcommittee (EPA, 1999);
b) a 1998 evaluation of the National Center for Environmental Research and Quality
Assurance (the predecessor of NCER) by the BOSC (BOSC, 1998);
c) information on the NCER web site (www.epa.gov/ncerqa); and
d) briefings and information provided by Agency staff at a Subcommittee meeting on
January 12-13, 2000.
In addition, the Subcommittee evaluated a sample of STAR grant folders containing grant proposals,
peer reviewer comments, and decision documentation.
-------
Conclusions
The Subcommittee concludes that the STAR Program is of vital importance to the Agency's
mission and to the national objective of improving the knowledge base for environmental assessment
and management. The Subcommittee's overall assessment is that the STAR Program is structured and
managed so as to generate high-quality science, conducted by well-qualified scientists, on topics that
are relevant to the environmental problems identified in the EPA Strategic Plan (EPA, 1997). Research
Coordination Teams are an excellent mechanism for planning solicitations, and there has been significant
and beneficial coordination with other agencies. Outreach to potential STAR applicants is strong, and
the peer review of proposals is rigorous. NCER has been exploring some new management
procedures such as multi-year program planning, web site key word search capabilities, and state-of-
the-science reports on selected topics. The Subcommittee encourages the use of these pilot processes
to strengthen planning and communication for the STAR Program. In sum, the Subcommittee believes
that the Program is well planned, well organized, and well managed.
It is in this spirit that we provide constructive suggestions for program management
improvements. The following sections describe a series of recommendations from the Subcommittee
that are designed to strengthen the program in the areas raised in the charge questions. Two
overarching observations worthy of more significant consideration are summarized first. They pertain to
staff resources and information transfer.
Staff Resources. Greater NCER staff resources are required to maximize the public's return
on investment in the STAR Program. Since its founding in 1995, the budget for the STAR Program has
grown from approximately $50 million in 1995 to over $100 million in FY99. The staff available to
manage the program, however, has remained at approximately 36 FTEs, including 18 Project Officers.
Thus, the staffs workload has grown significantly. The potential for significant increases in management
efficiencies within NCER appears to be small. The resulting staff resource deficiency threatens to
hinder the ability of the STAR Program to promote communications among researchers and EPA
offices in need of the research results. This staffing need was identified previously in the BOSC's 1998
review of NCERQA (BOSC, 1998). It appears that the problem has continued unabated since that
report was published.
Information Transfer. Greater emphasis and attention needs to be placed on developing and
implementing the tools, management processes and procedures for ensuring that the information and
results of the STAR Program are being rapidly and effectively transferred to Agency users. The
Subcommittee found the procedures for this to be uneven and insufficient. NCER has already been
exploring some procedures to strengthen information transfer, such as multi-year program planning for
Paniculate Matter and other programs, web site key word searches, and state-of-the-science reports.
The Subcommittee endorses these pilot processes and suggests additional mechanisms, such as the
establishment of STAR points-of-contact throughout the Agency. The Subcommittee strongly
encourages NCER to place even more emphasis on such tools and procedures. This will likely make a
-------
difference to the overall success of the program.
Recommendations
Keeping in mind the Subcommittee's overall favorable assessment of the STAR Program's
structure and management, a selection of additional recommendations for program management
improvements are presented below. Several of these recommendations refer to the need for stronger
information transfer.
a) The Agency should provide additional information in RFAs on research goals and
objectives and on budget and relevancy criteria that will be used to evaluate proposals
in order to raise the low proposal acceptance rate, increase the success rate of the best
quality applications, and enhance the relevance of the research.
b) The Agency should take steps to accelerate the peer review process for STAR results
so that the results are available to support Agency decisions in a timely manner.
c) The Agency should select several STAR research grants as case examples and evaluate
the effectiveness of the coordination with the relevant client offices and the degree to
which the awards are supporting the Agency's strategic goals. This could be part of a
broader evaluation of the effectiveness of current STAR Program communication,
technology transfer, and outreach efforts.
d) The Agency should consider means of strengthening communications between Agency
program staff and STAR grant recipients, such as: (1) meetings between principal
investigators and Agency program staff to discuss integration opportunities, and (2)
establishment of STAR points-of-contact at ORD laboratories and centers, as well as
in Agency program and regional offices.
e) The Agency should assess how well the needs and issues of the regional offices are
factored into the STAR planning process and consider additional mechanisms for
ensuring adequate regional involvement in STAR Program activities.
f) The Agency should request feedback on the success of the program review workshops
and should expand the workshop proceedings to include a record of discussions
regarding the relevancy of STAR results to the Agency's research and regulatory
agenda and to environmental decision-making.
g) Given the recent growth of STAR Program grants to academic researchers and the
evolving set of environmental issues facing the Agency, the Agency should develop and
implement a process for periodically assessing the Agency's portfolio in terms of its use
-------
of different funding instruments (contracts, grants, inter-agency agreements, and
cooperative agreements) and the reliance on different R&D performers (universities,
for-profit contractors, others).
-------
h) The Agency should continue and expand its partnerships with other agencies and
funding organizations, including possible joint funding of STAR research with private
foundations and with international agencies and research organizations.
i) The Agency should seek assistance from program evaluation and decision-analysis
experts to help ORD develop a monitoring and evaluation system for the STAR
Program. The evaluation system should include measures of the STAR Program's
contributions to Sound Science (i.e., measures of science quality and quantity) and to
Mission Advancement.
j) The Agency should budget sufficient resources to secure the services of a qualified,
highly respected, and independent organization to conduct and publish an evaluation of
the STAR Program's results, effectiveness, and impact. This evaluation could begin in
approximately two years, at which point sufficient time should have passed
(approximately seven years) for the results from the initial cohorts of STAR grants to
have been published and used.
-------
2. INTRODUCTION
In Fiscal Year 1995, the Office of Research and Development (ORD) launched a competitive
grants program called Science to Achieve Results (STAR). The STAR Program's mission is to include
this country's universities and non-profit centers in EPA's research program and to ensure the best
possible quality of science in areas of highest risk and greatest importance to the Agency. The STAR
Program, which is administered by ORD's National Center for Environmental Research, NCER
(formerly called the National Center for Environmental Research and Quality Assurance, NCERQA),
consists of four components:
a) Focused Requests for Applications (RFAs) targeted to national environmental science
needs as related to the mission of the Agency;
b) The Exploratory Research Grants Program, which provides support for investigator-
initiated grants in broad areas of environmental science;
c) The Graduate Fellowship Program, which provides support for master's and doctoral
students in environmental science, engineering and policy; and
d) The Environmental Research Centers Program, which focuses on long-term, multi-
disciplinary research issues.
Together, these components are designed to support the EPA's goals as they are described in the
Agency's Strategic Plan (EPA, 1997). Specifically, the STAR Program seeks to support the Agency
goal of "Sound Science" (Goal 8) through the improved understanding of environmental risk and
greater innovation to address environmental problems. In addition, the Program seeks to support the
Agency's various mission-oriented goals such as clean air, clean and safe water, and better waste
management. As an extramural program, STAR was intended to expand the relevant research
community outside the Agency to complement the intramural program and thereby to enhance the
scientific rigor and credibility of the research available to assist and inform its regulatory mission.
A joint subcommittee comprised of members from the Research Strategies Advisory
Committee (RSAC) of the Science Advisory Board and from the Board of Scientific Counselors
(BOSC) of the Office of Research and Development (ORD) was established to conduct a review of
the Agency's initial efforts to develop and implement the STAR Program. This joint subcommittee is
called the STAR Review Subcommittee.
This initial review should be viewed as part of a broader, longer term effort to examine all
aspects of the STAR Program. At this time there are an insufficient number of STAR grants completed
to evaluate the quality and impact of the STAR research results. The time lag between release of
-------
RFAs, selection and funding of research projects, completion of research, and availability of published
research results in some cases can be 7-10 years. At this point in the STAR Program, the
Subcommittee can only assess expected programmatic impacts of management decisions based on
existing structure and managerial approaches. The first review, therefore, focused on the processes
used in the STAR Program to request, select, and communicate research projects. Later reviews
should focus on the results, effectiveness, and impacts of the STAR Program. Nevertheless, the initial
review also addressed the question of metrics for which data should be collected to support subsequent
evaluations of the broader impact of the STAR Program.
Specifically, EPA's Charge to the STAR Review Subcommittee was:
a) Is the STAR Program structured to support outstanding scientists and technically
meritorious research? Is the outreach to potential applicants, the review of proposals,
and the management of awards structured to foster high quality science?
b) Is the STAR Program effectively integrated with ORD's in-house programs and with
other EPA programs? Are the topics for STAR solicitations selected consistent with the
priorities identified in the Agency's Strategic Plan? Are there other opportunities where
the STAR Program could significantly contribute to the Agency's strategic goals?
c) Is the STAR Program communicating well within the Agency, with the external scientific
and regulatory communities, and with other stakeholders? Is there sufficient leveraging
and coordination of research efforts?
d) What systems should be in place to monitor the Program's impacts, costs, credibility,
and effectiveness, and to what extent are these in place already? What metrics of
success in determining the effectiveness of grants to have impacts on Agency decisions
should be developed? What information should be collected today on metrics of
success for the STAR grants? How should program offices and other agency
customers for the grant products be involved in the establishment of criteria for
measuring the impacts of the program?
Prior to the meeting, the Subcommittee developed a list of self-study questions for the Agency.
The NCER responses to these questions (EPA, 1999), along with a previous evaluation of the
NCERQA by the BOSC (BOSC, 1998), information on the NCER web site (www.epa.gov/ncerqa),
and briefings and information provided by Agency staff at the meeting, formed the basis for
deliberations during the Subcommittee meeting on January 12-13, 2000. At the public meeting the
discussions included clarifying questions relative to the self-study responses and additional discussion on
issues arising from the discussions. In addition, the Subcommittee evaluated a sample of STAR grant
folders (containing grant proposals, peer reviewer comments, and decision documentation) in order to
assess the nature of the documentation, including that relating to relevancy review. At the meeting, the
7
-------
Subcommittee was also briefed on the Government Accounting Office's (GAO) on-going evaluation of
the STAR Program, which is expected to produce a report in July 2000.
-------
3. OVERVIEW OBSERVATIONS AND COMMENTS
The Subcommittee concludes that the STAR Program is a program of vital importance to the
Agency's mission, as well as to the national objective of improving the knowledge base for
environmental assessment and management. The program is structured and managed so as to play a
key role in generating high quality science, conducted by highly qualified scientists, on topics that are
relevant to the environmental problems identified in the EPA Strategic Plan (EPA, 1997). In materials
provided to the Subcommittee, the STAR Program's mission is defined as "to include this country's
university and non-profit centers in EPA's research program and to ensure the best possible quality of
science in areas of highest risk and greatest importance to the Agency" (EPA, 2000).
The Agency has created the infrastructure and management systems to accomplish this mission
in a rapid manner and has implemented an extensive extramural program without appreciable increases
in internal staffing. Primarily because of this mismatch in growth rates within NCER of grant dollars
relative to staffing levels, it is not surprising that some potential areas of concern have arisen in the
implementation of the Program. The Subcommittee's overall assessment is that the planning and
execution of the STAR program is well developed and appropriate. It is in this spirit that we provide
constructive suggestions for program management improvements. The following sections describe a
series of recommendations from the Subcommittee that are designed to make improvements in the
program in the areas raised in the charge questions. Two overarching observations worthy of more
significant consideration are noted below relative to staff resources and information transfer.
a) Staff Resources
Since its founding in 1995, the budget for EPA's Science to Achieve Results (STAR) program
has grown from approximately $50 million in FY95 to over $100 million in FY99. During each of the
last three years, the program processed 3,000 to 3,500 grant applications, awarded approximately $95
million in grants to about 300 grantees, and managed approximately 1,000 active research grants and
fellowships. Three NCER divisions administer the STAR Program: the Environmental Science
Research Division, the Environmental Engineering Research Division, and the Peer Review Division.
There is a total of approximately 36 staff in these three divisions, including 18 Project Officers. These
36 staff members manage the STAR Program as well as the EPA Small Business Innovation Research
(SBIR) Program and several university-based research centers that are not part of the STAR Program.
According to the NCER Director, the ratio of STAR Program applicants per Project Officer
and the ratio of active grants per Project Officer are comparable to ratios associated with the NSF
grants program. However, unlike NSF, the STAR Program is embedded within a mission-oriented
agency. As a result, the NCER Project Officers have a critical information transfer responsibility to
maximize the value of the STAR grants to the Agency. NSF grant officers do not have this additional
responsibility.
-------
It is the Subcommittee's understanding that STAR Program Project Officers' responsibilities
include:
(1) planning, through participation on Research Coordination Teams (RCTs) and
interagency committees;
(2) preparation of solicitations, outreach to potential applicants, and proposal review;
(3) monitoring and quality assurance during the life of the projects; and
(4) summarizing, communicating and marketing project results to promote their use by the
Agency (i.e., information transfer).
The workload of the project officers has increased significantly since 1995, as the STAR
Program has grown. The high workload threatens to hinder the ability of the STAR Program to
promote communication among researchers and EPA offices in need of the research results. A more
thorough and systematic approach to the information transfer task is critical to the STAR Program's
success, and EPA staff increases likely will be required to follow through with the information transfer
process.
The potential for significant increases in management efficiencies among NCER staff appears to
be small. The only task that the Subcommittee thought possibly could be scaled back is the visits of
Project Officers to STAR grantees. Perhaps a sorting process could be developed to identify and
eliminate the site visits that would offer the lowest retum-on-investment. With the exception of this one
possible budget-tightening opportunity, it appears that greater EPA staff resources are required to
maximize the public's return on investment in the STAR Program. This staffing need was previously
identified in the BOSC's 1998 review of NCERQA (BOSC, 1998). It appears that the problem has
continued unabated since that report was published. Future in-depth evaluations of the STAR Program
should assess the workload and responsibilities of the project officers.
b) Information Transfer
Greater emphasis and attention needs to be placed on developing and implementing the tools,
management processes and procedures for ensuring that the information and results of the STAR
program are being rapidly and effectively transferred to the Agency and other potential users. As the
acronym STAR implies, the focus is on the achievement of results through the science supported by the
program. Thus, it is critical to the Agency users and customers that the peer reviewed information and
results be transferred in a rapid and effective manner. The Subcommittee provides suggestions on
potential metrics and procedures for measuring the effectiveness of the program in Section 4.4. An
evaluation of the STAR Program's impact could begin in approximately two years, at which point
sufficient time should have passed (approximately seven years) for the results from the initial cohorts of
STAR grants to have been published and used.
10
-------
While it is too early to measure the impacts of the grants awarded to date, it is not too early to
focus on better processes, mechanisms, and tools to transfer the information and knowledge to Agency
users. NCER should be looking for ways to shorten the time frames for getting peer reviewed
information to users and making the information easily accessible by potential users. The Subcommittee
found the procedures to transfer STAR-generated information to Agency users to be uneven, with
some users having procedures such as liaisons and teams ready to accept the information and others
with no formal mechanisms. NCER has already been exploring some procedures such as multi-year
program planning for Particulate Matter and other program areas, web site key word searches, and
state-of-the-science reports and the Subcommittee endorses these pilot processes. The Subcommittee
strongly encourages NCER to place even more emphasis on developing additional tools, management
processes and procedures for the information transfer aspects of the program in order to achieve the
intended results. This clearly will be a challenge for NCER given the limitations on staff resources
presented above but will likely make a difference in the overall success of the program.
11
-------
4. RESPONSE TO THE CHARGE
4.1 Supporting High Quality Science
Charge Question 1: Is the STAR Program structured to support outstanding scientists
and technically meritorious research? Is the outreach to potential applicants, the review
of proposals, and the management of awards structured to foster high quality science?
The answer to this charge question is provided in three parts, corresponding to the STAR
Program's outreach to potential applicants, review of proposals, and management of awards.
Outreach to Potential Applicants. ORD uses the NCER web site and the maintenance of
an electronic announcement server for disseminating information about research opportunities. NCER
also publishes notices of research opportunities in scientific journals and in the Federal Register. The
Subcommittee did not have sufficient information or resources to assess the effectiveness of these
communication mechanisms for the majority of potential applicants. The Subcommittee noticed,
however, that a number of the NCER web pages had not been updated recently, even though newer
information should be available (for more discussion of the web site see 4.3). Subcommittee members
noted that some of the STAR information was updated on the web site following the January
Subcommittee review meeting.
The sheer number of applicants responding to STAR Program RFAs and the widespread
distribution of universities receiving STAR grants and fellowships is an indication that the Agency is
doing a good job of outreach, notification, and cultivation of interest in the STAR Program. In order to
determine whether additional outreach is needed to attract the best scientists and research, the Agency
should conduct a more thorough analysis of the sources of proposals and the success rate of those
sources.
Review of Proposals. While the Subcommittee did not conduct an independent review of the
scientific merits of rejected versus funded proposals, the STAR proposal review process does appear
to be well structured. The review process is structured similarly to high quality programs at the
National Science Foundation and the National Institutes of Health, and so in principle should be able to
achieve comparably meritorious results. Further, the criteria used in the grant review process seem
both appropriate and clearly stated. Whether or not the review process in fact has resulted in the
selection of the best scientific proposals is more difficult to ascertain. (Various measures of quality,
merit, and impact are discussed in Section 4.4.)
The responses to specific topical RFAs are evaluated using a two-tiered approach. First,
proposals are peer reviewed by independent panels to determine their scientific and technical merit.
Rating Categories consist of Excellent, Very Good, Good, Fair, and Poor. Following the technical
12
-------
review, proposals undergo relevancy review to ensure that the proposed research will address Agency
priorities and will complement in-house research efforts. Because of the significant number of
proposals received, only the proposals receiving Excellent or Very Good ratings by the peer review
panel undergo a relevancy review by the Agency before final funding decisions are made. The use of
such a cut-off for subsequent relevancy review seems appropriate. The Subcommittee's comments on
the relevancy review process are contained in Section 4.2.
The Subcommittee analyzed the success rates of STAR Program grant proposals for
solicitations closing in FY1998, based on summary information provided by the Agency at the
Subcommittee meeting (EPA, 2000). Of approximately 1,721 proposals received, approximately 181
(10.5%) were funded. This success rate is lower than those experienced by the National Science
Foundation (NSF), which is one of the largest flinders of environmental research. NSF success rates
have averaged approximately 30 percent (http://www.nsf.gov/bfa/bud/fyl998/resprj02.htm). The
difference is due, in part, to the inclusion of an EPA relevancy review in the STAR Program. The
STAR Program data for FY 1998 indicate that scientific merit peer review approved approximately
23% of the proposals (average ratings of Excellent or Very Good are required for approval). The
subsequent relevancy review, in combination with budgetary limitations, resulted in approximately 45%
of these highly rated proposals being funded.
Success rates varied from topic to topic (see Table 1). In general, proposals to the STAR
Exploratory Research Grant Program had lower than average success rates. This is not surprising.
One would expect a larger number of the exploratory grant proposals to score lower during the
relevancy review if the same criteria are applied as are used to evaluate proposals under the Focused
Requests for Applications.
The low overall success rate means that a considerable amount of time is being invested in the
preparation of unsuccessful proposals, which constitutes a significant drain on the nation's
environmental scientists. It also constitutes an extra burden on NCER staff. The Subcommittee
suggests that the Agency provide additional information in RFAs on research goals and objectives and
on budget and relevancy criteria that will be used to evaluate proposals so that the success rate of the
best quality applications increases while ensuring the relevance of the research to the Agency. This
should be done in a manner that preserves the principal investigators' freedom to frame the research
problem in novel ways and to invent new strategies for implementing the research.
The Subcommittee also encourages the Agency to prepare more detailed documentation of the
relevancy review so that the applicants will be better informed about what is being sought. This could
include defining relevancy criteria used and providing links to strategic plans and funding limitations in
order to open the relevancy decision process. In addition, it is recommended that applicants whose
proposals are not funded receive specific information about the reasons (including budgetary and
relevancy considerations) for their rejection. This was not done by NCER staff in the letters going to
applicants whose Center proposals were reviewed by the Subcommittee. Since their proposals were
13
-------
judged to be technically meritorious, it is important that the decision-making for these proposals be
explained clearly to their authors. The Subcommittee also encourages the Agency to provide individual
reviewer comments to the applicants who are declined, rather than just a summary of these documents.
Applicants would learn more if they receive the individual review comments from all principal reviewers
(in an anonymous fashion).
Table 1. Success Rates in 1998 of Various STARRFAs
RFA Topic
Topics with Lowest Success Rates
Indicators of Global Climate
Change
Exploratory Environmental
Engineering
Chemical Mixtures Toxicology
Bioremediation
Exploratory Environmental
Biology
Topics with Highest Success Rates
Hexavalent Chromium Risk
Reduction
Urban Air Toxics
Children's Environmental
Health Centers
Drinking Water
Regional Scale Analysis and
Assessment
Proposals
Received
-27
-199
-79
-49
-193
21
-18
-64
-54
-22
Proposals
Forwarded
to Relevancy
Review
3
30
16
7
41
4
9
8
16
4
Proposal
s Funded
0
10
4
3
-12
4
5
8
10
4
Overall
Success
Rate
0%
5.0%
5.9%
6.1%
6.2%
48%
28%
26%
19%
18%
Source: EPA, 2000.
Another recommendation for improvement pertains to the initial assignment of proposals to
reviewers. For example, in some programs at NSF, panelists have been asked to designate which
14
-------
proposals they might feel most comfortable reviewing based only on titles, as is done for the STAR
review process. Sometimes this results in mismatches. For this reason, at least one NSF program
sends reviewers the proposal abstract/summary at this first stage in order to enable the reviewers to
judge more confidently the content of the proposal and the match with their capabilities. A similar
process of sending abstracts to reviewers should be considered by EPA.
In addition, one program at NSF has found that, despite instructions, different panelists use the
5-point rating scale differently, and that normalizing the scores given by each panelist can produce
different overall ratings for a proposal. The Agency should examine if the tendency of some reviewers
to "grade on the curve" is impacting the overall rating levels. This could be done at the time of the Panel
meeting and could be discussed during the Panel's overall deliberations.
The configuring of review panels with high quality expert reviewers is a difficult challenge given
the number of reviewers needed. The ORD appears to be managing this well with a combination of its
own data base, with nearly 10,000 potential reviewers and a key-word search capability, combined
with a more recent reliance on the Community of Science (COS) data base. The review panel
selection also could be expanded to include more international experts. This would allow the process to
tap into the strong international expertise that exists in many environmental research areas1. In addition,
ORD should consider the use of ad hoc reviewers (i.e., reviewers who receive only one proposal to
review and do not attend panel meetings) in addition to panel reviews. Ad hoc reviewers might be
appropriate when proposals are highly diverse or in fields where recruitment of panelists is difficult.
Management of Awards. The Subcommittee encourages the Agency to adopt computerized
management systems to the maximum extent possible to manage the already heavy workloads and the
increasing demands made of the NCER staff. Electronic filing of grant applications and electronic
distribution of proposals to reviewers, when available, will be helpful in this regard.
4.2 Supporting the Agency's Strategic Goals
Charge Question 2: Is the STAR Program effectively integrated with ORD's in-house
programs and with other EPA programs? Are the topics for STAR solicitations selected
consistent with the priorities identified in the Agency's Strategic Plan? Are there other
opportunities where the STAR Program could significantly contribute to the Agency's
strategic goals?
Integration with ORD's in-house and Other EPA Programs. The involvement of ORD,
program offices and regional offices in the STAR Program occurs primarily during the planning process
researchers at academic or non-profit institutions located in the U.S., or with state or
local governments in the U.S., are eligible to apply for STAR grants.
15
-------
through the use of Research Coordination Teams (RCTs). The RCTs, which are organized largely
along media lines, include representatives from ORD, program offices and regional offices. These
teams develop the plan for research to be accomplished, consistent with Agency-wide and ORD
strategic plans, whether via intramural or extramural (e.g., STAR) means. This group also conducts the
final relevancy review of STAR applications that have received Very Good or Excellent ratings by
independent peer reviewers.
NCER has initiated a number of innovative activities to improve the integration of STAR
Program efforts and the communication of results to target audiences in the EPA and elsewhere. These
include:
a) joint solicitations with other federal agencies and organizations;
b) program review workshops, where grantees present interim results to other STAR
grantees working in related research areas, as well as to interested Agency staff;
c) research-in-progress reports;
d) state-of-the-science reports, which gather and integrate the research findings from
several grantees on related topics; and
e) a web site that posts STAR Program annual reports and summaries of final reports.
We applaud these efforts and recommend that NCER continue to expand cooperation and partnerships
with other parts of ORD, with EPA program and regional offices, and other federal, private, and
international research organizations as discussed in more detail below (4.3).
Planning. The process used to select RFA topics appears to be robust, appropriate, and well
integrated with program office needs and ORD and Agency-wide strategic plans. The use of the RCT
involving representatives from ORD, program offices, and regional offices appears to be a good, direct
approach for involving the key players and stakeholders.
One issue that warrants further examination is the extent to which regional office needs and
issues are factored into the planning process, given the wide diversity of EPA regions and the remote
locations that make their involvement difficult. There is only limited regional representation on the RCTs
and therefore involvement in RFA definition and relevancy reviews. The Agency should assess how
well their needs and issues are factored into the process and consider other mechanisms for ensuring
adequate regional involvement in STAR program planning and technology transfer activities.
NCER's self-study (EPA, 1999) also mentions an apparently new multi-year planning initiative
designed to demonstrate more clearly how the outputs of the STAR Program and other ORD efforts
support the strategic plan. This process would appear to be an effective mechanism for identifying
potential future RFA topics for STAR and the needs for information transfer from results of STAR
programs.
16
-------
Although the activities described in the self-study suggest that the STAR Program is indeed
effectively integrated with in-house research programs and with the strategic goals of the Agency, the
Agency should select several STAR research grants as case examples and evaluate the effectiveness of
the coordination with the relevant client offices and the degree to which the awards are supporting the
Agency's strategic goals. Case examples selected at random might be augmented by some examples
considered a priori to be exemplary. This self assessment could lead to ideas for further improvements
in integration.
17
-------
Relevancy Review. The involvement of representatives from ORD, program offices, and
regional offices in conducting the relevancy evaluation of the highest scientifically rated proposals is
another important means of keeping the EPA stakeholders involved in the STAR Program.
To strengthen the relevancy review and foster integration within ORD and across the Agency,
EPA might consider strengthening the involvement of ORD staff in the relevancy review. This could be
achieved by having EPA staff review the full proposals and peer reviews for all scientifically meritorious
STAR applications, instead of simply having the relevancy review conducted on the basis of the
abstracts and the peer review panel's summary comments. This would increase ORD staff awareness
of the contents of highly rated STAR proposals.
Interactions During the Course of the STAR Grant. Once the grants are awarded, there
are limited interactions between the grantee and the Agency. The interactions are generally
reporting-out type functions such as annual meetings, annual reports, project officer meetings with
grantees (held once during the grant cycle), and participation in a program review workshop. While
there is encouragement of the Agency scientists to work with the grantee and tools are made available
(such as web-based reporting and searches) for Agency personnel to follow the work, there are no
specific mechanisms to ensure closer working relationships and an ongoing awareness of EPA's
evolving needs.
Two possible new activities should be considered for strengthening communication between
Agency personnel and grant recipients: (1) meetings between STAR Program principal investigators
and Agency staff to discuss integration opportunities, and (2) establishment of STAR points-of-contact
at ORD's Labs and Centers, as well as in Agency program offices.
Program Review Workshops. Workshops are held approximately annually for each major
RFA area so that grantees can share information among themselves and with Agency staff on the
progress of STAR-funded research. Feedback on the success of the workshops relative to increasing
interactions between STAR researchers and relevant EPA personnel would be useful. Expansion of the
proceedings should include a record of discussions, exchange of ideas, integration across research
projects, and their relevancy for environmental decision-making.
These workshops also are used by NCER as an opportunity for the grantee to be briefed on
EPA's evolving needs, as a means of encouraging grantees to keep in mind the regulatory context of
their research. The Subcommittee notes that this process is critically important to the success of the
STAR Program. Such mechanisms to facilitate the ongoing consideration of Agency goals are
appropriate for a mission-oriented agency. The workshops help to address a potential weakness—the
distancing of research from Agency goals, which can occur if the grantees do not completely
understand or are not following EPA evolving needs. It also avoids the micro-management and project
re-direction that could result from more heavy-handed oversight.
18
-------
Balancing the Research Portfolio. The allocation of available research dollars between
intramural research, contract activities, and STAR grants is important and has been shifting, partly as a
result of the STAR Program's growing budget. As noted in EPA's 1996 Report to Congress on the
STAR Program (EPA, 1996, p. 8), "the growth of the STAR program has been accomplished largely
through the re-direction of cooperative agreements and inter-agency agreements." This has resulted in
a significant shift in the nature of the Agency's extramural R&D performers (toward university and non-
profit centers and away from for-profit contractors and interagency agreements) and in the financial
mechanisms used to secure these R&D resources. There is no fundamental reason why the STAR
Program should limit eligibility of applicants to academic and non-profit research organizations. The
Subcommittee recommends that the Agency assess and evaluate, on an on-going basis, the allocation
of resources across types of research organizations and funding mechanisms.
The Subcommittee commends EPA's new initiative to develop multi-year plans (e.g., for
Paniculate Matter and other programs) that will relate STAR and intramural research products to the
Agency's strategic goals for different program areas. These plans will help provide a framework for
the Agency to consider, and to explain, the balance of R&D performers in individual research areas.
The Subcommittee is aware that there are specific rules governing the use of grants, contracts, and
cooperative agreements. However, in selecting among available funding vehicles, the Agency also
should consider:
a) the extent to which the Agency researchers have unique expertise and/or laboratory
facilities;
b) the need for the Agency to establish and maintain a world class research credibility and
leadership role in areas within its mandate;
c) the extent to which the research is basic or more applied or product-oriented; and
d) the extent to which the enhanced credibility associated with independent research
institutions is needed.
The definition of RFA topics, and their budget levels, should be a reflection of the Agency's judgment
as to the appropriate balance between intramural and extramural research, and among different types of
extramural R&D performers and funding vehicles. To this end, the multi-year program planning (or
"research roadmapping") process being piloted by ORD should be expedited and completed as quickly
as possible.
Transfer of STAR Results. One of the most important components of the integration of the
STAR Program is the transfer of the results of the individual grants to potential users within and outside
the Agency. This information transfer will be critical to the success of the STAR Program in the future
as more grants are completed. In some program offices and for some research areas (e.g., paniculate
19
-------
matter research in support of the Office of Air and Radiation), the STAR Program activities are closely
tied to ongoing program activities and mechanisms are in place to readily transfer information to meet
the needs of the office. Other program offices (e.g., the Office of Wetlands, Oceans, and Watersheds)
have defined a liaison to work with the STAR Program and prepare for the receipt of the results.
However, other program offices are less active. The Agency must fully develop the strategy and
mechanisms for effective transfer of STAR results from every RFA and grant to the Agency users
(Program Offices, Regions and Researchers).
NCER has taken a number of initiatives to enhance the transfer of STAR results. For example,
the NCER web site includes annual reports and summaries of final reports and program offices are
invited to annual workshops. In addition, NCER conducts targeted web searches for Agency
personnel to allow them to readily identify relevant STAR grants. Nonetheless, more needs to be done
to effectively integrate results.
The Agency's Peer Review Policy (June 1994 policy, contained in EPA, 1998) states that
major scientific and technically based products generated by the Agency that will be used to support
Agency decisions should be peer reviewed. Thus, it is "critical to quality" that the STAR Program lead
to peer reviewed information in a timely manner. Currently the Agency is relying on grant recipients to
publish their results in peer reviewed journals, a process which may take several years from the time
that research results are reported to the Agency under the terms of the grant. The Agency should take
steps to expedite the peer review process for STAR results. NCER is exploring the use of "State of the
Science" reports which independently gather and integrate the research findings from several grants on
related topics; these synthesis reports could be peer reviewed prior to release. The Subcommittee
identified several other techniques that might result in more rapid peer review of STAR research results,
including arranging for special issues of journals to accelerate review and publication of results, and
conducting peer review panels of STAR research results in the same manner as the initial review of the
STAR applications.
4.3 Communication and Coordination
Charge Question 3: Is the STAR Program communicating well within the Agency, with
the external scientific and regulatory communities, and with other stakeholders? Is there
sufficient leveraging and coordination of research efforts?
Communication. ORD fosters communication about the STAR program within and outside
the Agency through the Internet, publications, and workshops, as well as site visits, informal
communications, and other means. It highlights the NCER web site as perhaps its flagship
communication device. The proposed "state-of-science reports," and STAR program review
workshops, to which investigators and selected persons outside the STAR community are invited, are
ways in which the program communicates with both internal and external scientists. These mechanisms
are appropriate communication techniques; all but the workshops, however, are relatively passive and
20
-------
rely on an interested audience that will actively search for new information. Other, more pro-active
mechanisms should be identified and tested to further enhance this critical component of the program.
The NCER web site could benefit from updating and possibly from further promotion. It is not
clear how widespread the use of this web site is outside ORD, including among grant recipients,
although the site receives thousands of "hits" a month (see Appendix A). The Agency might benefit
from surveying a representative group of potential users of STAR results to measure awareness of the
program and evaluate the effectiveness of its communication strategies. Informal surveying by members
of the STAR Review Subcommittee suggests that potential users and investigators are aware of the
STAR program, but formal investigations of the effectiveness of STAR communication efforts are
warranted. Follow-up also is needed on the discussion of the NCERQA communications plan
contained in the BOSC review of NCERQA (BOSC, 1998). At a minimum, evaluation of the program
review workshops might be done using a simple questionnaire of the sort used at many conferences.
Internal evaluation of the program's outreach effectiveness could provide insight into how to improve
current efforts, raise program offices' awareness of the STAR Program's potential usefulness to them,
and even suggest new ways of ensuring that STAR results reach the appropriate EPA users.
Partnerships with other Agencies. The partnerships established with other federal agencies
for joint research solicitations during the past five years, and more recently with private-sector
organizations, have the potential for multiple benefits. These partnerships foster coordination of national
efforts for research on environmental issues and have allowed NCER to leverage its resources for
extramural support by more than 20 percent. They also help to establish credibility (due to the
additional agency affiliation), increase STAR Program visibility, share the EPA workload of program
management, and attract an additional cadre of researchers who have worked with the partnering
agency. These partnerships appear to be appropriate and should be continued to the maximum extent
possible. As the STAR Program becomes more amenable to quantitative assessments of costs and
benefits, the relative payback to these partnerships should be appraised.
Leveraging was seen by the Agency as especially important at the beginning when the STAR
Program was first created, to establish credibility. The Decision Making and Valuation for
Environmental Policy program with the National Science Foundation (NSF) has attracted a large
number of applications from top researchers; the program is currently being evaluated by NSF and
EPA. Lessons learned from the evaluation may help EPA improve this and future joint efforts. The
arsenic program announcement with the American Water Works Association Research Foundation
(AWWARF) is also considered a success by EPA staff, but EPA currently has no joint activities with
AWWARF. One of the reasons for the demise of such partnerships is that cooperation with other
agencies and organizations has a large transaction cost. The more conflicts there are between the goals
of EPA and those of the partnering organization, the more difficult the partnership is likely to be. When
is it worthwhile and when not? This is an important question that EPA needs to answer before
committing to conduct joint programs.
21
-------
The success of joint endeavors is illustrated by recent outreach to EPA by other agencies
interested in developing joint solicitations (e.g., the U.S. Department of Agriculture and the National
Institute for Occupational Safety and Health). We encourage ORD to expand its joint endeavors,
including possible joint funding of STAR research with private foundations and with international
agencies and research organizations. The new EPA-National Institute of Justice joint solicitation looks
promising, as does the joint program with USD A, DOD, NSF, and the Office of Naval Research on
phytoremediation.
4.4 Metrics of Success
Charge Question 4: What systems should be in place to monitor the Program's impacts,
costs, credibility, and effectiveness, and to what extent are these in place already? What
metrics of success in determining the effectiveness of grants to have impacts on Agency
decisions should be developed? What information should be collected today on metrics of
success for the STAR grants? How should program offices and other agency customers
for the grant products be involved in the establishment of criteria for measuring the
impacts of the program?
The Subcommittee suggests that ORD consider multiple approaches to evaluate the STAR
Program's impacts, costs, credibility, and effectiveness. The National Academy of Sciences (NAS)
notes in Evaluating Federal Research Programs (1999) that there are four important ways that the
nation benefits from its investment in federal research: knowledge advancement, knowledge application,
human capital development, and mission advancement. The Subcommittee identified several potential
metrics of success as they relate to the following groupings used in the NAS report:
a) the STAR Program's contribution to EPA's Sound Science goal, which includes
knowledge advancement, knowledge application, and human capital development; and
b) the STAR Program's role in supporting EPA's environmental risk assessment and risk
management goals, which is the mission advancement benefit noted by the NAS and the
impacts of decision making in the EPA Strategic Plan.
Sound Science. Indices that emphasize quantity (number of publications per grant, number of
publications per grant dollar, etc.) are commonly utilized to evaluate research programs because the
data are easy to obtain. Measures that indicate quality, including the influence of STAR research grants
on other researchers or funding organizations, while more difficult to devise and measure, would be
powerful indicators. A better measure of scientific excellence is the rate of citations of peer reviewed
publications by other scientists. These citations may appear in other grant proposals, peer reviewed
papers in professional journals, and as references supporting regulations and legislation. Various
citation indices are available that could be used as a good measure of the frequency with which STAR
grant studies are referenced by others. However, 4 to 6 years must pass between the completion of the
22
-------
STAR-funded research and the use of citations to judge success. It takes time for the original research
to be published in peer reviewed journals, then it takes another 2 to 3 years for the work to be cited by
others in subsequent publications. Another measure of scientific excellence is the frequency of national
awards given out by professional research organizations for research conducted by STAR Program
awardees.
Possible measures of "influence diffusion" from the STAR Program might include the extent of
follow-up funding by other agencies (such as NSF, NIH, or DOE), and the movement of STAR-
funded researchers (e.g., doctoral students and post-doctoral fellows) to other research institutions. A
potential benefit of the STAR program is the creation of a pool of future environmental professionals
who can disseminate the knowledge gained in the grants and apply it to other relevant environmental
research. Both students conducting research under STAR grants and the STAR fellows can become
part of this pool to transfer the research findings outside the originating institution. ORD could ask
grantees and fellows to keep EPA apprised of the employment whereabouts of such students, at least
through the first post-university job. The evaluation of the program's success then could include the
fraction of the STAR beneficiaries who move on to environment-relevant jobs in academia,
government, and the private sector.
Mission Advancement. Relative to the impact of EPA's research on decision making, The
EPA Strategic Plan states the following: "The performance goal is for EPA's research organization to
transfer information, findings, and results effectively to users, partners, and the public. Performance
measures include:
1. Technical results are disseminated in a timely way to users; and
2. Research is communicated appropriately in formats accessible to a variety of audiences and
users in their decision making" (EPA, 1997).
Thus, using the EPA Strategic Plan as guidance, the measures of success of the STAR Program relative
to mission advancement should address the timeliness and dissemination of the information to the users,
including the program offices, regional offices and EPA researchers. In addition, given the focus on
audiences and users, it is critical that program offices and other Agency customers for the grant
products be involved in the establishment of criteria for measuring the impacts of the program. The
Subcommittee recommends that NCER directly engage these customers, perhaps through a newly
constituted RCT, in defining the criteria for evaluating the STAR Program's contribution to Agency
mission advancement.
Ideally, ORD eventually would be able to show how information gained from the STAR
Program has enabled the Agency or other regulators to implement better risk assessment and
management decisions. Examples of mission advancement would include:
a) managing hazards that had not been previously identified as conferring significant risks;
23
-------
b) implementing more cost-effective remedies for known hazards; or
c) reducing the stringency of regulations for hazards that are found to be less risky than
previously thought.
The Subcommittee recognizes that such a "value of information" (VOI) approach is difficult to
apply to research that is often exploratory in nature or useful only over the long term. Nevertheless, the
Agency should keep this concept in mind when designing an evaluation system.
24
-------
In addition to the suggestions above, ORD should consider the following types of metrics and
data collection activities:
a) Conduct a peer review of the results and reports of a sample of STAR grantees in a
manner similar to that for the proposals, to see how the research actually rates with
respect to scientific quality (poor to excellent). Consider the NAS study as well as the
EPA Strategic Plan criteria of relevance, quality, and leadership to evaluate each
product.
b) Evaluate the use of information generated by each grant relative to the EPA and ORD
goals. Request that grantees include in their summary reports a self-assessment of how
data should or could be used to address strategic goals. This information would allow
ORD quickly to assess the relevance of the research product and would force
researchers to think about possible applications of their results.
c) Conduct a relevancy evaluation (perhaps by the RCT) of each grant after review of the
products to see if they remain relevant and why or why not.
d) Evaluate citations to STAR project publications in EPA regulatory documents as
another measure of STAR'S success with respect to the Agency's mission.
e) Define lessons learned from these assessments to suggest different proposal review
methods, RFA specifications, or interaction mechanisms that can improve the quality
and relevance of future efforts.
f>
Determine the time frame required for information from the grant program to reach the
program office, regional office, or researcher.
g) Poll customers within and outside the Agency regarding the value of STAR products.
Suggested questions to include in a such a questionnaire are shown in the box below.
When a sufficient database of questionnaire responses have been accumulated, it should
be analyzed to see how many STAR products have had a discernible impact on EPA
or other programs and how these impacts are distributed with respect to degree of
impact and size of program impacted.
Implementation Issues. NCER should seek assistance from program evaluation and
decision-analysis experts to help ORD develop a monitoring and evaluation system for the STAR (and
other Agency research) program. Development of this monitoring system should begin within a year.
The monitoring and evaluation system should provide the necessary data for a future evaluation of the
impacts of the STAR Program, both within and outside of the Agency. Such an evaluation could begin
in approximately two years, at which point sufficient time should have passed (approximately seven
25
-------
years) for the results from the initial cohorts of STAR grants to have been published and used. To
accomplish this future in-depth review of the program, EPA should plan on securing the services of a
qualified, highly respected, and independent organization to conduct and publish the evaluation of the
STAR Program. Both of these actions have budgetary implications. In particular, sufficient funds
should be budgeted to compile and analyze program data, prepare well-documented case studies and a
final report, and define continuous improvement techniques.
Suggested STAR Program Evaluation Questions for Program
Offices
Have STAR products influenced any of your risk management
programs?
If so, which products have influenced which programs?
For each of the top three STAR products, which statement most closely
matches its impact on your programs?
a) Played an essential role in modifying the direction of the
program;
b) Made an important contribution to directing the program;
c) Contributed to the direction of the program in a minor way.
For the programs affected by the above three STAR products,
quantify, to the extent possible, the size of the program in terms of
health and/or ecological benefits and costs of implementation.
26
-------
5. PRINCIPAL RECOMMENDATIONS
Keeping in mind the Subcommittee's overall favorable assessment of the STAR Program's
structure and management, recommendations for program management improvements are presented
below. Several of these recommendations refer to the need for stronger information transfer.
a) The Agency should provide additional information in RFAs on research goals and
objectives and on budget and relevancy criteria that will be used to evalute proposals in
order to raise the low proposal acceptance rate, increase the success rate of the best
quality applications, and enhance the relevance of the research.
b) The Agency should take steps to accelerate the peer review process for STAR results
so that the results are available to support Agency decisions in a timely manner.
c) The Agency should select several STAR research grants as case examples and evaluate
the effectiveness of the coordination with the relevant client offices and the degree to
which the awards are supporting the Agency's strategic goals. This could be part of a
broader evaluation of the effectiveness of current STAR Program communication,
technology transfer, and outreach efforts.
d) The Agency should consider means of strengthening communications between Agency
program staff and STAR grant recipients, such as: (1) meetings between principal
investigators and Agency program staff to discuss integration opportunities, and (2)
establishment of STAR points-of-contact at ORD laboratories and centers, as well as
in Agency program and regional offices.
e) The Agency should assess how well the needs and issues of the regional offices are
factored into the STAR planning process and consider additional mechanisms for
ensuring adequate regional involvement in STAR Program activities.
f) The Agency should request feedback on the success of the program review workshops
and should expand the workshop proceedings to include a record of discussions
regarding the relevancy of STAR results to the Agency's research and regulatory
agenda and to environmental decision-making.
g) Given the recent growth of STAR Program grants to academic researchers and the
evolving set of environmental issues facing the Agency, the Agency should develop and
implement a process for periodically assessing the Agency's portfolio in terms of its use
of different funding instruments (contracts, grants, inter-agency agreements, and
cooperative agreements) and the reliance on different R&D performers (universities,
27
-------
for-profit contractors, others).
h) The Agency should continue and expand its partnerships with other agencies and
funding organizations, including possible joint funding of STAR research with private
foundations and with international agencies and research organizations.
i) The Agency should seek assistance from program evaluation and decision-analysis
experts to help ORD develop a monitoring and evaluation system for the STAR
Program. The evaluation system should include measures of the STAR Program's
contributions to Sound Science (i.e., measures of science quality and quantity) and to
Mission Advancement.
j) The Agency should budget sufficient resources to secure the services of a qualified,
highly respected, and independent organization to conduct and publish an evaluation of
the STAR Program's results, effectiveness, and impact. This evaluation could begin in
approximately two years, at which point sufficient time should have passed
(approximately seven years) for the results from the initial cohorts of STAR grants to
have been published and used.
28
-------
6. REFERENCES CITED
Board of Scientific Counselors. 1998. Program Review of the National Center for Environmental
Research and Quality Assurance. EPA Office of Research and Development, Board of
Scientific Counselors, Final Report of the Ad Hoc Subcommittee on the Review of NCERQA.
April 30, 1998.
National Academy of Science. 1999. Evaluating Federal Research Programs. National Academy
Press, Washington, DC.
U.S. Environmental Protection Agency. 1996. Report to Congress: The Science to Achieve Results
(STAR) Program. EPA/600/R-96/064.
U.S. Environmental Protection Agency. 1997. EPA Strategic Plan. EPA/190-R-97-002.
U.S. Environmental Protection Agency. 1998. Peer Review Handbook. EPA 100-B-98-001.
Prepared by the Peer Review Advisory Group of the EPA Science Policy Council.
U.S. Environmental Protection Agency. 1999. Responses to Self-Study Questions: Board of Scientific
Counselors' Review of ORD's Science to Achieve Results (STAR) Program. Office of
Research and Development's National Center for Environmental Research and Quality
Assurance. December 1999.
U.S. Environmental Protection Agency. 2000. EPA's Science to Achieve Results (STAR). Briefing
Materials for the SAB/BOSC Joint STAR Review Subcommittee Meeting, held January 12-
13, 2000. Prepared by EPA/ORD/National Center for Environmental Research. Washington,
DC.
29
-------
APPENDIX A: NCERQA Web Site Statistics for December 1999
Visitors to the NCERQA web site have access to monthly reports containing a wide variety of
web site statistics, including visitor profiles and most requested pages. A sample of this information is
presented below for December 1999 as an indication of the number and nature of visitors to the site.
More in-depth analysis of the site's statistics would be helpful in evaluating the success of the web site
as a communication device for the STAR Program and as a source of data for evaluating the impact of
the program.
(Source: http://es.epa.gov/stats/ncerqa/ncerqa-Dec99 01 b.htm.)
Hits
Hits by Organization Type
Visitor Sessions
Entire Site (Successful)
Average Per Day
Company (.com)
Education (.edu)
Network (.net)
Government (.gov)
Visitor Sessions
Visitor Sessions from U.S.
International Visitor Sessions
Sessions of Unknown Origin
129,762
4,185
39.48%
29.36%
19.24%
9.01%
30,330
66.56%
11.76%
21.52%
Most Active Countries
1
2
3
4
5
6
7
8
Countries
United States
Canada
UK
Japan
France
Germany
Italy
Spain
Visitor Sessions
20,251
544
399
233
154
154
148
134
A-l
-------
9
10
AU
Netherlands
127
106
Most Requested Pages
1
2
3
4
11
15
Pages
Welcome to the National Center for
Environmental Research and Quality
Assurance http://es.epa.gov/ncerqa/
2000 Environmental Research Grant
Announcements
http : //es . epa. gov/ncerqa/rfa/
NCERQA Grants and Cooperative
Agreements
http://es.epa.gov/ncerqa/ grants/
Tabular Query of NCERQA Program
and Solicitation Information
http : //es . epa. go v/ncerqa/ru/
Science to Achieve Results (STAR)
Graduate - Fellowships
http ://es . epa. gov/ncerqa/fellow/
FY 2000 Science to Achieve Results
http://es.epa.gov/ncerqa/rfa/forms/dow
nlfhtml
Views
9,788
5,165
2,669
2,193
1,048
632
%of
Total
Views
8.47%
4.47%
2.31%
1.89%
0.9%
0.54%
Visitor
Sessions
6,788
4,650
2,541
2,038
983
582
Avg. Time
Viewed
00:00:48
00:01:40
00:00:27
00:02:03
00:00:58
00:04:05
A-2
-------
United States Science Advisory Board EPA-SAB-EC-00-008
Environmental (1400A) March 2000
Protection Agency Washington, DC
A JOINT SAB/BOSC
REPORT: REVIEW OF
THE SCIENCE TO
ACHIEVE RESULTS
(STAR) PROGRAM
PREPARED BY THE STAR
REVIEW SUBCOMMITTEE, A
JOINT SUBCOMMITTEE OF
THE SCIENCE ADVISORY
BOARD AND THE BOARD OF
SCIENTIFIC COUNSELORS
------- |