www.epa.gov/ord
HUMAN HEALTH RESEARCH PROGRAM
Overview of Human Health Research Program (HHRP):
Documentation of Progress and Strategic Directions for the Future
Prepared for the 2009 HHRP BOSC subcommittee
Sally Perreault Darney, Ph.D.
National Program Director (acting)
Executive Summary:
The Human Health Research Program (HHRP, www.epa.gov/hhrp ), one of ORD's
larger programs, is crossing-cutting by design and provides research results that EPA uses to
solve environmental health problems across environmental media. As articulated in the
HHRP Multi-Year Plan (MYP, 2006), the overall goal of this program is to characterize and
reduce uncertainties in extrapolations inherent in the risk assessment process by elucidating
the fundamental determinants of exposure and dose, and the basic biological changes that
result from exposure to environmental contaminants. The over-arching premise of the
program is that by defining and understanding the linkages in the exposure-to-effect
continuum, this program will enable EPA decision makers to predict risk and reduce harmful
exposures with increasing accuracy and precision. The program addresses four long term
goals (LTGs) using an interdisciplinary approach wherein scientists with expertise in many
disciplines (toxicology, systems biology, chemistry, exposure science, engineering, public
health, bioinformatics) work together across ORD Laboratories and Centers and as grantees
funded through ORD's Science to Achieve Results (STAR) program, to understand real
world risks in our communities and develop improved means by which EPA can evaluate the
effectiveness of its risk management decisions.
This documentation package supports ORD's second full review of the HHRP by the
Board of Scientific Counselors (BOSC). The review consists of two public conference calls
(October 10, 2008 and December 1, 2008) and a 3-day face-to-face meeting (January 13-15,
2009). The BOSC's charge is to "... conduct a retrospective and prospective review of
ORD's Human Health Research Program, and evaluate the program's relevance, quality,
performance, and scientific leadership." The materials in the documentation package,
including those in electronic format (see Table of Contents), are designed to provide
information pertinent to this charge, including the following:
• Goals and evolution of this program and its context within ORD and EPA.
• Research progress and accomplishments toward meeting its goals since the last full
BOSC review in February 2005 through 2008, which defines the current evaluation
period.
U.S. Environmental Protection Agency
Office of Research and Development
-------
• Impact of the Program for EPA and its partners as evident since the HHRP was
instituted in 1998, focusing on the current evaluation period.
• Changes made in response to recommendations from the mid-cycle BOSC in January
2007, and new drivers that have emerged more recently.
• Emerging drivers and issues that will influence future directions for this program in
anticipation of revising the program plan in 2009.
The scientific accomplishments of the program and their impact for EPA are
presented in detail during the second conference call and face-to-face meeting. The BOSC is
asked to provide a rating for each LTG; therefore this documentation is presented by LTG.
Senior scientists selected to coordinate the presentation of the documentation for each LTG
provide overview presentations accompanied by abstracts and corresponding posters for each
project. These LTG overview presentations highlight the regulatory bases and drivers for the
research program, illustrate how each goal is related to and informs the others, and show how
the program has impacted or is expected to impact EPA's decision making ability.
The poster sessions provide the HHRP BOSC subcommittee members the opportunity
to interact with the ORE) scientists who conduct the research. Posters form the "meat" of the
review and are constructed to convey the scientific questions, aims, approach, results and
impact of program projects. Unlike poster presentations at scientific meetings that report the
results of individual studies, the BOSC posters present programmatic goals and outputs on a
broad scale, telling a story that includes how intramural and extramural research addresses
the fundamental questions laid out in the MYP. The face-to-face meeting also includes oral
testimonials by some of the program's partners who describe the impact and usefulness of
HHRP research products for their organizations.
Because the HHRP addresses broad questions that cut across programs, its research
products are used by diverse partners within and outside EPA. Most of the partners listed
below are represented on HHRP's Research Coordinating Team and provide regular input to
insure that the HHRP remains responsive to their highest priority needs.
• EPA Program Offices, especially the Office of Air and Radiation (OAR), the Office
of Water (OW), and the Office of Prevention, Pesticides and Toxic Substances
(OPPTS), and EPA's Office of Children's Health Protection.
• EPA Regions (along with state and local government and health organizations)
through collaborations such as RARE and CARE projects.
• ORD's Research Programs, especially those with health components such as Clean
Air, Drinking Water, Safe Products/Safe Pesticides, and Endocrine Disrupters.
• ORD Centers:
o The National Center for Environmental Assessment (NCEA), with HHRP
providing data and models related to the goals of the Human Health Risk
Assessment MYP;
o The National Center for Computational Toxicology (NCCT), with HHRP
collaborating on projects related to the CompTox implementation plan.
• Federal partners, particularly at NIH: CDC, NICHD, NIEHS.
-------
This documentation package also provides information derived from a variety of
program assessment/evaluation approaches, some of which are relatively new to the BOSC
review process. For example, it includes the results of a survey that ORD conducted to
solicit feedback from the HHRP partners listed above regarding the relevancy and usefulness
of HHRP data, methods, models and advice for their regulatory and risk assessment needs. A
thorough bibliometric analysis of the HHRP publications since 1998, and an analysis of EPA
decision documents for citations of HHRP products, are also provided to help the BOSC
evaluate the quality and relevance of HHRP outputs. In addition, a summary of ORD's
internal tracking of annual performance goals with associated performance measures is
included as an indicator of the program's timeliness and accountability. These and other
measurement approaches are responsive to recommendations in the 2008 National Research
Council (NRC) report Evaluating Research Efficiency in the U.S. Environmental Protection
Agency.
As mentioned above, the BOSC is charged with conducting a review that is
prospective as well as retrospective. Hence this documentation package includes
information about new drivers that will influence the strategic directions of this program in
the future. Clearly NRC's report Toxicity Testing in the Twenty-first Century: A Vision and
a Strategy published in 2007 shortly after the mid-cycle BOSC, is a major driver for this
program and for ORD in general. Other imminent NRC reports expected to significantly
influence this program include: Health Risks ofPhthalates (expected in 2008 and directly
relevant to cumulative risk assessment), Improving Risk Analysis Approaches used by the US
EPA (anticipated in 2008), Public Health Decision-Making under Uncertainty (anticipated
in 2009), and A New Biology for the 21st Century: Ensuring that the United States Leads the
Coming Biology Revolution (anticipated in 2010). The fields of environmental health science
and risk assessment are on the cusp of a revolution in which information gathered in the past,
often on individual contaminants and under specific situations, will be integrated and
evaluated using a systems approach that takes advantage of advanced computer power and
modeling strategies. This documentation package will illustrate how ORD is positioning
itself to lead this revolution.
-------
Evolution of ORD's Human Health Research Program
The current HHRP MYP (2006), provided with this documentation package,
represents an extension and revision of the first formal HHRP MYP (2003); however, its
roots go back much farther in time. ORD has long recognized the need for research to
strengthen and improve EPA's risk assessment methods and approaches and reduce the
inherent uncertainties. In the 1990s EPA's National Health and Environmental Effects
Research Laboratory (NHEERL) formalized a program called "Research to improve health
risk assessment" with this specific goal in mind. Subsequently, a team of ORD scientists,
under the executive direction of Dr. Hal Zenick (Laboratory Director, NHEERL) developed
the Human Health Research Strategy document (2003) that identified and prioritized the
research needed to "improve the scientific foundation for health risk assessments." This
document (provided electronically) laid out two overall strategic directions that continue to
guide the program. From its formal inception as a major ORD research program, HHRP has
been viewed as a cross-cutting program involving multidisciplinary research with results and
products that are of general use to multiple EPA program offices and regions. Congress has
provided relatively stable funding for this program over the last ten years, indicative of its
support for a program of broad scope and that includes a significant extramural component.
The first full BOSC review of HHRP took place in February of 2005, evaluating
programmatic relevance, quality, performance and scientific leadership over the preceding
four to five years. Thus, the initiation date for the HHRP and the body of research defined by
its bibliography was somewhat arbitrarily set at 1998, although human health research has
been ongoing in ORD since the 1980s. During the previous performance period HHRP
included research conducted by the National Center for Environmental Research (NCEA).
Since that time NCEA, under the direction of Dr. Peter Preuss, has developed its own MYP
(www.epa.gov/hhra and is now considered an HHRP partner (user of HHRP research
findings). HHRP research is currently designed and prioritized to provide data, methods and
models to NCEA (along with its longer standing program office partners).
Shortly after the last BOSC review, ORD inaugurated the National Center for
Computational Toxicology (NCCT, www.epa.gov/comptox) under the direction of Dr.
Robert Kavlock. Some of the scientists working in HHRP and having expertise in modeling
structure-activity and dose-response relationships migrated to this new center, which is now
viewed as an HHRP partner. During this transition, HHRP scientists have continued to
collaborate with NCCT to provide data (particularly on molecular and genomic toxicity
pathways and dose response) for incorporation into NCCT models. With an increased focus
on toxicity pathways, application of systems biology approaches, and the development of
virtual models in both programs, this partnership between HHRP and NCCT is expected to
grow ever stronger in the future.
The current MYP (2006) reflects these changes. The four LTGs are similar to those
in the 2003 plan but reflect the increasing emphasis on the need to integrate data, methods,
and models relevant to individual compounds in order to address cumulative risk from
related chemicals (with respect to both exposure models and effects predictions) and to
extend those findings to assess community risks from a large variety of chemical and non-
-------
chemical stressors. The LTGs were also reformulated to focus on outcomes, i.e. use of the
research products by partners, and to address susceptibility issues based on life stage with
emphasis on children's health and impacts on aging populations. In addition, the 2006 plan
placed increased emphasis and resources on identifying indicators by which EPA could
evaluate the effectiveness of its risk management decisions. This shift is evident in research
conducted by ORD's intramural program and in the definition of research needs articulated
in NCER's Requests for Applications (RFAs). Detailed descriptions of these changes and all
responses to the 2005 BOSC review are provided in the materials for the 2007 mid-cycle
review which are included with this documentation package.
The MYP lays out a critical path for each LTG, including the identification of Annual
Performance Goals (APGs) as stepping stones along the path, and Annual Performance
Measures (APMs) as milestones to achieving the APGs. ORD has since revised its approach
for MYPs, recognizing that MYPs must be "living documents" based on the inherent nature
of scientific research. Thus, the APMs identified in 2006 have changed over time as the
research results come in and adjustments are made. ORD has decided that future MYPs will
provide APMs as an appendix that is updated on an annual basis considering research
findings, emerging research issues, budgetary constraints and changing Agency priorities.
ORD tracks the completion of APGs, along with the APMs contributing to those APGs. A
summary report is provided as a measure of program performance (see section on program
management and evaluation).
Achieving the ambitious goals set forth in the 2006 MYP requires both intramural
and extramural efforts and their integration. Over the last ten years, NCER's Science to
Achieve Results (STAR) program has developed RFAs
that are directly responsive to HHRP goals, and has
funded an array of outstanding grantees, either as
individual grants or as program projects. A spreadsheet is
provided with this documentation package that
summarizes these RFAs and provides electronic links to
program and project descriptions on NCER's website.
As will be detailed under LTG 3 presentations, the
Centers for Children's Environmental Health Research
program, co-funded with NIEHS, has a stellar record for
implementing epidemiological research on the impact of
environmental factors on children's health. The major
accomplishments of the first decade of this program were
recently synthesized in an NCER document "A Decade
of Children's Environmental Health Research" (a key
product included in this documentation package). This
Center program will transition into its second decade
with the release of a new RFA in November 2008. The
list of past, ongoing, and planned NCER RFAs funded by
HHRP illustrates the responsiveness of NCER to the
LTGs and objectives of the HHRP (see text box).
NCER RFAs funded by HHRP with primary L TG
identifier*
Centers for Children's Environmental Health
& Disease Prevention Research, 1998, 2001,
'03, '05, '08 (LTG 3, supports all LTGs)
Children's Vulnerability to Toxic Substances
in the Environment, 2001 (LTG 2&3)
Complex Mixtures, 2000 (LTG 2)
Biomarkers for the Assessment of Exposure
& Toxicity in Children, 2002 (LTG 3)
Lifestyle & Cultural Practices of Tribal
Populations & Risks from Toxic Substances in
the Environment, 2002, 2007 (LTG 2)
Application of Biomarkers to Environmental
Health & Risk Assessment, 2004 (LTG 2)
Early Indicators of Environmentally Induced
Disease, 2004 (LTG 3)
Interpretation of Biomarkers using
Physiologically Based Pharmacokinetic
Modeling, 2007 (LTG 2)
Development of Novel Environmental Health
Outcome Indicators, 2007 (LTG 4)
Planned: Community-based Cumulative Risk
Assessment (LTG 2)
Planned: Novel Approaches for Assessing
Exposure for School-Aged Children in
Longitudinal Studies (LTG 2 & 3)
'Black text indicates past RFA, red indicates
current, and blue indicates planned.
-------
The HHRP planning process insures that the extramural program solicits research that
the intramural program is either not well equipped to conduct, or that can best be conducted
in a cooperative manner. For example, a new RFA underdevelopment will solicit
epidemiological research to explore non-chemical stressors and economic and sociological
factors that impact vulnerability to environmental contaminants, and thereby capitalize upon
the broader expertise available in the academic community.
Changes in the Human Health Research Program in response to the mid-
cycle review (2007) and newly emerging drivers
HHRP underwent a mid-cycle BOSC review in January 2007. Based upon the
program's accomplishments (late 2004 through late 2006) and responsiveness to the
recommendations from the 2005 BOSC review, the BOSC rated HHRP "meets expectations"
in its report of July 23, 2007. That report included a list of new and continuing
recommendations to which ORD, in turn, responded in a report to the BOSC Executive
Committee in January 2008. The three documents associated with the mid-cycle BOSC are
provided for reference with this 2009 documentation package, and summarized below.
Many of the 2007 BOSC recommendations related to new directions and initiatives
described at the 2007 mid-cycle review. For example, the first recommendation encouraged
HHRP to increase involvement of its partners in research planning and to pursue the
initiatives described for LTG 4 on evaluating the effectiveness of risk management decisions
and on community based risk assessment (LTG 2). Overviews and posters under LTGs 2 and
4 of the current review provide details of HHRP-specific progress on these initiatives.
Seven of the 19 recommendations related to clarifying and/or increasing efforts under
LTG 4. In response, HHRP committed to: "... .continue to work toward greater partner
involvement in planning and evaluating research products and develop emerging research
areas such as community risk assessment and evaluation of public health impacts of risk
management decisions;.. .to work with its scientists and partners to define the scope of LTG
4 to reflect the growing emphasis on evaluating and demonstrating the impact of its research
on improving environmental health;" and "...on approaches to capture information on public
health impacts of risk management decisions;" and to ".. .broaden its mission statement to
reflect the greater diversity of information and participation necessary to achieve the
objectives of LTG 4."
Progress towards this end is detailed in the LTG 4 overview and six associated
posters. Briefly, HHRP contributed to the Health Chapter of the 2008 Report on the
Environment in which the need for better environmental health indicators and research on
evaluating public health impacts is emphasized. This chapter now serves as a major driver
for future HHRP planning. HHRP scientists also developed a document entitled A
Framework for Assessing the Public Health Impacts of Risk Management Decisions (2007)
and held a workshop in January 2008 on this topic. This workshop featured updates on two
pilot projects currently underway in collaboration with scientists in Region 1, and grants
funded under a 2007 NCER RFA on this topic (Development of Novel Environmental Health
-------
Outcome Indicators). Another RFA on this topic is under consideration for release in 2009
reflecting the increased emphasis on LTG 4. Consideration is being given to funding some
of these projects as cooperative agreements whereby intramural scientists can collaborate
with STAR grantees.
HHRP appreciates that the regions encounter real world exposures wherein ORD's
advice and assistance is particularly relevant. HHRP scientists are currently collaborating on
several projects in ORD's Regionally Applied Research Effort or RARE program. HHRP is
also initiating meetings with ORD's Office of Science Policy (OSP) to foster collaborations
between HHRP's intramural scientists and EPA's regional scientists related to regional
needs. Furthermore, HHRP is developing a workshop in collaboration with Region 5 on
cumulative risk assessment (planned for late 2009) that is obviously relevant to LTG 2. In
summary, HHRP has expanded its focus and efforts in LTGs 2 and 4 and is seeking
opportunities for leveraging these efforts with other organizations. Recommendation 14 was
to obtain additional resources for demonstration projects in LTG 4. In 2008, funding added
to the ongoing projects was made possible by a congressional add-back to the Human Health,
and future plans will depend upon the findings of the two pilot projects. However, funding
for new projects is uncertain. Therefore HHRP is adding emphasis on partnering with
ongoing studies or using existing data to address this goal. Especially for observational and
epidemiological studies, the STAR program can serves as a platform for such collaborative
research. Future emphasis on the potential impacts of global warming or biofuels on human
health may provide additional opportunities for new projects under LTG 4.
The NRC released its report on Toxicity Testing in the Twenty-first Century: A Vision
and a Strategy (2007) about six months after the HHRP mid-cycle review. In response to
that report and as will be detailed under LTG 1, EPA has entered into a memorandum of
understanding with the National Human Genome Research Institute and the National
Toxicology Program to collaborate in efforts to transform toxicity testing. The approach is to
develop and validate high through put assays and elucidate molecular pathways of toxicity to
improve the efficiency and effectiveness to testing, and to conduct targeted in vivo research
to anchor the predictive power of the high through put systems. HHRP will participate in
this effort in cooperation with NCCT. This new effort, including the movement towards
incorporating a systems biology approach, is detailed further in the last three posters under
LTG1.
The mid-cycle BOSC also suggested that the HHRP mission statement should be
reformulated beyond addressing uncertainty in risk assessment to better convey the
objectives articulated in LTG 4. LTG 4 is envisioned to design tools at the relatively local
level to address regional and national problems, extrapolating from the laboratory to the real
world and from the individual to the population level. These concepts will be carried over
into planning activities for revising the HHRP in 2009, taking under consideration additional
input from the current BOSC subcommittee.
-------
Program Evaluation Measures
Most of the other recommendations from the 2007 mid-cycle BOSC related to the
need for better articulation of the process through which the program involves partners in
planning, better ways to evaluate research products for relevance and quality, clear
performance-based measures, and better articulation of investment criteria, i.e. how HHRP
determines the relative effort in each LTG, and between the intramural and extramural
programs.
Involvement of partners in program planning and evaluation:
Ongoing since the last review, HHRP continues to communicate its program
accomplishments and solicit input from its partners through several channels. One of these is
via regular (monthly) conference calls with the HHRP Research Coordination Team (RCT).
The HHRP includes representatives from partners in ORD labs and centers, Program Offices
and Regions
HHRP Research Coordinating Committee, 2008 membership
Sally Darney, ORD, NPD (Acting)
Carlos Nunez, Assistant Laboratory Director for Health, NRMRL
Ross Highsmith, Assistant Laboratory Director for Health, NERL
Andrew Geller, Assistant Laboratory Director for Health, NHEERL
Devon Payne-Sturges, Assistant Center Director for Health, NCER
Stan Barone, Assistant Center Director for Health, NCEA
Jerry Blancato, Deputy Director, NCCT
Ray Putnam (Region 1)
Marian Olsen (Region 2)
Ravi Rao (Region 4)
David Macarus (Region 5)
Lesley Vazquez-Coriano, Santhini Ramasamy, Crystal Rogers-Jenkins, Kesha Forest,
Sandhya Parshionikar, OW
Michael Firestone, Office of Children's Health Protection and Environmental
Education (OCHPEE)
Scott Jenkins, OAR
Jeff Evans, AnnaLowit, OPPTS.
HHRP is a cross-cutting program and as such, it has multiple and diverse partners,
each with its own particular research needs. Because the problems addressed by HHRP are
long term in nature, and not specific to a particular program office, immediate benefits of the
research program may be less apparent to certain partners. Therefore, where possible,
efforts are being made to target specific program elements to specific partners.. For example,
one of the projects in LTG 2 is an interdisciplinary effort underway in collaboration with
OPPTS to conduct research in support of OPPTS' upcoming cumulative risk assessment for
pyrethroid pesticides. While long term in nature, and designed to provide methods and
models for cumulative risk assessment that can be used by any program office, this project is
-------
staged to provide specific information for a series of meetings scheduled by OPPTS's
Science Advisory Panel on Pyrethroids. The study team includes a member of OPPTS's
pyrethroids team (who is also a member of the HHRP RCT). This insures that HHRP
products will be responsive to the program office's needs and delivered in timely fashion.
As mentioned above, HHRP also considers seeking solutions to EPA's regional
offices' "real world" problems to be of high priority. The NPD and ALDs are expanding
efforts to visit with partners in EPA regions (e.g., NPD spent a day with R3 recently), and to
participate in program office meetings in partnership with the NPDs for the targeted research
programs. For example the HHRP will participate in upcoming senior management meetings
with OAR (December 2008), and OW and OPPTS (early 2009), along with the program-
specific NPDs. Also, HHRP Assistant Laboratory Directors gave presentations on program
office relevant aspects of HHRP when representatives from OPPTS, OW and NCEA visited
the laboratories this past year. Furthermore, HHRP contributed research highlights to the
Office of Children's Health Protection for incorporation into its 2007 and 2008 Annual
Children's Health Highlights reports.
Performance-based measures of program relevancy, quality and performance
As mentioned in the response to the mid-cycle BOSC ORD developed a partner
survey in 2008 and sent it to more than 200 partners in EPA program offices, regions, ORD
Centers (NCEA and NCCT), and others. Examples of key research products (including those
provided in this documentation package) were listed by LTG (with electronic links to these
products) and partners were asked to score the relative usefulness and quality of these
products. A summary report of the survey findings is provided and shows that HHRP
partners consider its key products both useful and relevant to their needs, although the extent
to which this is true varies somewhat by LTG. Feedback from the BOSC on the usefulness
of this evaluation tool is welcome.
Peer reviewed journal articles are an established currency of research quality and
program productivity. In fact, only peer reviewed data can be used in risk assessments. In
addition, periodic reviews of the literature provide valuable summations of the state of the
science and help to identify data gaps and rationale for research planning and prioritization.
Likewise, reviews or syntheses of the results of multiple HHRP products at the end of a
project, as when an Annual Performance Goal is met, also serve as indicators of program
performance and effectiveness. The latter can sometimes be more useful to EPA partners
and decision makers than the individual, incremental research papers. An updated
bibliography of papers and chapters published under the HHRP program is included with this
documentation package. It is provided in electronic format with links to the abstracts in
PubMed where the full articles are often available.
To evaluate the quality of these publications, ORD conducted a thorough
Bibliometric analysis of all HHRP publications (peer reviewed journal articles) published
from 1998 through the first half of 2008 inclusive. A similar analysis was conducted for the
mid-cycle BOSC. This analysis reports the frequency and timeliness with which HHRP
papers are cited in the scientific community and the quality of the journals in which they are
-------
published. The full report is included with this documentation package. The following
summary indicates that HHRP publications continue to be of high quality and many are
highly cited.
Summary of 2008 Bibliometric Analysis
More than one-fourth of the 2,520 Human Health journal publications are highly cited papers. 644
(25.6%) of the 2,520 Human Health journal publications qualify as highly cited when using the ESI criteria
for the top 10% of highly cited publications. This is 2.6 times the number expected. 88 (3.5%) of the
Human Health journal papers qualify as highly cited when using the ESI criteria for the top 1%, which is 3.5
times the number expected. 10 (0.4%) of the Human Health publications qualify as very highly cited when
using the criteria the ESI criteria for the top 0.1% of highly cited publications, which is 4 times higher than
the number expected. 2 (0.1%) of the Human Health publications qualify as extremely highly cited when
using the criteria for the top 0.01% threshold for the most highly cited papers. This number is 10 times the
number of papers expected to meet this highest threshold.
The Human Health journal publications are more highly cited than the average paper. Using the ESI
average citation rates for papers published by field as the benchmark, in 16 of the 21 fields in which the
2,520 Human Health journal papers were published, the ratio of actual to expected cites is greater than 1,
indicating that the Human Health journal publications are more highly cited than the average papers in those
fields. For all 21 fields combined, the ratio of total number of cites to the total number of expected cites
(47,067 to 27,118.1) is 1.7, indicating that the Human Health journal papers are more highly cited than the
average paper.
More than one-half of the Human Health journal papers are published in high impact journals as
determined by Impact Factor. 1,287 of the 2,520 journal papers were published in the top 10% of journals
ranked by JCR Impact Factor, representing 51.1% of the Human Health journal publications. This number is
5.1 times higher than expected.
More than one-third of the Human Health journal papers are published in high impact journals as
determined by Immediacy Index. 1,027 of the 2,520 papers appear in the top 10% of journals ranked by
JCR Immediacy Index, representing 40.8% of EPA's Human Health journal publications. This number is 4.1
times higher than expected.
There were 15 hot papers among the 2,520 Human Health publications. Using the hot paper thresholds
established by ES/ as a benchmark, 15 (0.6%) hot papers were identified in the analysis. This is six times
higher than the number expected. Hot papers are papers that are highly cited shortly after they are
published.
The authors of the Human Health journal publications cite themselves much less than the average
author. 1,732 of the 47,067 total cites are author self-cites. This 3.7% author self-citation rate is well below
the accepted range of 10-30% author self-citation rate.
78 (1.1%) of the 6,882 authors of the Human Health journal publications are included in
ISIHighlyCited.com, which is a database of the world's most influential researchers who have made key
contributions to science and technology during the period from 1981 to 1999.
The 81 nonjournal publications were cited 220 times in journals and the authors cited themselves 16 times
(7.3% self-citation rate), which is less than the literature-reported 10-30% range for author self-citation.
10
-------
As shown in the next table, the results of this recent analysis compare favorably with
those of the previous analysis conducted in 2006, as might be expected because the analysis
is inclusive of the papers evaluated at that time (1998- most of 2006).
Comparison of the 2006 Human Health Research Program Bibliometric Analysis
Results and the 2008 Human Health Research Program Bibliometric Analysis Results
Bibliometric Analysis
No. of Papers Analyzed
No. of Papers Cited At Least Once in a Journal
Highly Cited Publications (Top 10% Threshold)
Highly Cited Publications (Top 1% Threshold)
Highly Cited Publications (Top 0.1% Threshold)
Highly Cited Publications (Top 0.01% Threshold)
No. of Times Cited
Expected No. of Times Cited
Ratio of Actual Cited to Expected Cites
Papers in High Impact Journals by Impact Factor
Papers in High Impact Journals by Immediacy
Index
No. of Author Self Cites (%)
No. of Hot Papers (%)
Human Health
2006
1,835
1,561 (85%)
462 (25.2%)
64(3.5%)
6 (0.3%)
0 (0%)
22,937
13,742
1.7
932 (50.8%)
938(51.1%)
992 (4.3%)
15 (0.8%)
Human Health
2008
2,520
2,249 (89%)
644 (25.6%)
88 (3.5%)
10 (0.4%)
2(0.1%)
47,067
27,118
1.7
1,287(51.1%)
1,027 (40.8%)
1,732(3.7%)
15 (0.6%)
The following chart summarizes the number of papers published since 2004 arrayed
by LTG. Note that the number for 2008 captures only about half the year. Also research
products for LTG 4 are not anticipated yet because the projects are still underway. Also note
that the number of papers is highest for LTG 3, reflecting the outputs of the Children's
Environmental Health Centers, many of which also address problems in the other LTGs.
LTG 3 accounts largely for discrepancies in the total number of papers each year. These
numerical outputs are consistent with the allocation of FTE and resources by LTG as well
(see below).
11
-------
Number of HHRP papers published per year (2004-08)
2004 2005 2006 2007
2008
Long-Term Goal 1:
Use of Mechanistic
Information in Risk
Assessment
Long-Term Goal 2:
Aggregate/Cumul ati ve
Risk
Long-Term Goal 3:
Susceptible Populations
Long-Term Goal 4:
Evaluate Public Health
Outcomes
Totals for Each Year
87
49
141
3
280
90
52
234
2
378
92
24
258
10
384
51
30
145
5
231
24
20
85
3
132
A Bibliometric analysis does not necessarily indicate the relevancy of the papers to
EPA's needs and their use in the conduct of risk assessments, setting regulations or justifying
risk mitigation. Therefore, ORD is also "mining" EPA dockets and other toxicology and risk
assessment documents for citations of HHRP papers. The results of this analysis will be
included in the final documentation package.
Another indication of scientific quality and excellence is the credentials of the
scientific staff who contribute to this HHRP. Over 120 principal investigators contributed to
the research profiled in the posters. Their biosketches are provided in electronic format for
reference and as documentation of their expertise, productivity and leadership. A list of these
scientists with their position titles is also included to illustrate the diversity of expertise and
capabilities in ORD laboratories and centers upon which HHRP projects can draw. This
expertise is necessary for multi-disciplinary research that addresses the full range of
processes from exposure to effect.
The BOSC is also charged with evaluating the scientific leadership of this program.
A summary report of leadership activities conducted by HHRP scientists is provided in order
to highlight their contributions to the scientific community, and advice and assistance
provided to the agency. Mentorship of pre- and post-doctoral students is also considered a
measure of program effectiveness because one of ORD's goals is to train the next generation
of environmental scientists, including future scientific leaders for EPA. A list of trainees
12
-------
conducting research funded by HHRP that includes their current place of employment shows
the effectiveness of HHRP mentors in this regard.
Programmatic leadership at the national and international levels is important and the
mid-cycle BOSC asked for more information on the extent to which HHRP research was
coordinated with other research organizations these levels. This question relates to both
leadership and efficiency measures since it is obviously desirable for EPA to partner
effectively in accomplishing its research mission, avoiding overlap with the efforts of other
agencies, while strategically addressing the problems of most importance to EPA. A report is
provided with examples of where this is being done at the HHRP and ORD level by
interactions with organizations such as the Office of Economic Cooperation and
Development (OECD) and leadership in planning international meetings with such
organizations. Many scientists who contribute to HHRP research also contribute to targeted
programs such as ORD's Drinking Water, Safe Pesticides/Safe Products, Endocrine
Disrupters and Clean Air programs. Thus their participation on work groups and higher level
harmonization and decision-making activities on the national and international scale reflects
positively upon ORD.
In summary, a variety of program evaluation measures are provided with this
documentation package. Nearly all of them are indicative of program performance. On the
other hand, some of them are also specific indicators of program relevancy, quality and/or
leadership as conveyed in the following chart.
Documentation
MYP
HHRP overview
APM/APG fact sheet
NCER RF A summary
LTG overviews
Abstracts and Posters
Bibliometric Analysis
Bibliography
Biosketches
Decision document analysis
Partner Survey Report
Advice & Assistance to EPA
Leadership in Scientific
Community
Trainee Summary
Key Products
Relevance
X
X
X
X
X
X
X
X
X
Quality
X
X
X
X
X
Performance
X
X
X
X
X
X
X
X
X
X
X
X
Leadership
X
X
X
X
X
X
X
X
X
X
13
-------
Investment Criteria:
Congress allocates human and capital resources to the HHRP under the program area
Health and Ecological Research. As presented at the first conference call (October 10,
2008), and shown in the following graph, the allocations for HHRP have been relatively
stable since 2005 and include ~%16 million/year for NCER (grants program). However,
salaries have increased as have costs of laboratory equipment and supplies. Once these fixed
costs are considered the amount of funds available for contract and cooperative agreements is
greatly diminished. Also, funds available in the 2009 President's Budget (under the current
continuing resolution) are lower than the 2008 enacted budget. As a consequence, the
conduct of some projects, particularly observational and epidemiological studies that depend
on contracts and cooperative agreements, will be impacted.
History of funding for HHRP (2003-2008)
Funding levels reflect total program including payroll, travel, working capital fund, and operating
expenses. STAR: Science to Achieve Results extramural grants
The following pie charts convey the allocation of HHRP resources by LTG. These charts
show a relative increase in allocation to LTG 4, consistent with the increased emphasis on
this goal, with a corresponding decrease to LTG 3, although LTG 3 remains the most heavily
invested LTG. The higher investment in LTG 3 is in part a reflection of EPA's continued
commitment to protecting children's health through research conducted by the EPA-NIEHS
co-sponsored Children's Environmental Health Research Centers program.
14
-------
FY 2007 Enacted
$60.9 M
FY 2008 Enacted
$63.2 M
FY 2009 President's Budget $56.3 M
These charts are consistent with the allocation of FTEs by LTG shown below. The
number of FTEs allocated to LTG 4 has increased modestly with the institution of the pilot
projects, although the total investment is low. Funding of the STAR RFA Development of
Novel Environmental Health Outcome Indicators has increased research effort in this goal
without increasing the number of federal FTE associated with it. That is, the federal FTE
allocations do not reflect the number of grantees engaged in the research. The level of effort
in LTG 1 is relatively stable. This number does not, however, reflect an increasing number
of FTE in NCCT who are engaged in collaborative research with FfflRP scientists. The
decrease in FTE for LTG 3 reflects a modest decrease in the intramural program balanced by
a modest increase in LTG 2 efforts reflecting an increased emphasis on community exposure
assessment.
15
-------
Summary of HHRP FTE by LTG, 2007-09
FY 2007 Enacted FY 2008 Enacted FY 2009 Request
Total 187.9 185.8 192.7
Performance Goals.
As described earlier, the HHRP MYP (2006) diagrams the intermediary steps towards
achieving its LTGs, articulating and tracking these as Annual Performance Goals (APGs).
Ideally the APGs are planned to culminate in a plenary synthesis document (or "key
product"). ORD tracks these APGs and the APMs within them to provide a measure of
program performance. As summarized below, the HHRP has completed all APGs scheduled
since the last full BOSC review.
Human Health Research Program, Annual Performance Goals, 2005-2008
APGs represent a program's major milestones toward accomplishing its long-term goals.
The following charts outline the program's success in meeting its planned APGs on time.
FY 2005
Products
APG Title
Status Associated Key
Provide risk assessors and managers with methods
and tools for measuring exposure and effects in
children, characterizing risk to children, and
reducing risks to children in schools from harmful
environmental agents to support EPA risk
assessment and risk management.
Met
FY 2006
Develop measurement-based models and modules
that represent the source-exposure-dose-effect
relationships for aggregate exposures.
Met
16
-------
Develop partnerships for data collection and new
indicators and metrics for exposure and health
effects.
Met
FY 2007
Evaluate the variation in vulnerability to
environmental agents as a result of health status as
reflected by nutritional status and pre-existing
disease.
Develop Health Chapter for Report on the
Environment.
Provide Assessment and Analytical tools for
Region and Program Offices in using biomarkers
for assessing and addressing cumulative risks.
Met
Met
Met
EPA' s Report on the
Environment, 2008
Biomonitoring Workshop
Report, 2007; Community
Based Risk Assessment
Needs Workshop Report,
2007
FY 2008
Identify PK/PD issues underlying uncertainties for
extrapolation
Identify MO As for Risk Assessment
Provide refined models, methods and guidelines
for assessing aggregate exposures and effects
Develop methods for longitudinal research.
Identify pharmacokinetic / pharmacodynamic
issues underlying uncertainties for extrapolation.
Determine utility of emerging technologies in
harmonizing risk assessment.
By 2008, provide risk assessors and managers with
methods and tools for assessing differences in
exposures and responses to harmful environmental
agents between the elderly and younger adults.
Provide improved tools, systems, methods, models
into framework by which EPA and others can
measure or model changes in public health from
risk management options.
Provide Refined models, methods, and guidelines
for assessing aggregate exposures and effects.
By 2008, analyze and demonstrate the role of
genetic factors in causing cancer and non-cancer
endpoints.
Met
Met
Met
Met
Met
Met
Met
Met
Met
Met
Important Exposure
Factors for Children,
2007
EPA contributions to
NCS Research Plan, 2008
A Decade of Children's
Environmental Health
Research, 2007
Framework for Assessing
the Public Health Impacts
of Risk Management
Decisions, 2007
Community Based Risk
Assessment Needs
workshop, 2007
17
-------
ORD's Office of Management and Accountability tracks performance based on the
timely completion of all Annual Performance Measures (APMs) that contribute to the APGs,
and sets future targets for program progress on the other measures of program performance.
The following report indicates that HHRP has achieved its performance objectives during the
current rating period (2005- 2008).
Summary of HHRP Long-Term Outcome Measures: BOSC ratings
Measure
2007 Baseline
2009 Target
Rating of the appropriateness, quality, and use
of ORD methods and models for risk assessors
and risk managers to evaluate the
effectiveness of public health outcomes.
Meets
Expectations
(Mid-Cycle
program rating*)
Exceeds
Expectations
Rating of the appropriateness, quality, and use
of ORD methods, model, and data for risk
assessors and risk managers to characterize
aggregate and cumulative risk in order to
manage risk of humans exposed to multiple
environmental stressors.
Meets
Expectations
(Mid-Cycle
program rating*)
Exceeds
Expectations
Rating of the appropriateness, quality, and use
of ORD methods, models, and data for risk
assessors and risk managers to use
mechanistic (mode of action) information to
reduce uncertainty in risk assessment.
Meets
Expectations
(Mid-Cycle
program rating*)
Exceeds
Expectations
Rating of the appropriateness, quality, and use
of ORD methods, models, and data for risk
assessors and risk managers to characterize
and provide adequate protection for
susceptible subpopulations.
Meets
Expectations
(Mid-Cycle
program rating*)
Exceeds
Expectations
*Mid-Cycle program ratings are not formal data points.
Decision Document Analysis Measures
Through analyses of partner documents, the program assesses the extent to which partners
use ORD science in decision-making. This process was first applied for the 2005 BOSC
providing a baseline for the program. An improved process is underway, considering the
updated HHRP bibliography and will be provided to the BOSC subcommittee before the
Face-to-Fact meeting.
18
-------
Measure
Percentage of peer-reviewed EPA risk assessments
in which ORD's characterization of
aggregate/cumulative risk is cited as supporting a
decision to move away from or to apply default
risk assessment assumptions.
Percentage of peer-reviewed EPA risk assessments
in which ORD's mechanistic information is cited
as supporting a decision to move away from or to
apply default risk assessment assumptions.
Percentage of peer-reviewed EPA risk assessments
in which ORD's methods, models or data for
assessing risk to susceptible subpopulations is
cited as supporting a decision to move away from
or to apply default risk assessment assumptions.
2005
Baseline
5%
15%
3%
2009
Target
5.5%
16.5%
3.5%
2013
Target
6%
18%
4%
Bibliometric Analysis Measures
As mentioned above, the bibliometric analyses conducted every two years reports, among
other measures, the extent to which the scientific community cites HHRP publications. This
indicator is being tracked over time. The percentage based on the 2008 analysis shows a
slight increase in this measure.
Measure
Percentage of Human Health
program publications rated as
highly cited papers in research
journals.
2005
Baseline
24%
2006
Actual
25%
2008
Actual
25.6%
2010
Target
26.5%
Annual Output Measures
At the end of each fiscal year, the program reports on its success in completing its planned
annual outputs (APMs). This chart shows the HHRP has consistently met its APM
commitments.
Percentage of planned outputs delivered in support of mechanistic data long term goal.
2000
Baseline
100%
2001
Actual
100%
2002
Actual
100%
2003
Actual
100%
2004
Actual
100%
2005
Actual
93%
2006
Actual
92%
2007
Actual
100%
2008
Actual
100%
19
-------
Percentage of planned outputs delivered in support of the cumulative risk long term goal.
2000
Baseline
80%
2001
Actual
83%
2002
Actual
80%
2003
Actual
87%
2004
Actual
88%
2005
Actual
86%
2006
Actual
100%
2007
Actual
100%
2008
Actual
100%
Percentage of planned outputs delivered in support of the susceptible populations long term
goal.
2000
Baseline
100%
2001
Actual
100%
2002
Actual
100%
2003
Actual
93%
2004
Actual
98%
2005
Actual
100%
2006
Actual
100%
2007
Actual
100%
2008
Actual
100%
Percentage of planned outputs delivered in support of the public health outcomes long term
goal.
2000
Baseline
100%
2001
Actual
100%
2002
Actual
100%
2003
Actual
100%
2004
Actual
100%
2005
Actual
100%
2006
Actual
100%
2007
Actual
100%
2008
Actual
100%
Annual Efficiency Measure
To monitor and improve its efficiency in grants processing, the program reports
annually on grants processing time. In 2008 NCER met its goal of processing grants in less
time than in preceding years. Note: The program may replace or alter this measure as ORD
considers recent National Academy of Sciences (NAS) recommendations on research
efficiency measurement.
Average time (in days) to process research grant proposals from RFA closure to submittal to
EPA's Grants Administration Division.
2003 Actual
405
2004 Actual
350
2005 Actual
340
2006 Actual
277
2007 Actual
254
2008 Actual
250
Strategic Directions for the Future:
By definition, research is unpredictable and dynamic. So, too, are the challenges facing EPA
because the environment and the contaminants entering it are constantly in flux. Thus EPA
frequently calls upon the NRC to research and report upon environmental and risk
assessment issues of paramount importance. The resulting NRC reports help ORD determine
its priorities and strategic research directions.
A number of NRC reports relevant to HHRP goals are anticipated in the near future (0 to 18
months from the time of the HHRP BOSC review), as detailed on the NRC website.
1. An NRC committee has reviewed the human health risks and the potential for conducting
a cumulative risk assessment for phthalate esters. The review included critical scientific data
and the report, expect in December 2008 will address questions related to human relevance
of experimental data, modes of action, exposure information, dose-response assessment, and
20
-------
the potential for cumulative effects. It is expected to include discussion of the strengths and
weaknesses of cumulative assessment approaches that will be informative and directly
relevant to efforts in HHRP LTG 2.
2. NRC was also asked in 2006 to develop recommendations for Improving Risk Analysis
Approaches used by the US EPA, based upon a thorough review of current concepts and
practices. The report will be comprehensive and address analyses applied to contaminants in
all environmental media (water, air, food, soil) and all routes of exposure (ingestion,
inhalation, and dermal absorption). The report is expected to have broad implications for
human health risk assessment and to comment upon many topics under study in HHRP
including quantitative characterization of uncertainty and the use of sound science to derive
uncertainty factors, approaches for assessing cumulative risk, variability in susceptible and
sensitive populations, and dose-response relationships. .
3. Another committee began work in 2007 on the topic Public Health Decision-Making
Under Uncertainty. This committee will prepare a report for EPA on decision-making about
environmental threats to human health under various types of uncertainty.
4. A longer-term and broad-based effort initiated in 2008 will focus on " A New Biology for
the 21st Century: Ensuring that the United States Leads the Coming Biology Revolution."
This committee will address all-encompassing questions of central importance for EPA and
other Federal Agencies:
"How can a fundamental understanding of living systems reduce uncertainty about the future
of life on earth, improve human health and welfare, and lead to the wise stewardship of our
planet? Can the consequences of environmental, stochastic or genetic changes be understood
in terms of the related properties of robustness and fragility inherent in all biological
systems?
" How can federal agencies more effectively leverage their investments in biological research
and education to address complex problems across scales of analysis from basic to applied?
In what areas would near term investment be most likely to lead to substantial long-term
benefit and a strong, competitive advantage for the United States? Are there high-risk, high
pay-off areas that deserve serious consideration for seed funding?
" What federal initiatives could be considered to ensure that the US is positioned to take
maximum advantage of a vast increase in biological data and understanding, and position
itself to be the leader in technologies derived from it? Is the biology research portfolio
appropriately balanced among biology subdisciplines and new areas that cross traditional
biology subdisciplines? Are new funding mechanisms needed to encourage and support
cross-cutting, interdisciplinary or applied biology research?
ORD, in concert with its partners, is discussing new ways to mobilize its talents to
address broad problems of national significance in response to environmental challenges
faced by a world impacted by the activities and resource utilization of our ever-growing
human population. Furthermore we face fundamental changes in global climate with many
impacts on human health and ecosystems predicted. These issues, coupled with revolutions
in genomics and toxicology testing forecast by the NRC in its 2007 report mentioned earlier,
will be major drivers as ORD shapes its future research programs, including HHRP. Several
changes in approach have already been discussed, including an increased emphasis on using
21
-------
a systems-based approach in toxicology, incorporation of advanced modeling approaches in
exposure assessment, and development of improved indicators of public health for use in
evaluating the effectiveness of EPA's risk management decisions. We face these challenges
as research budgets are diminished, motivating increased efforts to collaborate with other
agencies and partners in order to leverage limited resources and take advantage of combined
expertise. As ORD revises its fundamental research program in human health, the priorities
of the new US administration will also be brought into play. The recommendations of this
BOSC subcommittee are more important now than ever before.
Acknowledgements:
Many ORD scientists, program managers and support staff deserve recognition for
their contributions to the preparation of the HHRP documentation package. The leadership
of the LTG leads who worked with teams of scientists to organize the poster sessions and
prepare insightful LTG overviews was central to this review:
Julian Preston, Associate Laboratory Director for Health, NHEERL, LTG 1
Linda Sheldon, Associate Laboratory Director for Health, NERL and Ross Highsmith,
Assistant Laboratory Director for Health, NERL, LTG 2
Devon Payne-Sturges, Assistant Laboratory Director for Health, NCER, LTG 3
Andrew Geller, Assistant Laboratory Director for Health, NHEERL, LTG 4
A list of poster titles with presenting authors is included in the documentation
package. These scientists from ORD Labs and Centers are acknowledged for working with
ORD principal investigators and selected grantees to develop an integrated set of posters for
each LTG and insure their completeness. Over 100 ORD scientists contributed to the
abstracts and posters (see list of contributors). Many more bench scientists generated the
data behind the posters, and many support staff contributed to the generation of the
documentation materials.
Program evaluation tools such as the partner survey, report tracking completion of
annual performance goals and measures, and the bibliometric and decision document
analyses were developed in cooperation with ORD's Office of Research Management and
Accountability, particularly Philip Jeungst, Mya Sjorgren and Julie Hyman. Myles Morse
(NCER) coordinated the Bibliometric and decision document analyses.
Behind the scenes, IO staffers Laurel Schultz and Shirley Jarvis and student
contractor Christian Hughes provided invaluable programmatic support and assistance.
Finally, graphics support from NHEERL's graphics contractor insured optimal visual
presentation of the information in poster format.
22
------- |