PRO Office of Research and Development's Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (Final report received December 2009) June 2010 BOSC Human Health Research Program Subcommittee: James E. Klaunig (Chair), Indiana University Henry Falk (Vice-Chair) — Centers for Disease Control and Prevention Paul D. Blanc — University of California San Francisco George P. Daston — The Procter & Gamble Company David G. Hoel — Medical University of South Carolina Donald Mattison - National Institutes of Health, NICHD Edo Pellizzari — RTI International Christopher J. Portier — National Institute of Environmental Health Sciences Joel Schwartz — Harvard University School of Public Health Submitted by: Sally Perreault Damey, PhD National Program Director for Human Health Office of Research and Development ------- UNITED STATES ENVIRONMENTAL PROTECTION AGENCY WASHINGTON, D.C. 20460 OFFICE OF RESEARCH AND DEVELOPMENT Gary S. Sayler, Ph.D Chair, Board of Scientific Counselors Center for Environmental Technology The University of Tennessee 676 Dabney Hall Knoxville, Tennessee 37996 Dear Dr. Sayler: On January 13-15, 2009, the Human Health Research Program Subcommittee of the Board of Scientific Counselors (BOSC) met in Research Triangle Park, North Carolina to evaluate the Office of Research and Development's (ORD) Human Health Research Program (HHRP). The Subcommittee presented a report of its findings and recommendations to the Executive Committee of the BOSC on August 6, 2009, and the Executive Committee, in turn, provided a final BOSC report to ORD on December 1, 2009. With this letter, I am pleased to enclose the Agency's response to the final BOSC report of its review of the HHRP. The Human Health Research Program greatly appreciates the insights, advice, and recommendations offered by the BOSC. The attached document presents an overview of specific recommendations made by the BOSC and provides ORD's response to each of the recommendations and a timeline for action. It also includes a table that summarizes each recommendation, the action to be taken, and the timing for completion of these actions. As you are aware, ORD conducts periodic reviews of its research programs at intervals of 4 to 5 years. The purpose of these reviews is to evaluate research relevance, quality, and performance. The reviews also focus on identifying how the scientific community and our programmatic partners use ORD's scientific results to protect human health and the environment. In addition to these formal reviews, ORD will be providing a mid-cycle progress report to update the BOSC on progress made implementing the actions described in this document. The timing for the HHRP mid-cycle progress report will likely be in early 2012. In this context, we look forward to working with the BOSC again. Since Ke Deputy for Science ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) The U.S. Environmental Protection Agency's (EPA) Office of Research and Development (ORD) relies on its Board of Scientific Counselors (BOSC) to conduct independent expert reviews of its environmental research programs every four to five years. The Human Health Research Program (HHRP) Subcommittee of the BOSC met in Research Triangle Park, NC on January 13-15, 2009, and the BOSC Executive Committee provided a final report in December, 2009. The principal charge to the BOSC reviewers was to evaluate ORD's HHRP from a program assessment framework relative to program relevance, structure, performance, quality, leadership, communication, and outcomes. A second priority was to provide a summary assessment and performance ranking for each of the four long-term goals identified with the HHRP. A set of specific charge questions was used to guide the Subcommittee through the review, producing a number of recommendations and observations with regard to the program. The Subcommittee met by conference call in October, 2008, and December, 2008, and for a face-to-face meeting in January, 2009, in Research Triangle Park, North Carolina. The face-to- face meeting consisted of an in-depth review of all aspects of the Program. Sally Darney, National Program Director for Human Health, presented an overview of the HHRP including its broad strategy, history, general structure, goals and resources. Each of the Program's four long term goals (LTG) was then introduced by a leader of the respective LTG who provided a more in depth description of each goal and oriented the subcommittee to the respective poster session that followed. Posters for each of the four sessions were arranged in sub-groups by topic area with an overview poster. All posters employed a consistent format that was structured to present the science questions, summarize general methods and approaches, integrate results for a large body of work, and summarize the impact of that work for EPA partners and others who use it to guide risk assessment and risk management decisions. Poster booklets were sent to the subcommittee ahead of the review and included a written abstract of the poster and a list of key products (original research papers and synthesis papers) associated with it. The Subcommittee also heard from the key partners in the Agency's program offices and regions who rely on the information and scientific expertise provided by the HHRP, as well as external users of HHRP products. The Subcommittee began drafting its report at the face-to-face meeting. A draft report was reviewed by the Subcommittee in February, 2009, and again in April, 2009. Overall, there was consensus among the Subcommittee members that there has been a maturing of the HHRP. The Program is much more integrated, and the level and quality of science has improved. There is considerably more emphasis on human health and human health-related issues, and there is movement toward more of a public health-themed program. The HHRP, as a whole, appears to be robust and responsive to emerging issues. The scientific content is excellent and, compared to previous reviews, is more integrated within each LTG and among the LTGs as well. There appears to be good evidence for strong scientific productivity and a formidable impact of the work produced by the Program overall. In general, the members found the Program leadership to be excellent to outstanding from the senior level to the laboratory/field study levels. ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) The purpose of the following narrative is to respond to the specific recommendations made in the Review of the Office of Research and Development's Human Health Research Program at the US Environmental Protection Agency, received December 1, 2009. UPDATE ON ORD PLANNING AND CONTEXT FOR HHRP RESPONSE: ORD activities since the HHRP review in January 2009 are transforming the way ORD research is being planned, organized and implemented. Specifically, ORD is placing significant emphasis on an integrated transdisciplinary research paradigm in order to help solve important national environmental problems. As a means to this end, ORD is developing a "pilot" or "vanguard" program to integrate research around the broad problem of "Safer Products for a Sustainable World" (SPSW). This activity incorporates a significant portion of the current HHRP themes. This pilot program is directed at assuring the safety of chemicals, which is one of the EPA Administrator's primary goals. SPSW will integrate exposure and toxicology research across ORD, capturing those elements of HHRP and the Safe Pesticides/Safe Products Program that are specific to chemical evaluation, and integrating these with ORD's Computational Toxicology Program (NCCT), Endocrine Disrupters Program, Nanotechnology Program, and risk assessment methods in the Human Health Risk Assessment (HHRA) Program. Importantly, SPSW is being planned with partner engagement, including EPA Program Offices, Regions and others from the start, with a concerted effort to engage their participation throughout the process. Thus, we will engage appropriate partners in problem formulation, prioritization of science questions, and development of research products that are useful and timely to the Agency. This new planning approach addresses many of the issues raised by the HHRP Subcommittee related to research planning, outreach to partners and effective delivery of research products to users. Indeed, comments and recommendations from the 2009 HHRP BOSC subcommittee were consistent with the need for this new approach and have contributed substantially to our thinking. Accordingly, recommendations related to HHRP will be addressed in the context of this broader ORD process. To date, the SPSW steering committee has organized several workshops that gathered preliminary input from EPA program and regional partners and plans to solicit input from outside partners (other agencies, industry) in the near future. Based on this input, there is consensus that EPA needs to change current approaches for chemical screening and testing to be far more efficient, effective and systematic. New approaches would address toxicity and exposure of both new and existing chemicals according to the principles outlined in NRC's 2007 report "Toxicity Testing in the 21st Century." This involves concerted research to resolve toxicity pathways using genomics and systems approaches to revolutionize toxicity testing. Furthermore, principles of green chemistry and life cycle analysis are viewed as critical to the new SPSW program in order for commerce to create sustainable products and processes. ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) SPSW is expected to incorporate aspects of research underway in the current Human Health Research Program, particularly as described in LTG-1 and LTG-2 which support this objective and are responsive to EPA's Strategy for Toxicity Testing in the 21st Century report (2009). Preliminary descriptions of these projects were included in the January 2009 BOSC review. As part of SPSW planning, these projects (as well as new project proposals) will be reviewed by a program planning group that includes ORD partners and will be prioritized according to criteria that include responsiveness, relevance and innovation. Since January, 2009, HHRP has completed many of the Annual Performance Goals outlined in the 2006 HHRP MYP and continues to build capacity in exposure science, modeling and computational toxicology prerequisite for the new SPSW program. Ongoing efforts in HHRP and other ORD programs contributing to SPSW include development of common "mine-able" databases on exposure and toxicity, verification of methods for using high throughput and high content data, development of high throughput computational tools for toxicity testing and risk prediction, and toxicity pathway analysis in risk assessment. An essential underlying assumption is that this research program must inform how vulnerability, according to exposure factors and inherent susceptibility, can be considered in risk assessment and mitigation efforts to ensure protection of vulnerable groups such as children. Targeted animal based testing and Mode of Action (MoA) research will be designed to meet existing regulatory needs and to verify the extent to which the new toxicity pathway approaches predict in vivo toxicity. Accordingly, the selection of chemicals for study and of models/tools for development will continue to be made in close partnership with program office and regional partners. This new program is being developed with the expectation that its products will directly inform regulations, including those resulting from upcoming TSCA reform, and also make significant strides to further the use of green chemistry and enhance sustainable product development. Of relevance to HHRP going forward, ORD plans to sustain a human environmental health program designed to use information on chemical risk derived from SPSW and other ORD programs to evaluate complex risks in real world community settings. While extending concepts developed in the current 2006 MYP (LTG-2 on cumulative risk; LTG-3 on Susceptible Populations and LTG-4 on Evaluating Risk Management Decisions), the program will be integrated to align more specifically with the Administrator's goal of "Cleaning up our communities" and the cross-cutting themes of: protecting susceptible populations, especially children; working for environmentalism and environmental justice; and, building partnerships with States, Tribes and community groups. An unofficial title "Healthy People in Sustainable Communities" (HPSC) will be used in this report to distinguish this community program component from SPSW. HPSC will continue to create and refine the tools and models necessary for community-based participatory research and actions, particularly for communities at risk and in support environmental justice objectives. HPSC will be informed by, and responsive to, recommendations from the 2010 EPA symposium "Strengthening Environmental Justice Research and Decision Making: A Symposium on the Science of Disproportionate Environmental Health Impacts." It will also continue to support ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) research to ensure children's health protection in the context of family, school and community stressors (only some of which involve chemicals), including collaborations with the National Children's Study. To further the goals of EPA's Report on the Environment and our ability to track changes and trends in environmental public health, HPSC will also continue to identify and interpret biomarkers of exposure and effect, as well as public health indicators of environmentally-related diseases such as asthma, and will continue to investigate linkages between environmental exposures before and after birth on children's health and disease. HPSC will be planned and reviewed as described above with an emphasis on partner engagement from the start, particularly the engagement of EPA Regions, Tribal groups, and EJ communities, as well as with other agencies such as CDC, NIH/NIEHS, HUD and the Department of Education, which work toward related broad goals. In summary, HHRP is actively engaged in the ORD transformation towards integrated transdisciplinary research and in alignment with the Administrator's goals for the Agency. SPSW and its companion program, HPSC, have in common the goal of finding innovative approaches for assessing and ultimately preventing environmental health risks. RESPONSE TO RECOMMENDATIONS ON LTG 1 Recommendation 1: The BOSC recommends that, through close collaborations with the Integrated Risk Information System (IRIS) staff, examples be developed in which the Mode of Action (MoA) for a chemical actually changes or influences the quantitative risk estimates IRIS makes for the chemical. Response: In responding to this recommendation, it is necessary to point out that the objective of MoA research in HHRP LTG 1 is not specifically linked with the IRIS assessments. The LTG 1 research focuses on broad science issues that relate to multiple chemicals. While we partner with the National Center for Environmental Assessment (NCEA) on many projects, we typically have not set out to fill specific needs for MoA for IRIS compounds, nor do we select chemicals for study based on upcoming IRIS assessments. IRIS compound selection is more often made for chemicals for which a large body of data already exists. On the other hand, HHRP research contributes conceptually to health risk and assessments performed by NCEA and EPA Program Offices. For example, HHRA and HHRP scientists are partnering on a project driven by the NRC's Framework for Risk Based Decision-Making. Collaboratively, we are refining PBPK models, applying bioinformatics-based knowledge mining, and mode of action research to augment the "Next Generation (Nex Gen) Risk Assessment Program." This effort is incorporating information derived from HHRP and NCCT research into prototype assessments in order to evaluate how new types of high throughput/high content data can augment, extend or replace traditional health assessment data. Prototype assessments are mode of action driven, with a minimum of two sentinel chemicals each: respiratory injury via inflammation (ozone and ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) chlorine), endocrine disruption via cell signaling alterations (BPA, perchlorate and phthalates), and cancer via genotoxicity (benzene and PAHs). These prototypes will incorporate both chemical-specific data and modeling approaches developed by HHRP. Related to respiratory damage, for example, a PBPK model for chlorine (described in poster HHRP BOSC poster 1-14) is undergoing completion. Also relevant to this recommendation, LTG 1 research on mode of action is designed to provide well characterized examples addressing fundamental science issues that can be used to advance Agency risk assessments by reducing dependence on default assumptions and their inherent uncertainties. Such research may fill specific gaps in current EPA assessments in NCEA (IRIS or Integrate Science Assessments/ISAs), but more typically has been undertaken in response to specific EPA program office requests. HHRP research on conazole pesticides, directed at addressing the issue of cancer vs. non-cancer modes of action, is designed for use by the Office of Chemical Safety and Pollution Prevention (OCSPP) to harmonize its risk assessment approaches. HHRP research on common modes of action of chemically-related triazide pesticides, with atrazine as a central focus, continues to inform OCSPP registration activities for atrazine and related pesticides. Mixtures research in LTG 2 has been used by OCSPP in its cumulative risk assessments for a number of pesticide classes: carbamates, organophosphates and currently pyrethroids, and has contributed to the development of models for cumulative risk anticipated to be needed in the future as EPA places more emphasis on cumulative risk assessment. Action/Timeline: The NCEA "Next Gen" projects are ongoing. Integrated teams (HHRP, NCCT and NCEA scientists and extramural grantees) are drafting prototype concepts for review at upcoming workshops. A final report (NCEA product) for at least one prototype is expected in late 2011. Future presentations to the BOSC we will more clearly explain how our research products are used in risk assessments. HHRP will participate in Science Advisory Panels culminating in OCSPP assessments of Conazoles (2011) and Pyrethroids (2010). Recommendation 2: The BOSC recommends more integration of the MoA science with the quantitative risk assessment generated by the epidemiology studies. Response: ORD appreciates this recommendation and the importance of linking toxicology and exposure data with epidemiology and public health studies. Efforts are progressing toward the new SPSW program described above, in which information on mode of action derived using in vitro screens and computational tools will be used to predict toxicity and prioritize chemicals for further testing and/or inform epidemiology studies based on both exposure and toxicity. The validity of these predictions will be tested in a systematic manner using higher order systems or whole animals in a highly targeted fashion. For chemicals or mixtures with unacceptably high risks based on exposure and toxicity, further testing may be designed to better define modes and mechanisms of action as needed to help design safer substitutes. Where feasible, results of epidemiology studies conducted by EPA or others may be used to verify these predictions as ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) well as inform new toxicology and exposure assessments. Also, research in HHRP - HPSC is determining which public health indicators make it possible to track environmentally associated diseases in exposed populations. This will enable EPA to better evaluate the accuracy of risk predictions based on exposure and toxicity, as well as the effectiveness of its risk management decisions (remediation efforts). Action/Timeline: Linkages will be made between the chemicals program (SPSW) and epidemiology and children's health studies to translate chemical/toxicology information into public and community health applications. This effort will be ongoing. Recommendation 3: Increased interactions (data sharing and research planning) among the researchers in LTG 1 with those in LTGs 2 and 3 are recommended. Response: We are addressing this recommendation on both fronts (data sharing and research planning). Actions for Data Sharing: Improved linkage in the exposure to health outcome continuum has been an underlying goal of HHRP since its inception, but has been hampered by lack of methods for linking different types of databases and information that has not previously been in a searchable database. Accordingly, ORD is investing in software and protocols for data sharing. The general goal is to build capacity for collaborations not only across LTGs and within SPSW, but also across ORD programs, EPA offices and federal agencies. For example, data sharing is a big component of the interagency Tox 21 collaboration among EPA, NIEHS, HHS and FDA. With this investment, ORD will be able to take a systems approach to identifying, predicting and preventing risks of chemicals. Specifically, in 2009 and 2010, HHRP has allocated funds to NERL and NHEERL to develop, populate and link exposure and toxicology databases and ensure that they can connect with NCCT's ACTOR information highway. This linkage is essential for information generated in the current HHRP LTG 1 on toxicology to be compatible with cumulative exposure information generated in LTG 2 (supporting exploration of the exposure to effects continuum) and then to translate the integrated information for research in community/real world settings (LTG 3). Such translation will ultimately address susceptibility of populations, especially children, enable apportionment of risk among chemical and non-chemical stressors, and provide the means by which to evaluate the effectiveness of risk management decisions (LTG 4). NCER is also making provision for increased data sharing among its grantees (LTG 2, 3 and 4) and providing access to data generated from the grants program for EPA scientists and regulators to use. Research Planning: As capacity is being built, research planning for SPSW and HPSC components is being coordinated across ORD Labs and Centers. For example, ORD is currently engaged in reviewing its portfolio of chemical-based research and building interdisciplinary teams to address the most important problems in an integrated manner. HHRP is also partnering with the National Children's Study where we see potential for long term impact in two directions. First, we want to contribute the best exposure science and environmental ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) epidemiology approaches to the NCS so that this ambitious national study will achieve its long term objectives. Second, we want to ensure that EPA scientists maintain the capacity to access and evaluate NCS data over the next 21 years in order to address key questions about environmental impacts on children's growth and development. Related to this effort is HHRP research that is developing practical tools for gathering exposure data for very young children, in their homes and in child care settings, and make such data available to others, including community and regional decision-makers. Action/Timeline: Investments in database building are expected to continue in 2011 to facilitate data sharing within and across ORD programs. The planning process being implemented for the new chemicals program, SPSW, and for its companion program in HHRP, HFSC, will involve transdisciplinary teams of scientists from ORD Labs and Centers as well as collaborating scientists from EPA Programs and/or Regions/Tribes, communities and other Agencies who will be involved in planning from the start and continue to participate in the design, implementation and interpretation/use of the research results. This effort will be ongoing. RESPONSE TO RECOMMENDATIONS ON LTG 2 Recommendation 1: The BOSC recommends that the Human Health Research MYP include a concerted educational outreach effort to the program offices, regional offices, and states regarding the use of sophisticated models and new knowledge developed through its research. Response: This important recommendation is being implemented in several ways. 1. Inclusion of users in tools/model development. Sophisticated models and information need to be developed with the end user in mind. Inclusion of the user and the developer from the start ensures that the final model can be readily transferred to the user(s). To address issues outlined in the Food Quality Protection Act, and other drivers for assessing cumulative risks of chemicals, ORD has collaborated with OCHPP/OPP for over ten year to design, develop and evaluate our Stochastic Human Exposure Dose Simulation (SHEDS) modeling system and companion Exposure Related Dose Estimating Model (ERDEM), along with other generic PBPK models. These models have been used to generate the science and data supporting a variety of FQPA risk assessments including the organophosphate, carbamate and pyrethroid risk assessments. The new science and knowledge gained from each risk assessment has been used to design the future research and improve how the models characterize and address the key variables influencing variability and uncertainty. The future SPSW will build off these programs. The users will be fully integrated in all phases of the collaborative research planning, implementation and application activities to ensure that the final model will be readily transferred to the user(s). 2. Providing web-based, user friendly interfaces ORD is developing, refining and distributing its models and databases through ORD and Agency web-based, user friendly, interfaces. In addition to ORD's SHEDS, ERDEM and PBPK models discussed above, a new ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) major HHRP effort, the Community-Focused Exposure and Risk Screening Tool (C-FERST), is being designed, refined and field-tested in close collaboration with several Regions, states, and communities. This tool provides access to state of the art exposure and risk characterizations for communities to use to understand the key stressors influencing their health, including primary routes and pathways of exposure and to design site specific plans for reducing exposures to chemical stressors. By providing a suite of simple, yet robust tools that incorporate innovative, high quality science into a user-friendly, web-accessible interface, C-FERST enables communities to evaluate the cumulative impacts of multiple stressors, prioritize environmental issues, identify communities at risk, and assess impacts of risk management actions. It employs user-friendly Google-Maps interfaces with a range of problem formulation and modeling capabilities, including online training materials. ORD is making a concerted effort to demonstrate these tools at national workshops and meetings. For example, the C-FERST tool was demonstrated at a Superfund Basic Sciences meeting in December 2009, and at an ORD- sponsored workshop in March on "Strengthening Environmental Justice Research and Decision Making." C-FERST was also featured at an ORD-Regional Science Workshop on cumulative, community-based risk assessment (July 2009, Chicago), at the 2010 Regional Applied Research Effort (RARE)/Community Action for a Renewed Environment (CARE) project officers training course, and the 2010 Tribal Science Forum. Action/Timeline: Collaborative research to enhance ORD's SFIEDS and companion dose models will focus on linking these tools to support rapid risk assessments and for characterizing variability and uncertainty. C-FERST continues to be evaluated and enhanced. Over the next few years, emphasis will be placed on enhancing exposure science, evaluating the tool in selected communities, and developing the capabilities to translate the science for community actions. The C-FERST prototype is being field tested internally in 2010 with expected public release in 2011. Additional collaborations with OCSPP on developing publicly-accessible GIS- based exposure models are also moving forward in 2010-2012. Recommendation 2: The BOSC recommends that goals or guidelines be defined that describe the threshold of acceptable accuracy for source-to-dose-to-health models and methods used in making assessments. Further characterization of the uncertainty of models, similar to that described in the source-to-dose paper by Ozkaynak et a/., is highly endorsed. Response: EPA conducts a wide variety of assessments, ranging from very sophisticated chemical-specific risk assessments to less robust assessments for site specific decisions. ORD researchers have incorporated sophisticated methods and approaches in the development of ORD's SFIEDS and dose models (as noted above) for characterizing variability and uncertainty. Future ORD exposure and dose modeling research will continue to employ state-of-the-science techniques for further characterization of uncertainty. As noted by the BOSC, the acceptability of the models and data used by the multiple Program Offices and Regions for their assessments varies greatly based on regulatory mandate and/or the criteria established by the Agency Office for their decision-making processes. ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) This BOSC recommendation will be considered in the design of the new ORD SPSW, which incorporates a focus on improving the relevance of assessment methods. As SPSW systematically conducts research to support "Intelligent Testing," while factoring in the inherent properties of chemicals, the program will integrate available knowledge and databases to produce the next generation of science and tools (including data attributes) needed to support Agency assessments. Action/Timeline: In addition to intramural projects, such as the one referenced above, an NCER RFA on this topic will be yielding new relevant models in 2010-11. As we develop the SPSW pilot, ORD researchers will continue to collaborate with program office scientists in the design and implementation of research that develops better ways to characterize variability and uncertainty in models used to support Agency decisions. Recommendation 3: As part of future BOSC reviews and as an accountability goal, the BOSC recommends that evidence (in summary narrative form) be provided on the use of completed research products in cumulative risk assessments. Response: ORD considers this recommendation as being important for all program reviews. Specific to HHRP, tools (e.g., SHEDS, ERDEM) developed for aggregate and cumulative risk assessments are being applied by the Program Offices and Regions to address regulatory decisions. As an example, ORD's MoA information and exposure models are being integrated and applied in the Agency's anticipated cumulative risk assessment of the pyrethroid insecticides in 2011. Numerous documents citing the ORD research continue to be referenced in the various Science Advisory Panel meetings supporting this landmark assessment. In this case, the primary research is designed to develop the fundamental science supporting future cumulative risk assessments, using the pyrethroids as a case study. As such, the documentation of the science in the open literature is as important as the documentation of how the ORD tools are being used to support the risk assessment. Moving forward, ORD is developing methods to better track usage of its products in risk assessments and other actions. To do this effectively, it will be particularly important to consider that many regulations only cite review articles. We will need to evaluate the references cited in the reviews in order to locate our primary research products and document their importance. We anticipate that the increased emphasis on engaging partners and users in SPSW and HPSC program planning, along with maintaining that engagement throughout the life of the program, will not only increase the usefulness of our research products but also facilitate tracking. Furthermore, ORD is considering developing a centralized tracking system. This would help identify all ORD products that contribute to Agency cumulative risk assessments regardless of which program generated them, and even when program structures change. Action/Timeline: For future program reviews, specific examples in HHRP will be captured not only in the posters for work done within the evaluation period (as in the 2009 review), but also 10 ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) for research products from previous evaluation periods. This is necessary because HHRP, as a "core" program, develops models and solutions over time, not necessarily to meet a regulatory deadline, and these models may be used in more than one risk assessment. Furthermore, we agree that this information should be synthesized in a summary table that clearly links the program goals, outputs and usage by risk assessors. Analyses of dockets done for the 2009 HHRP BOSC showed that we need to capture information over longer periods of time since it can take years before a research product or model actually is used in a risk assessment and that risk assessment is finalized, peer reviewed and released to the public. Nevertheless we will also attempt to show interim steps, such as inclusion of ORD data/models/products in interim stages of a risk assessment or health assessment document. Recommendation 4: The BOSC recommends the continuation of the general framework for planning, with the inclusion of greater planning efforts and knowledge sharing among LTG 1, LTG 2, and LTG 3, and with other Agencies. Response: We anticipate the current structure (4 LTGs) will be replaced by goals developed for the SPSW and HPSC programs. These programs will be implemented by scientists working on interdisciplinary teams. This strategy will address this BOSC recommendation by linking exposure, health, risk assessment and risk mitigation components within a project. We also plan to engage partners from other Agencies, as appropriate, to achieve the goals of the overall program and component projects. Action/Timeline: Ongoing. With respect to inclusion of other Agencies, planning for the integrated SPSW program already involves partners from NIH, NIEHS and CDC, as well as industry. Partnerships will be expanded to include parties interested in green chemistry and sustainability. Planning to better integrate HHRP research in cumulative community risk, children's health and EJ already includes NIH, NIEHS, CDC and HUD, and will be expanded to include other agencies, especially as the newly reconfigured President's Taskforce on Children's Health and Injury Prevention evolves its national children's health agenda.. To this end, we have established close ties with the new director of the Office of Children's Health Protection, Dr. Peter Grevatt, who also serves as an advisor to Administrator Jackson on children's health issues. Involvement of our partners will be documented in the next review by including collaborating scientists in the posters and presentations. Recommendation 5: The BOSC recommends that researchers who have extensive experience in "non- chemical stressors" be included in the overall plan for community-based research. Response: ORD recognizes the need to include social scientists in our programs. The EPA Science Advisory Board has also made this recommendation, and ORD has incorporated this need into its intramural workforce planning efforts. HHRP NCER grants provide an excellent 11 ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) means of integrating social science expertise into the program. New efforts include an NCER solicitation, released in 2009, focusing on the question of how to incorporate the influence of non-chemical stressors into community-based research and risk assessment. An announcement of these grants is imminent. NCER's Tribal Grants program, funded under HHRP, also makes major contributions to our understanding of the importance of non-chemical stressors. Each Tribal grant includes social science expertise to capture cultural, social and economic factors in tribal communities, as well as exposures that may impact health. In 2010, NCER communicated the results of this program broadly through a webinar series and by placing fact sheets on the web. A new solicitation for Tribal Grants is planned for 2011, with similar emphasis on non- chemical stressors. Furthermore, a new solicitation (for 2010 release) on how school environments, defined broadly, impact student and teacher performance is also expected to involve social scientists (as well as educators) in competitive grant applications. Finally, the new round of Children's Centers, co-funded with NIEHS, includes social science expertise, as have previous Centers in this successful program. Action/Timeline: The NCER grants described above will greatly expand ORD research in this direction. Results will be published over the next three to five years. ORD will continue to address this need in all its programs. Recommendation 6: As a future goal, the BOSC recommends more engagement of the regional offices in planning and identifying areas in which they need tools, methods, and data from ORD. Response: ORD's transformation includes the goal of more effective partnerships with States and Tribes, in alignment with the Administrator's priorities. ORD AA, Dr. Paul Anastas, charged all National Program Directors with improving our engagement with, and responsiveness to, Regional and Tribal needs. ORD's Office of Science Policy (OSP) is placing increased emphasis on collaborative research with Regional partners through better coordination of the RARE program. HHRP is now capturing RARE products in its tracking and communication of accomplishments, and working to incorporate results into future planning to address "next steps." ORD participated in a recent meeting of Regional Risk assessors who provided input into an upcoming (fall 2010) ORD-Regional Science Workshop on children's risk assessment, which will include definition of future research needs. An ORD-Regional Science workshop on Cumulative Community Based Risk Assessment, held in July, 2009, has built ongoing connections between HHRP's community based program and regional science and tribal programs. HHRP includes regional and OSP scientists on our Research Coordinating Team, and the NPD is working closely with OSP to further this goal at its fall retreat. As detailed under recommendation #1 above, C-FERST is a good example of collaborative research with our regional partners. Developed with place-based input from users, this tool will serve as a communications vehicle to disseminate innovative, high quality science on exposure 12 ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) and risk to the regional, state, tribal and community partners for use in evaluating the cumulative impacts of multiple stressors. Action/Timeline: A report from the 2009 Cumulative Risk Workshop, a 2010 HHRP APM product, will provide a synthesis of the workshop presentations and recommendations to inform future planning for this program. Actions ORD leaders can take to ensure regular communication and coordination with Regions will be discussed at the ORD/OSP 2010 fall retreat, for implementation in 2011. Recommendation 7: The BOSC suggests an added influx of resources into developing the science in cumulative risk assessments, if such assessments are to be effective in a reasonable timeframe. Response: ORD agrees that improvements in cumulative risk assessment methods are needed, and that this should be a priority for HHRP. In 2009, ORD sponsored a highly successful workshop with the Regions on this topic. In 2009-10, HHRP re-directed significant resources to build capacity in, and advance the science of, cumulative risk assessments. Resources are supporting ongoing efforts to build and link exposure databases with toxicity databases, extend exposure models to address complex exposures (e.g. SHEDs multi-media), and link exposure information obtained from multiple sources into a user-friendly tool for community use (C- FERST: Community-Focused Exposure and Risk Screening Tool). Extramural funding is providing better methods and models for cumulative risk assessment, with increased consideration of the extent to which non-chemical stressors impact responses to chemical stressors. A 2009 NCER RFA on this topic will fund a series of grants to be announced in 2010. Action/Timeline: Direction of significant resources to this end is providing tools for cumulative risk assessments. A report of the ORD-Regional Workshop to be published in 2010 will summarize how new tools can be brought to bear on cumulative risk assessments at the community level. RESPONSE TO RECOMMENDATIONS ON LTG 3 Recommendation 1: The BOSC recommends developing a more fully elucidated conceptual framework for vulnerability and susceptibility. Response: ORD agrees that a more fully elucidated conceptual framework for vulnerability and susceptibility would be helpful for future program planning and to prioritize research. The SPSW conceptual framework under development includes susceptibility from biological perspectives, i.e. inherent factors that impact response to toxicants such as gender, developmental stage, genetics, pre-existing disease, metabolism (e.g. child specific absorption- 13 ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) distribution-metabolism-elimination that can result in different vulnerability to a given level or type of ambient exposure). In turn, the HPSC program conceptual framework under development places emphasis on vulnerable populations based on where people live and how they behave (e.g. child-specific exposure factors based on behaviors of children at different ages). This concept will include consideration of social and economic factors that can impact vulnerability by, for example, determining where people live and how close they are to industrial or mobile sources of pollution, or by placing people at greater risk of adverse health impacts due to lack of adequate medical care, poor general health, or excessive stress associated with poverty and social injustice. The conceptual framework will include recognition that biological susceptibility at the individual level, as considered in toxicology studies, underlies the broader definition of vulnerability at the population level, as considered in community-based and epidemiology studies. Thus, a fully elucidated conceptual framework for vulnerability and susceptibility would be expected to be important for both SPSW and HPSC. The two programs will inform each other in this regard, and the framework will be applicable to other ORD programs. Action/Timeline: In response to this recommendation and outputs of the Spring 2010 Symposium on Environmental Justice mentioned previously, ORD will build capacity in community-based participatory research and environmental justice awareness. These activities are necessary as the foundation for a more fully elucidated framework for vulnerability and susceptibility for articulation in SPSW and HPSC planning. Recommendation 2: The BOSC recommends redressing a program imbalance within the life- stage arm of LTG 3, such that the strengths of the childhood susceptibility research thrust are matched with an expanded research program addressing the elderly, as well as potential subgroups across the entire age range. Response: While research on environmental impacts on Children's Health remains a critical element of the research program, we understand the importance of considering health impacts of the environment at all stages of the lifecycle. HHRP is leveraging efforts with the Air Program dealing with exposures to PM and air toxics and the resulting health impacts for aging people. As shown in posters 09 and 10, EPA has established a cross-ORD research program to address the susceptibility of older adults. By 2030, the number of older persons is expected to double to more than 70 million. EPA has previously launched an Aging Initiative, and ORD's research feeds directly into this as part of a broader focus on Community Health. For example, we have published handbooks on community-based UV radiation risk education, targeted to both children and older adults and senior citizens. One aspect of the initiative is the program "Age Healthier, Breathe Easier," and we are directing research to better inform this program. Clinical and animal studies currently underway are looking at the effect of air pollution on older individuals. In addition to direct research on aging, epidemiological studies are focusing on pollutant effects on conditions that become more prevalent in an aging population, such as diabetes, metabolic disease, atherosclerosis and subclinical cardiac disease. We are also studying whether intervention strategies focused on the elderly could reduce their susceptibility to pollution. For 14 ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) example, a current study is expected to lead to an advisory on nutrition that could reduce cardiovascular injury from air pollutants. Action/Timeline: Consistent with budgetary availability, ORD is addressing certain issues relevant to susceptibility of other life stages, particularly for older persons, and for people with chronic diseases associated with both pollution and aging. Engagement in community-based participatory research in the Children's Centers, Tribal grants program, and projects with Regions should provide opportunities to address other life-stages which may present stage- specific vulnerabilities. Recommendation 3: Rethinking the approach to asthma as a target condition so that it is not simply approached as a surrogate of childhood susceptibility to new disease onset, but rather considered across the entire age range and considered also in terms of vulnerability in pre- existing disease, is recommended. Response: We agree with the recommendation to re-evaluate the approach to asthma research. To that end, ORD is convening an Asthma Research Workshop on October 7l 2010. This workshop will be focused on two themes: Asthma and Community Health, and Asthma and Global Health. The goal is to provide information on the path forward to establish a research framework that can be used by a cross-Agency writing group to update the ORD Asthma Research Strategy. As recommended, current and future research is expanding the focus of asthma to other parts of the age range. For example, ORD is currently performing controlled exposure studies on older asthmatics. Studies are focusing not only on respiratory outcomes but also on cardiovascular injury caused by exposure to inhaled particles, to understand whether older asthmatics are at increased risk to endothelial injury and accelerated coagulation. At the other extreme, ORD researchers are studying developmental origins of disease and whether in utero exposure to pollutants will result in increased risk of developing asthma. As recommended, research in asthma as a pre-existing disease that confers vulnerability is being initiated. ORD scientists have initiated studies to understand the role of severity of asthma disease in conferring susceptibility to pollutants. As part of this research, we are integrating with the SPSW program and conducting research to determine whether toxicity pathways are perturbed by pollutants in different ways (e.g., different gene expression or micro RNA profiles) in individuals with different asthma severity. Furthermore, research proposed in the new round of Children's Centers will address fundamental questions about the interactions among biological allergens and ozone in the etiology and progression of asthma, and about controlling asthma symptoms by dietary intervention. Action/Timeline: The studies and workshop described above will provide results and reports in 2011 that will inform research planning for 2012. Asthma research in the Children's Centers 15 ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) will occur over the next five years. Recommendation 4: In addressing preexisting conditions, the BOSC recommends the program consider expansion beyond asthma to encompass other airway disease (in particular chronic obstructive pulmonary disease [COPD]) and, beyond lung diseases, consider other classes of disease such as neurological and endocrine disorders. Response: As recommended, we have commenced studies on respiratory diseases beyond asthma. These mechanistic studies are focused on airway inflammatory pathways that may underlie many respiratory diseases. Animal models and human clinical studies are looking at how pollutants can interact with viruses and other pathogens to impact host defense, and thereby cause increased bronchitis. Epidemiological studies, such as those studying the impact of wildfires on health outcomes, have considered hospital admissions and emergency room visits for respiratory diseases as a whole, as well as individual diseases (asthma, COPD etc). As part of these studies, admissions for other classes of disease, particularly cardiovascular disease, are being tracked. New studies focusing on individuals with other disease classes, e.g., diabetes, metabolic syndrome, overt and sub-clinical cardiac disease, have been initiated. As part of this effort, new animal models of different diseases are being established. Action/Timeline: New studies are addressing this recommendation as detailed above. Recommendation 5: The BOSC recommends better integration of LTG 3 across the other LTGs, in particular with LTG 2 in terms of cumulative exposure. Response: ORD agrees with this recommendation, which will be addressed in SPSW and HPSC research programs. SPSW will transform how the Agency identifies and manages chemical risks, particularly to mixtures that are encountered in "real world" settings. An important element of SPSW is to improve our understanding of inherent biological susceptibility to chemicals and contaminants. Specifically, toxicity pathway research will help determine who is likely to be susceptible, e.g., people with a defect in genes/proteins involved in a pathway. Conversely, the models used for toxicity pathways research will need to incorporate susceptibility, e.g., by using cells reflecting a disease or lifestage, and will need to be validated to ensure that they are applicable for susceptible/vulnerable groups. SPSW research will elaborate linkages between contaminant exposures (including mixtures) and the initiation and exacerbation of disease, while considering inherent factors that impact personal susceptibility. A major focus on the Virtual Embryo project in SPSW, for example, will define exposures that alter developmental pathways predictive of abnormal fetal and childhood 16 ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) development, and that can be confirmed in animal models. Together with improved exposure data and exposure factor information for young children, this information will help in the design of epidemiology studies and will be used to develop public health guidance for the public. A recent ORD workshop (spring 2010) provided guidance for the National Children's Study on the best metrics to use for linking: air pollution/allergens and asthma; pesticides and abnormal neurological development; and endocrine disrupters and altered reproductive development. Experts in exposure science, toxicology and epidemiology were convened for this purpose, and their discussions resulted in a report that has been provided to the NCS and its advisory groups. The Children's Centers also provide fundamental exposure and toxicology data that inform community-based programs, as will be summarized at a workshop planned for October, 2010 (Children's Health Month) by ORD in partnership with NIEHS, EPA's Office of Children's Health Protection and EPA/ATSDR's Pediatric Environmental Specialty Health Units. This workshop is designed to foster translation of children's health research to clinical practice and public health policies. Action/Timeline: As described above, current and future planning will include the need to articulate linkages between ORD's integrated chemicals program (SPSW) with research in the companion program (SDSC) that will focus on populations in community settings, where numerous chemical and non-chemical stressors interact to impact public health. Workshop reports that summarize "lessons learned" will be published in 2011. Recommendation 6: The BOSC recommends using successful intra-agency collaborations with the National Institute of Environmental Health Sciences (NIEHS) and the Centers for Disease Control and Prevention (CDC) in regard to childhood asthma as a model to address other vulnerable subpopulations. For example, collaboration with the National Institute on Aging (NIA) can address the potential susceptibility of the elderly to selected environmental exposures, such as those linked to neurodegenerative disease. Response: ORD agrees that it is critical to reach out to a broad range of partners. This recommendation will be addressed as ORD seeks more effective partnerships with NIEHS and CDC related to public health tracking and integration of exposure indicators into longitudinal health studies dealing with aging groups. ORD envisions increased interactions with new partners relevant to community public health, such as The National Partnership for Action to End Health Disparities, and the Federal Collaborative for Health Disparities Research. See also response to recommendation #2 in LTG 3. Action/Timeline: ORD will reach out to a broader range of partners and participate in inter- agency workgroups and taskforces with common goals. 17 ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) RESPONSE TO RECOMMENDATIONS ON LTG 4 Recommendation 1: The BOSC recommends improving interaction and linkage with other federal agencies and state agencies. Response: ORD agrees with the need for improved interactions and linkages with other federal and state agencies. Efforts in this direction are inherent in the new planning process. To this end, ORD is participating in efforts directed by OCHP to support re-invigoration of the Interagency Taskforce on Children's Health (anticipated in 2010) and the President's Taskforce on Obesity. These taskforces address children's health holistically, considering the many ways by which the environment may impact health. The possibility that in utero exposure to endocrine disrupters and other contaminants may contribute to childhood obesity by modifying metabolism was raised as an emerging issue at an international meeting planned by ORD, in partnership with NIEHS, FDA, CDC and international partners (December 2009). New approaches in exposure science, developmental biology, epidemiology and public health tracking are needed to evaluate if this risk is real. Thus, linkages with other Agencies concerned with the obesity epidemic and other diseases, such as asthma, are clearly needed to advance a national health agenda. ORD's community-based research will no doubt benefit from a recent agreement between HUD and EPA directed at healthy and sustainable communities. ORD is participating with CDC in public health tracking efforts and in projects designed to help CDC interpret biomonitoring data, such as that obtained through NHANES. A new grants program related to school environments has been developed with input from the Department of Education. Action/Timeline: ORD is strengthening and expanding partnerships with other agencies, particularly Nffl (NICHD, NIEHS), CDC and HUD. Recommendation 2: Developing a means to capture and preserve institutional memory to improve long-term assessment of programs is recommended Response: This is a challenge. ORD is working to centralize and track how its products are being used to impact Agency decisions. How and whether those decisions produce the effect desired and actually improve indicators of human health and environmental quality remains difficult. The goal of HHRP LTG-4 has been to have evaluation of risk management decisions incorporated into each of the media-specific areas. "Accountability" has a reasonably strong foothold in the Air program, with contributions from an NCER RFA on this topic and results of the New Haven accountability project in HHRP. The HHRP Water accountability project has broken new ground with the development of salivary antibodies as indicators of recent water- borne disease. New projects specific to this goal are not planned at this time. Action/Timeline: HHRP model projects have demonstrated the feasibility of accountability research in Air and Water. Tools developed may be useful in new community-based programs and showcase communities. 18 ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) Recommendation 3: The BOSC suggests making the ROE more prominent and influential in the Agency. Response: ORD agrees that this is an important goal. HHRP provides primary input for the health chapter of ROE. ROE is web-based and updated with new data on 85 indicators as captured quarterly by NCEA. A recent user analysis report, prepared for OMB, showed that the 2008 pdf was downloaded 9,024 times from December 2008 through March 2010, with about 29% of users being from the Federal government. The report was widely used beyond the Federal government, being distributed as follows: State government, 6%; local government, 5%; individuals, 13%; foreign, 14%; academia, 9%; non-profit, 5%; tribal governments, 12%; and schools, 1%. This analysis provides baseline information from which future influence can be measured. The SAB advises ORD on the ROE. A new edition is planned for 2012. Action/Timeline: Indicators from the ROE are being incorporated into indices to assess the effects of environmental quality on human health and well being, and into tools to identify communities at risk to support risk management decisions by EPA Regions. The ROE is widely consulted as a source of critical data in these actions, rather than simply a snapshot of America. Recommendation 4: The BOSC recommends expanding the use of health databases used to evaluate improvements in human health related to improvements in the environment, remaining cautious in interpreting these types of ecological analyses. Response: ORD agrees with this recommendation, although, in practice, access to health data is limited in the US today. In the New Haven project, available health outcome data was less useful than the air pollution data for detecting impacts of changes in national air pollution standards, coupled with local actions taken to reduce air pollution (Lobdell et al. EHP, submitted). ORD is piloting the use of restricted data from the census and from NHANES (CDC) in developing census block level assessments of exposure to priority contaminants. These estimates can serve to inform community level risk and as benchmarks for measuring the success of risk management actions. New techniques, including using exploratory epidemiological simulations (Baxter, JESEE 20, 2010; Lobdell et al., submitted) permit the estimation of changes in exposure levels (resulting from risk management actions) to bring about specified levels of changes in health outcomes. HHRP is working with CDC to support efforts to link health databases, and also to develop and link exposure databases. HHRP is also funding extramural research on the development and use of public health indicators. A synthesis of research from a previous RFA on this topic is being prepared, and a series of new grants on this topic will be announced in 2010. Action/Timeline: Increased effort is being directed towards the use of health databases and public health indicators in order to evaluate the Agency's risk management decisions. Final products of HHRP LTG-4 accountability projects are being published in 2010-11, along with reports from a recent NCER RFA on public health indicators. A series of new grants on public 19 ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) health indicators is being announced in 2010. Recommendation 5: The BOSC recommends expanding the use of direct estimates of the health implications of environmental interventions by calculating burden of disease or similar appropriate measures of risk. Response: ORD agrees with this recommendation, contingent on adopting a broader definition of health, akin to the WHO, that would allow for more broadly defined indicators of community health to be considered in the burden of environmental risk. The narrow definition of health as absence of disease restricts estimates of the burden of disease to physiological health outcomes. The difficulty of doing this has been acknowledged in the NRC report (2008) on Evaluating Research Efficiency in the U.S. EPA. Related to this concept, new efforts are being directed at developing an "environmental quality index" that will incorporate a wide range of both environmental and health indicators into a user- friendly tool. This tool could be used to measure risks of exposures and other stressors to human health and well being, and to measure benefits of Agency actions for environmental quality and public health. Action/Timeline: NCER RFAs on Accountability and on Non-Chemical Stressors will provide data and models for population-based assessment of environmental burden of disease, and better attribution of chemical vs. non-chemical determinants of disease for ecological analyses. In addition, ORD's Ecological Services Research Program is exploring the use of metrics of human well-being and broad indicators of population health. Recommendation 6: The BOSC recommends incorporating additional case studies into LTG 4 and attempting to extrapolate from existing case studies to other examples. Response: Case studies to evaluate community-based cumulative risk to assess community exposure and health risk are being pursued as part of the program that was once under LTG 2. OW is interested in pursuing further applications of the research on indicators of exposure and effects of waterborne pathogens, in particular to recreational waters. Lessons learned in New Haven about the use of hybrid air quality modeling as a surrogate for exposure measures, using emissions profiles in conjunction with meteorological and monitoring data, can be extended to other communities. Action/Timeline: Case studies are underway as collaborations between ORD and the Agency's Community Action for a Renewed Environment (CARE) and EJ Showcase programs. Additional case studies are expected to be forthcoming from a new series of grants on public health indicators (see previous), new Children's Centers funded in 2010, and a new RFA for Tribal grants in 2011. 20 ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) RESPONSE TO RECOMMENDATIONS FOR THE OVERALL PROGRAM Recommendation 1: The BOSC recommends that the partner survey be improved so that it is informative, or it should be abandoned. Response: We agree that the partner survey needs improvement. Therefore, the previous format of partner surveys has been abandoned and we are piloting a different mechanism for obtaining partner feedback. Action/Timeline: The revised approach is being piloted with the Drinking Water Research program and is expected to be complete by February, 2011. Recommendation 2: The BOSC recommends an increase in the expertise and integration of epidemiology and biostatistics throughout the LTGs. Response: ORD agrees that expertise in, and integration of, epidemiology and biostatistics is needed throughout HHRP. SPSW efforts are building capacity in database linkages and computational toxicology that will enable higher level analyses and foster predictive toxicology that will inform future epidemiology studies. We can acquire necessary expertise through careful workforce planning and through increased partnerships with other Agencies, such as CDC and NIH, with whom we can leverage our public health efforts. Action/Timeline: ORD will put a priority on gaining expertise in epidemiology and biostatistics through combined efforts in workforce planning, extramural research funding, and partnering with other Agencies. Recommendation 3: The BOSC recommends a reevaluation and reassessment of LTG groupings, with the goal of increasing communication within and among the various LTGs and decreasing silos. Response: The integrated SPSW program, as described in the introduction to this report, is predicated on the importance of combining expertise in exposure science, toxicology, computational toxicology, risk assessment and risk management, in order to holistically manage and prevent risks of chemicals. The companion HPSC program will be developed to translate SPSW information into community and public health contexts which, in turn, will inform needs for future research in SPSW. These two new programs will eliminate disciplinary silos. Action/Timeline: This recommendation is being addressed through ORD's integrated 21 ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) transdisciplinary research planning process. Recommendation 4: Development of a systematic process of prioritization and selection for determining which agents will be prioritized will create needed transparency and is recommended. Response: ORD agrees. As stated above, SPSW will conduct targeted animal-based testing and Mode of Action (MoA) research designed to fill existing regulatory needs and to verify the extent to which the new toxicity pathway approaches predict in vivo toxicity. Accordingly, the selection of chemicals for study and of models/tools for development will continue to be made in close partnership with Program and Regional partners. This new program is being developed with the expectation that its products will directly inform regulations, including those resulting from upcoming TSCA reform, and also make significant strides to further the use of green chemistry and enhance sustainable product development. Action/Timeline: The SPSW will use a systematic and transparent process for selecting problems and chemicals for study. Recommendation 5: The BOSC recommends that a communication plan be implemented with the intent to disseminate the impact of Program research throughout the Agency, clients, and the general public. Response: ORD's integrated transdisciplinary research principles will be used in planning SPSW and HPSC, including the development and use of a communication plan as recommended here. Action/Timeline: Ongoing under the SPSW and HPSC planning and implementation actions. Recommendation 6: The BOSC recommends that the HHRP explore more opportunities to collaborate with other Agencies and with academia to strengthen the program, save resources, and leverage external expertise. Response: ORD's integrated transdisciplinary research principles will be used in planning SPSW and HPSC, including engagement of relevant partners throughout the planning, implementation and application of research. This includes collaborating with other Agencies and academia to strengthen the program, save resources, and leverage external expertise. Consideration is being given to special hiring authorities that enable outside experts to participate in planning and research. ORD is strengthening its extramural STAR grants program 22 ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) to this end. Action/Timeline: ORD is exploring opportunities to collaborate with other agencies and with academia to strengthen the program, save resources, and leverage external expertise. Also, increased emphasis on community-based participatory research will increase partnerships with community groups, States and Tribes. Recommendation 7: The BOSC recommends that susceptibility factors examined in children's health be expanded to all life stages and across all LTGs. Response: ORD agrees. As detailed previously, ORD is incorporating biological susceptibility factors into SPSW, which addresses toxicity pathways relevant to all life stages, and other susceptibility factors into HPSC which considers the response of populations to complex exposures and environments encountered at the community level. Action/Timeline: ORD will incorporate susceptibility factors such as those examined in children's health into SWSP and HFSC research addressing all life stages. Ongoing RESPONSE TO RECOMMENDATIONS FOR THE REVIEW PROCESS Recommendation 1: There appears to be a good scientific impact of the Program, but the bibliometric analysis is difficult to interpret and understand, especially with the co-mingling of intramural and extramural publications. The BOSC recommends that this analysis be modified and improved or discontinued. Response: We recognize that many of the BOSC subcommittees have found the bibliometric analyses difficult to interpret. Although we believe that such analyses may still have utility for some of the programs, we are no longer measuring high impact and highly cited publications for its research programs. We are exploring more appropriate measures, methods and tools to replace the former bibliometric measures. ORD is awaiting OMB direction on performance evaluation, and will continue working with the interagency Science of Science Policy committee, along with the National Academies, to find more appropriate measures for research. Action/Timeline: ORD anticipates having new measures by June, 2011. Recommendation 2: The Subcommittee members found it challenging to navigate the Program evaluation materials, not only in terms of quantity but in how the material was presented. The BOSC recommends adding one poster at the beginning of each session that highlights all work 23 ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) done to date under each LTG to enhance each poster session. Inclusion of posters presented at national scientific meetings during the previous two years, or an abstract book detailing such posters, also would be helpful to the reviewers. Response: Since this review, ORD has revised the BOSC Program Review process by focusing the charge questions and the materials provided for the reviews. For future reviews, ORD will be making the linkage between the materials provided and the charge questions more explicit. This organization of materials should help the BOSC put the material in context to improve understanding and use during the review. The poster sessions primarily demonstrate the breadth of the research program and are generally focused at a higher level than individual research projects. While keeping the focus of the BOSC reviews at the programmatic level, ORD will consider these recommendations and others in order to provide the BOSC a better understanding of specific research projects and their outputs in future reviews. Action/Timeline: ORD will implement these changes for future BOSC reviews. Recommendation 3: Additionally, the Subcommittee would have benefitted from hearing about more specific partner interactions. The BOSC recommends that in future reviews, program partners and clients be included in the review, and that they justify how they use program products. One suggestion is to include partner testimonials in the poster sessions so that there can be more interaction between Subcommittee members and partners and clients. Response: ORD interprets this recommendation as a request for more detailed information regarding interactions with partners than what is typically given during partner testimonial sessions. Because of the limitations and feasibility of having many partner and clients present at the meeting, the poster session is meant to fill this role by demonstrating client use of ORD research and outcomes, and collaborations with other Programs, Regions, or non-Agency scientists. In future reviews, ORD will try to more fully demonstrate how the research products and information are being used by the partners in future program reviews. Action/Timeline: ORD will implement this change for future BOSC reviews. 24 ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) Table 1. Summary of the BOSC's Recommendations and ORD's Response and Proposed Actions Associated with Review of the Human Health Research Program Recommendation ORD Response ORD Actions (Timing) LTG 1 Recommendations Through close collaborations with the staff at IRIS, examples should be developed in which the MOA for a chemical actually changes or influences the quantitative risk estimates IRIS makes for the chemical. The objective of MOA research in HHRP LTG 1 is not specifically linked with the IRIS assessments. HHRP contributes data and models to Agency risk assessments conducted in OCSPP, OW, OAR and NCEA. HHRP is also partnering with NCEA on "Next Gen" risk assessments responsive to the NRC "Science and Decisions: Advancing Risk Assessment" Integrated teams (HHRP, NCCT and NCEA scientists and extramural grantees) are drafting prototype concepts for review at upcoming workshops. A final report (NCEA product) for at least one prototype is expected in late 2011. In future reviews we will more clearly explain how our research products are used in risk assessments. HHRP will participate in Science Advisory Panels culminating in OCSPP assessments of Conazoles (2011) and Pyrethroids (2010). More integration of the MOA science with the quantitative risk assessment generated by the epidemiology studies is needed. ORD appreciates this recommendation and the importance of linking toxicology and exposure data with epidemiology and public health studies. Efforts are progressing towards the new SPSW program described above in which information on mode of action derived using in vitro screens and computational tools will be used to predict toxicity and prioritize chemicals for further Linkages will be made between the chemicals program (SPSW) and epidemiology and children's health studies to translate chemical/toxicology information into public and community health applications. 25 ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) Recommendation ORD Response ORD Actions (Timing) testing and/or inform epidemiology studies based on both exposure and toxicity. Where feasible, results of epidemiology studies conducted by EPA or others may be used to verify these predictions as well as inform new toxicology and exposure assessments. Increased interactions (data sharing and research planning) among the researchers in LTG 1 with those in LTG 2 and LTG 3 are recommended. We are addressing this recommendation on both fronts (data sharing and research planning). Investments in database building (2010-11) will facilitate data sharing within and across ORD programs. The planning for SPSW and HPSC, will involve transdisciplinary teams of scientists from ORD Labs and Centers as well as collaborating scientists from EPA Programs and/or Regions/Tribes, communities and other agencies who will be involved in planning from the start and continue to participate in the design, implementation and interpretation/use of the research results. LTG 2 Recommendations The MYP should include a concerted educational outreach effort to the program offices, regional offices, and states regarding the use of sophisticated models and new knowledge developed through its research. As detailed above the program is including users in tools and models development, and providing web-based, user friendly interfaces with instructions. ORD is demonstrating these tools for program offices, regional groups and at a variety of national meetings The C-FERST prototype is being field tested internally in 2010 with expected public release in 2011. Additional collaborations with OCSPP on developing publically- accessible GIS-based exposure 26 ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) Recommendation ORD Response ORD Actions (Timing) and workshops. models are also moving forward in 2010-2012. Future emphasis will be on using these in community based projects. Goals or guidelines should be defined that describe the threshold of acceptable accuracy for source-to-dose-to-health models and methods used in making assessments. Further characterization of the uncertainty of models similar to that described in the source-to-dose paper by Ozkaynak, et al., is highly endorsed. ORD researchers have incorporated sophisticated methods and approaches in the development of ORD's SFtEDS and dose models (as noted above) for characterizing variability and uncertainty. Future ORD exposure and dose modeling research will continue to employ state-of-the-science techniques for further characterization of uncertainty. As noted by the BOSC, the acceptability of the models and data used by the multiple Program Offices and Regions for their assessments various greatly based on regulatory mandate and/or the criteria established by the Agency office for their decision-making processes. Intramural and STAR grantee projects are yielding relevant models in 2010-11 timeframe. As we develop the SPSW pilot, ORD researchers will continue to collaborate with program office scientists in the design and implementation of research that develops better ways to characterize variability and uncertainty in models used to support Agency decisions. As part of future BOSC reviews and as an accountability goal, evidence (in summary narrative form) should be provided on the use of completed research products in cumulative risk assessments. ORD considers this recommendation as being important for all program reviews. For FfflRP, numerous documents citing the ORD research continue to be documented in Science Advisory Panel meetings supporting the landmark pyrethroids cumulative risk assessment. For future program reviews, specific examples in FtHRP will be captured not only in the posters for work done within the evaluation period (as in the 2009 review) but also for research products from previous evaluation periods because it can take many years for the Agency to complete a risk assessment. 27 ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) Recommendation ORD Response ORD Actions (Timing) The general framework for planning should be continued, with the inclusion of greater planning efforts and knowledge sharing among LTG 1, LTG 2, and LTG 3, and with other agencies. We anticipate the current structure (4 LTGs) will be replaced by goals developed for the SPSW and HPSC programs. These programs will be implemented by scientists working on interdisciplinary teams. This strategy will address this BOSC recommendation by linking exposure, health, risk assessment and risk mitigation components within a project. We also plan to engage partners from other agencies as appropriate to the goals of the overall program and component projects. Future programs will link exposure and toxicology research with epidemiology and public/community health research. Researchers who have extensive experience in "non-chemical stressors" should be included in the overall plan for community-based research. Agreed. NCER grants in chemical/non- chemical stressors, Tribal grants and the Children's Centers program are/will incorporate researchers with expertise in social science and economics. The influence of non-chemical stressors on health impacts of chemicals will receive significantly more emphasis in HHRP funded NCER grants over the next 3-5 years. Intramural workforce planning in ORD will also address this need. As a future goal, there should be more engagement of the regional offices in planning and identifying areas in which they need tools, methods, and data from ORD. Agreed In response to Dr. Anastas's challenges in the path forward, ORD will increase engagement of regional offices to address this recommendation. 28 ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) Recommendation ORD Response ORD Actions (Timing) There should be an added influx of resources into developing the science in cumulative risk assessments if such assessments are to be effective in a reasonable timeframe. Agreed Direction of significant resources to this end is providing tools for cumulative risk assessments. A report of the ORD-Regional Workshop to be published in 2010 will summarize how new tools can be brought to bear upon cumulative risk assessments at the community level. LTG 3 Recommendations A more fully elucidated conceptual framework for vulnerability and susceptibility should be developed. Agreed, as discussed above. In response to this recommendation and outputs of the spring 2010 Symposium on Environmental Justice mentioned previously, ORD will build capacity in community- based participatory research and environmental justice awareness needed to build and address a more fully elucidated framework for vulnerability and susceptibility for articulation in SPSW and HPSC planning. 29 ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) Recommendation ORD Response ORD Actions (Timing) Program imbalance within the life-stage arm of LTG 3 should be redressed such that the strengths of the childhood susceptibility research thrust are matched with an expanded research program addressing the elderly as well as potential subgroups across the entire age range. While research on environmental impacts on Children's Health remains a critical element of the research program, we understand the importance of considering health impacts of the environment at all stages of the lifecycle. We are leveraging research with the Air program and maintaining an emphasis on aging. Ongoing efforts in aging, and new community-based and Tribal research will help us address the current imbalance despite lack of new resources, and without decreasing our commitment to children's health. Rethinking the approach to asthma as a target condition so that it is not simply approached as a surrogate of childhood susceptibility to new disease onset, but rather considered across the entire age range and considered also in terms of vulnerability in pre-existing disease, is recommended. As recommended, current and future research is expanding the focus of asthma to other parts of the age range including studies with older asthmatics. Studies on cardiovascular injury caused by exposure to inhaled particles, will shed light on why older asthmatics are at increased risk to endothelial injury and accelerated coagulation. At the other extreme, ORD researchers are studying developmental origins of disease and whether in utero exposure results to pollutants will result in increased risk of development of asthma. An asthma workshop in fall of 2010 will address this recommendation by defining future research needs and priorities for ORD within a new research framework. In addressing preexisting conditions, the program should consider expansion beyond asthma to encompass other airway disease (in particular COPD) and, beyond lung diseases, consider other classes of disease such as neurological and endocrine disorders. Agreed. Mechanistic studies (both in house and in the Children's Centers) are focused on airway inflammatory pathways that may underline respiratory diseases, and host- pathogen interactions in the etiology of bronchitis. Others are integrating stressors (e.g. wildfires) with multiple outcomes (asthma, COPD, cardio vascular). New and ongoing studies are addressing this recommendation. 30 ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) Recommendation ORD Response ORD Actions (Timing) Better integration of LTG3 across LTGs is recommended, in particular with LTG 2 in terms of cumulative exposure. ORD agrees with this recommendation which will be addressed in SPSW and HPSC research programs. Current and future planning will include the need to articulate linkages between ORD's integrated chemicals program (SPSW) and research in the companion program (SDSC) that will focus on populations in community settings where numerous chemical and non- chemical stressors interact to impact public health. Workshop reports that summarize "lessons learned" will be published in 2011. Successful intra-agency collaborations with the NIEHS and the CDC in regard to childhood asthma should be used as a model to address other vulnerable subpopulations, for example, collaboration with the National Institute on Aging to address the potential susceptibility of the elderly to selected environmental exposures, such as those linked to neurodegenerative disease. ORD agrees that it is critical to reach out to a broad range of partners. ORD is seeking more effective partnerships with NIEHS and CDC related to public health tracking and integration of exposure indicators into longitudinal health studies dealing with aging groups. Other include The National Partnership for Action to end Health Disparities and the Federal Collaborative for Health Disparities Research. See also response to recommendation #2 in LTG 3. ORD will reach out to a broader range of partners and participate in Inter-Agency workgroups and taskforces with common goals. 31 ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) Recommendation ORD Response ORD Actions (Timing) LTG 4 Recommendations Interaction and linkage with other federal agencies and state agencies should be improved. Agree. Interaction with CDC for Environmental Public Health tracking, indicators, and health impact assessments are critical elements for making the linkages necessary in the accountability framework. In addition, interaction with NIEHS Partners in Environmental Public Health affords opportunities to work with communities implementing risk management actions. Further interaction is also necessary within the Agency, specifically with the Regions and Enforcement offices that work with states and communities on implementation and need to directly assess outcomes of the broad range of risk management decisions. ORD is strengthening and expanding partnerships with other agencies, particularly NIH (NICHD, NIEHS), CDC, and HUD. E.g., MOU signed with CDC; connections made with NIEHS PEPH. A means to capture and preserve institutional memory to improve long-term assessment of programs should be developed. This is a challenge. The goal of HHRP has been to have evaluation of risk management decisions incorporated into each of the media- specific areas. "Accountability" has a reasonably strong foothold in Air and in some parts of Water. HHRP model projects have demonstrated the feasibility of accountability research in Air and Water. Tools developed may be useful in new community based programs and showcase communities. 32 ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) Recommendation ORD Response ORD Actions (Timing) The ROE should be made more prominent and influential in the Agency. HHRP provides primary input for the health chapter of ROE. ROE is web-based and updated with new data on 85 indicators as captured quarterly by NCEA. Indicators from the ROE are being incorporated into indices to assess the effects of environmental quality on human health and well being and into tools to identify communities at risk to support risk management decisions by EPA's Regions. With these, ROE is widely consulted as a source of critical data, rather than simply a snapshot of America. Decisions about ROE's future enhancements will be made at Agency level. The use of health databases used to evaluate improvements in human health related to improvements in the environment should be expanded, remaining cautious in interpreting these types of ecological analyses. Agree, although access to health databases remains a challenge. ORD is piloting the use of restricted data from census and NHANES in developing census block level assessments of exposure to priority contaminants. These estimates can serve to inform community level risk and as benchmarks of the success of risk management actions. New techniques, including using exploratory epidemiological simulations (Baxter, JESEE 20, 2010; Lobdell, et al, submitted) permit the estimation of changes in exposure levels (resulting from risk management actions) to bring about specified level of changes in health outcomes. Increased effort is being directed towards the use of health databases and public health indicators in order to evaluate the Agency's risk management decisions. Final products of HHRP LTG-4 accountability projects are being published in 2010-11 along with reports from a recent NCER RFA on public health indicators. A series of new grants on public health indicators is being announced in 2010. 33 ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) Recommendation ORD Response ORD Actions (Timing) The use of direct estimates of the health implications of environmental interventions should be expanded by calculating burden of disease or similar appropriate measures of risk. Agree with this recommendation, contingent on adopting a broader definition of health, akin to the WHO, that would allow for more broadly defined indicators of community health to be considered in the burden of environmental risk. The narrow definition of health as absence of disease restricts estimates of the burden of disease to physiological health outcomes. The difficulty of doing this has been acknowledged in the NRC report (2008) on Evaluating Research Efficiency in the US EPA. NCER RF As on Accountability and on Non-chemical Stressors will provide data and models for population-based assessment of environmental burden of disease and better attribution of chemical vs. non-chemical determinants of disease for ecological analyses. In addition, ORD's Ecological Services Research Program is exploring the use of metrics of human well-being and broad indicators of population health. Additional case studies should be incorporated into the LTG and the program should attempt to extrapolate from existing case studies to other examples. Case studies to evaluate community-based cumulative risk based on community exposure and health risk are being pursued as part of the program that was once under LTG 2. OW is interested in pursuing further applications of the research on indicators of exposure and effects of waterborne pathogens, in particular to recreational waters. Lessons learned in New Haven about the use of hybrid air quality modeling as a surrogate for exposure measures, using emissions profiles in conjunction with meteorological and monitoring data, can be extended to other communities. Case studies are underway as collaborations between ORD and the Agency's Community Action for a Renewed Environment (CARE) and EJ Showcase programs. Additional case studies are expected to be forthcoming from a new series of grants on public health indicators (see previous), new Children's Centers funded in 2010, and a new RFA for Tribal grants in 2011. 34 ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) Recommendation ORD Response ORD Actions (Timing) Overall Program Recommendations The partner survey should be improved so that it is informative, or it should be abandoned. The previous format of partner surveys has been abandoned. ORD is piloting a different mechanism for obtaining partner feedback. The revised approach is being piloted with the Drinking Water Research program and is expected to be complete by February 2011. An increase in the expertise and integration of epidemiology and biostatistics throughout the LTGs is recommended. SPSW efforts are building capacity in database linkages and computational toxicology that will enable higher level analyses and foster predictive toxicology that will inform future epidemiology studies. ORD will put a priority on gaining expertise in epidemiology and biostatistics through combined efforts in workforce planning, extramural research funding, and partnering with other Agencies. A reevaluation and reassessment of LTG groupings is recommended, with the goal of increasing communication within and among the various LTGs and decreasing silos. The integrated SPSW program, as described in the introduction to this report, is predicated upon the importance of combining expertise in exposure science, toxicology, computational toxicology, risk assessment and risk management in order to holistically manage and prevent risks of chemicals. The companion HPSC program will be developed to translate SPSW information into community and public health contexts which, in turn, will inform needs for future research in SPSW. These two new programs will eliminate disciplinary silos. This recommendation is being addressed through ORD's integrated transdisciplinary research planning process. 35 ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) Recommendation ORD Response ORD Actions (Timing) Development of a systematic process of prioritization and selection for determining which agents will be prioritized will create needed transparency and is recommended. SPSW will conduct targeted animal-based testing and Mode of Action (MO A) research designed to fill existing regulatory needs and to verify the extent to which the new toxicity pathway approaches predict in vivo toxicity. Accordingly, the selection of chemicals for study and of models/tools for development will continue to be made in close partnership with program office and regional partners. The SPSW will use a systematic and transparent process for selecting problems and chemicals for study. A communication plan should be implemented with the intent to disseminate the impact of program research throughout the Agency, clients, and the general public. Agreed. ORD's integrated transdisciplinary research principles will be used in planning SPSW and HPSC and as such will include the development and use of a communication plan as recommended here. ORD's integrated transdisciplinary research planning will include a strategic communication plan. The HHRP should explore more opportunities to collaborate with other agencies and with academia to strengthen the program, save resources, and leverage external expertise. Agreed. ORD's integrated transdisciplinary research planning process for SPSW and HPSC will include engagement of relevant partners throughout the planning, implementation and application of research. SPSW is reaching out to academia, other agencies and industry as described above. HPSW is also reaching out to community groups, States, Tribes and Regions. Susceptibility factors examined in children's health should be expanded to all life stages and across all LTGs. Agreed, recognizing budget priorities. ORD is incorporating biological susceptibility factors into SPSW which addresses toxicity pathways relevant to all life stages, and other susceptibility factors into HPSC which considers the response of populations to complex exposures and environments encountered at the community level. ORD will incorporate susceptibility factors such as those examined in children's health into SWSP and HPSC research addressing all life stages. 36 ------- March 2010 ORD Response to the Board of Scientific Counselors Report on Review of ORD's Human Health Research Program (final report received December 2009) Recommendation ORD Response ORD Actions (Timing) Review Process Recommendations The bibliometric analysis is difficult to interpret and understand, especially with the co-mingling of intramural and extramural publications. This analysis should be modified and improved or discontinued. ORD is no longer measuring high impact and highlighted cited publications for all of its research programs. We are exploring more appropriate measures, methods and tools to replace the former bibliometric measures. ORD anticipates having new measures by June 2011. The Subcommittee members found it challenging to navigate the program evaluation materials, not only in terms of quantity but in how the material was presented. The Subcommittee recommends adding one poster at the beginning of each session that highlights all work done to date under each LTG to enhance each poster session. Inclusion of posters presented at national scientific meetings during the previous 2 years, or an abstract book detailing such posters, also would be helpful to the reviewers. For future reviews, ORD will be making the linkage between the materials provided and the charge questions more explicit. ORD will consider the other recommendations in order to provide the BOSC a better understanding of specific research projects and their outputs in future reviews. ORD will implement these changes for future BOSC reviews. The Subcommittee would have benefitted from hearing about more specific partner interactions. The Subcommittee recommends that in future reviews, program partners and clients be included in the review, and that they justify how they use program products. A suggestion by the Subcommittee is to include partner testimonials in the poster sessions so that there can be more interaction between Subcommittee members and partners and clients. In future reviews, ORD will try to more fully demonstrate how the research products and information are being used by the partners in future program reviews. ORD will implement this change for future BOSC reviews. 37 ------- |