UnifedStates ; i,::,;J* Environmental Protectio'n Agency | :• ; - i ] iAdmirif'trator -1(11:02) jl'.t'-^; Guidance for OdrtaTOtWHa Peer Review/ of Regulatory IV!c5|lel$ g; : ; Agency Task Fc^rce on ! Environmental Regulatory Modeling ------- ------- UNITED STATES ENVIRONMENTAL PROTECTION AGENCY WASHINGTON, D.C. 20460 JUL I 5 1994 OFFICE OF THE ADMINISTRATOR MEMORANDUM SUBJECT: Establishment of Agency-wide Guidance for Conducting External Peer Review of Environmental Regulatory Models TO: Assistant Administrators General Counsel Inspector General Associate Administrators Regional Administrators Deputy Assistant Administrators Deputy Regional Administrators As Chairperson of the Agency's Science Policy Council, I am pleased to issue the attached "Guidance for Conducting External Peer Review of Environmental Regulatory Models" for Agency-wide use. The Guidance complements the Agency policy for peer review of scientifically and technically based work products. The Guidance should be considered and referenced as appropriate in the current development of standard operating procedures for peer review when those procedures involve model development and/or application. Agency managers who rely on modeling results may use the Guidance to help determine which products need peer review. Portions of this document may also prove helpful to those outside EPA who rely on environmental modeling. The Agency Task Force on Environmental Regulatory Modeling (ATFERM) developed the Guidance as part of its charge to address the need for improvement in the development and use of modeling for environmental decision making. The Science Advisory Board (SAB) originally expressed that need in its resolution on modeling (SAB-EEC-89-012). Comments and questions regarding this Guidance may be directed to its principal authors: Michael Firestone (OPPTS), Alan Cimorelli (Region 3), and John Yearsley (Region 10). Robert M. Sussma.n Deputy Administrator Printed on Recycled Pa/x ------- Attachment cc: Science Policy Council Science Policy Council Steering Committee Agency Task Force/Environmental Regulatory Modeling Council of Science Advisors Science Advisory Board Executive Committee ------- NOTE The following Guidance has been updated since April 24, 1994 when it was transmitted to the Deputy Administrator, to reflect the latest (June 7, 1994) statement regarding EPA's Peer Review Policy. ------- ------- AGENCY GUIDANCE FOR CONDUCTING EXTERNAL PEER REVIEW OF ENVIRONMENTAL REGULATORY MODELING Developed by: Agency Task Force on Environmental Regulatory Modeling (ATFERM) ------- ------- Acknowledgements This guidance document was developed by the Peer Review Subcommittee of the Agency Task Force on Environmental Regulatory Modeling (ATFERM). The principal authors included: Michael P. Firestone, Science Advisor, Office of Prevention, Pesticides and Toxic Substances Alan Cimorelli, Lead Meteorologist, Air and Radiation Division, Region III John Yearsley, Environmental Scientist, Environmental Services Division, Region X We would like to acknowledge the contributions of other members of the Peer Review Subcommittee including: Robert Ambrose, Environmental Research Laboratory at Athens, Office of Research and Development; Russell Kinerson, Office of Science and Technology, Office of Water; Joseph Tikvart, Office of Air Quality Planning and Standards, Office of Air and Radiation; and Lawrence Zaragoza, Hazardous Site Evaluation Division, Office of Solid Waste and Emergency Response. The authors are especially grateful for the support provided by ATFERM's Executive Secretary Mary Lou Melley, Office of Solid Waste and Emergency Response, and ATFERM's co-chairs, Larry Reed, Office of Solid Waste and Emergency Response, and Stephen Cordle, Office of Research and Development. We would like to express our appreciation to William Raub, Science Advisor to the Administrator, for his sound guidance regarding peer review and his assistance in refining the final document. We also thank Donald Barnes, Science Advisory Board Director, and Lynn Goldman, Assistant Administrator for Prevention, Pesticides and Toxic Substances, for their guidance with respect to peer review policy and practices at the Environmental Protection Agency. Finally, the authors acknowledge the contributions provided by the external peer review comments developed by the Environmental Engineering Committee of the Science Advisory Board, including Richard Con way, Chair, James Mercer, Modeling Peer Review Subcommittee Chair, and Kachig Kooyoomjian, EPA's Executive Secretary, and the internal peer review support provided by the Agency's Council of Science Advisors. ------- ------- AGENCY GUIDANCE FOR CONDUCTING EXTERNAL PEER REVIEW OF ENVIRONMENTAL REGULATORY MODELING I. INTRODUCTION According to EPA's Peer Review Policy Statement dated June 7, 1994, (see attachment): Major scientifically and technically based work products related to Agency decisions normally should be peer-reviewed. Agency managers within Headquarters, Regions, laboratories and field components determine and are accountable for the decision whether to employ peer review in particular instances and, if so, its character, scope, and timing. These decisions are made in conformance with program goals and priorities, resource constraints, and statutory or court-ordered deadlines. For those work products that are intended to support the most important decisions or that have special importance in their own right, external peer review is the procedure of choice. Peer review is not restricted to the penultimate version of work products; in fact, peer review at the planning stage can often be extremely beneficial. By contrast, the policy specifically excludes "non-major or non-technical matters that Agency managers consider as they make decisions." Clearly, environmental models (i.e., fate and transport, estimation of contaminant concentrations in soil, groundwater, surface water and ambient air, exposure assessment) that may form part of the scientific basis for regulatory decision making at EPA are subject to the peer review policy. However, it can not be more strongly stressed that peer review should only be considered for judging the scientific credibility of the model including applicability, uncertainty, and utility (including the potential for mis-use) of results, and not for directly advising the Agency on specific regulatory decisions stemming in part from consideration of the model output. The purpose of this guidance is to provide a resource for those program managers responsible for implementing the peer review process. More specifically, this guidance is provided as an aid in evaluating the need and, where appropriate, conducting external peer review related to the development and/or application of environmental regulatory modeling. This specific guidance for modeling has been prepared to complement general peer review guidance currently being developed by the Agency's Council of Science Advisors (Council). Thus, Section II (framework for peer review) and Section IV (peer review mechanisms and general criteria) reflect the latest Council guidance, and will be revised in the future as Agency-wide guidance on peer review evolves. The relationship of external peer review to the process of model development and application, including consideration of peer review at various stages in the process is described in Section III of this guidance. ------- Section V concerning "Documentation of the Peer Review Process" has been included in response to comments from the Agency's Science Advisory Board who cited the need for more detailed guidance on the mechanics of the review process. The specific elements of what could be covered in an external peer review of model development and application are presented in Section VI of this guidance. These elements are not meant to be prescriptive or limit the nature of peer review, but rather are intended as an aid to improve the thoroughness and consistency of peer review. To reiterate one of the major recommendations in the March 1992 report commissioned by former Administrator William K. Reilly re: "Safeguarding the Future: Credible Science, Credible Decisions (The Report of the Expert Panel on the Role of Science at EPA)": "Quality assurance and peer review should be applied to the planning and results of all scientific and technical efforts to obtain data used for guidance and decisions at EPA, including such efforts in the program and regional offices. Such a requirement is essential if EPA is to be perceived as a credible, unbiased source of environmental and health information, both in the United States and throughout the world." In conclusion, this document is intended to provide guidance to program managers in their efforts to consider, and apply where appropriate, external peer review to environmental regulatory modeling. The guidance contained in this document must not be construed as rigid requirements.1 II. FRAMEWORK FOR PEER REVIEW OF ENVIRONMENTAL REGULATORY MODELING Peer review can be an important tool in assisting the Agency to document the quality and credibility of the science upon which its regulatory and policy decisions are made. Modeling to provide the scientific support for environmental regulatory decision making at EPA can be thought to conceptually involve three stages including: (A) model development; (B) model application; and (C) consideration of modeling results in decision making. The guidance set out in this document is not final Agency action. It is not intended, nor can it be relied upon, to create any rights enforceable by any party hi litigation with the United States. EPA officials may decide to follow the guidance provided herein, or act at variance with the guidance, based on an analysis of specific circumstances. The Agency also reserves the right to change this guidance at any time without public notice. ------- External peer review, as discussed in more detail below, is generally relevant to the first stage of model development and may be relevant in appropriate cases to the second stage of model application. Although external peer review is not directly germane to the regulatory or policy decision itself, it is important at this third stage to bring forward information regarding prior peer review comments and the Agency's response related to model development and/or application. This information may aid the decision maker in interpreting and weighing the utility of modeling results along with all the other considerations (e.g., field data; risk, cost, and benefits information; requirements to use best available technology; environmental justice issues; etc.) in reaching a regulatory decision. Broader guidance regarding the applicability of peer review, as discussed in the Introduction, is currently being developed by the Council of Science Advisors. As this broader guidance develops and evolves in the future, the framework for external peer review of environmental regulatory modeling, as discussed below, will be revised. Ai Model Development Models are developed for a variety of reasons, including: 1. use as research tools to explore new scientific issues; 2. simplification and/or refinement of existing model paradigms or software; 3., use as screening tools; and 4. to estimate compliance with regulatory requirements (e.g., National Ambient Air Quality Standards). This guidance document, by its terms, does not directly address models developed for reasons other than to support regulator/ decision making (e.g., research tools). Models developed expressly for and used exclusively within a research program should be subject to essentially the same review process as other research results (e.g., informal critique by scientific colleagues, formal appraisal by senior scientists and managers, publication in refereed journal, etc.). However, if the purpose of a research model is expanded in the future to develop scientific information for Agency decision making, then the program manager should consider arranging for reassessment of the model in accordance with the guidance presented below. From a general scientific perspective, a well-conceived model is one that provides an acceptable mathematical approximation of a physical, chemical, biological, social or economic system. Model development frequently touches on the interface involving the ------- state-of-the-art in several areas including: (1) the technical and scientific understanding of processes and mechanisms; (2) applicable solution techniques (e.g., analytical, numerical); (3) computer science and technology; and (4) the Agency's need for scientific decision-support tools. The goal of model development is to provide methods which can be applied to improve Agency analysis and decision making. As such, the development of environmental models cannot be performed in a scientific vacuum. There must be a coherent class of applications in mind against which the appropriateness of the science can be judged. At the model development stage, a key step is to define and compare to existing models, the set of conditions under which the use of a model is scientifically defensible - this is known as the "application niche." Peer review of model development would be expected to include evaluation of the application niche, along with consideration of uncertainty and other areas of model performance. Approaching peer review from this perspective should help the decision maker understand the limitations of the scientific basis of the model and confidence in its results. It is only with this firm knowledge that the Agency can develop sound regulatory and policy decisions. External peer review of major technical issues related to environmental regulatory modeling is generally a lengthy process. This is precisely why external peer review should be identified as an important and integral aspect of an action plan for model development. At first glance external peer review might be expected to slow down the process. However, initiating peer review at early stages may, in fact, save time by redirecting misguided initiatives, identifying alternative approaches, or providing strong technical support for a potentially controversial position. B.. Model Application The middle stage, model application, begins with examining the stated application niche and its applicability to current needs and/or exploring whether a model can be tailored to fit a new niche. For existing models, especially models developed outside of EPA, peer review may be appropriate to the extent that either: 1) new information becomes available which calls into question the appropriateness of the previously defined application niche; or 2) a model might be considered for application outside the niche for which it was originally developed. Peer review of a model's applicability should, where possible, be planned well in advance of any decision making involving use of the model's results. In this way, the formation of "sound science" is distinct from the regulatory decision in which it is considered. The results of such a peer review can aid in the ultimate judgment by a decision maker to consider whether or how to use a particular model's result in the regulatory process. ------- Normally, the first application of a model should undergo peer review. For subsequent applications, a program manager should consider the scientific/technical complexity and/or novelty of the particular circumstances as compared to prior applications (also see Section IV-B). Peer review of all similar applications should be avoided because this would likely waste precious time and monetary resources while failing to provide the decision maker with any new relevant scientific information upon which to base a regulatory or policy selection. Nevertheless, a program manager may consider conducting peer review of applications upon which costly decisions are based or applications which are likely to end up in litigation. As an alternative to peer review of model application, the Office of Air Quality Planning and Standards has developed a formal review and acceptance procedure through rulemaking to evaluate the utility of alternative models for a particular application. C Environmental Regulatory Decision Making The final stage involves consideration of modeling results in the decision making process. This stage may include consideration of: (1) selecting among available models and assumptions/standard defaults; and/or (2) whether/how to consider modeling results in the regulatory process. Information previously derived from peer review of scientific issues may provide key information for the decision maker in understanding the uncertainties and utility (and potential for mis-use) of modeling results. When possible, public discussion of the scientific and technical underpinnings associated with the earlier stages of model development and/or application in advance of the final stage of regulatory decision making is expected to reduce criticism regarding the Agency's use of "sound science." Environmental regulatory and policy decisions also involve other scientific and non-scientific factors, and are by law, required to be made by the responsible Agency decision makers. Thus, such decisions are not appropriate subjects for scientific peer review. Rather, the process of public comment is frequently employed by the Agency at this stage. III. RELATIONSHIP OF EXTERNAL PEER REVIEW TO THE PROCESS OF ENVIRONMENTAL REGULATORY MODEL DEVELOPMENT AND APPLICATION The Agency's peer review policy notes that "properly applied, peer review not only enriches the quality of work products but also atdds a degree of credibility that cannot be achieved in other way. Further, peer review early in the development of work products in some cases may conserve future resources by steering development along the most efficacious course." Since this subject guidance focuses on the role of external peer ------- review, its relationship to other levels of peer review can best be understood by considering where external peer review may fit into the total process of environmental regulatory modeling. The following paradigm is meant to illustrate the general process: Step 0: Step Ob: (optional) Step 1: Step 1b: (optional) Step 2: Step 3: Step 4: The program manager within the originating office (AA-ship or Region) identifies elements of the regulatory process which would benefit from the use of environmental models. A review/solicitation of currently available models and related research should be conducted. If it is concluded that the development of a new model is necessary, a research/development work plan would be prepared. The program manager may consider internal and/or external peer review of the research/development concepts to determine whether they are of sufficient merit and whether the model is likely to achieve the stated purpose. The originating office develops a new or revised model or evaluates the possible novel application of model developed for a different purpose. The program manager may consider external peer review of the technical or theoretical basis prior to final development, revision or application at this stage. For model development, this review should evaluate the stated application niche. Initial Agency-wide (internal) peer review/consultation of model development and/or proposed application may be undertaken by the originating office. Model design, default parameters, etc. and/or intended application are revised (if necessary) based on consideration of internal peer review comments. External peer review is considered by the originating office. Model design, default parameters, etc. and/or intended application are revised (if necessary) based on consideration of external peer review comments. Final Agency-wide evaluation/consultation may be implemented by the originating office. This step should consist of consideration of external peer review comments and documentation of the Agency's response to scientific/technical issues. ------- (Note: Steps 2 and 4 are relevant where there is either an internal Agency standing or ad hoc peer review committee or process). A program manager may decide that peer review (step 2 -internal, and step 3 - external) should take place at more than one time during the processes of model development and model application (i.e., optional steps Ob and/or 1b). This decision would depend on the nature and complexity of scientific issues that are presented. Additionally, a program manager may also consider the utility of employing the broader concept of peer involvement - that is, to augment staff efforts by soliciting subject-matter experts from outside the program. Based in part on the results of a scientific peer review of model development and application, as well as other non-technical issues, the program manager would be responsible for advising the decision maker as to the consistent, equitable, and appropriate use of the model and its output for environmental regulatory purposes. IV. MECHANISMS AND GENERAL CRITERIA FOR CONDUCTING EXTERNAL PEER REVIEW The mechanisms and general criteria for conducting external peer review presented below are based upon generic peer review guidance currently being developed by the Council of Science Advisors. As this broader guidance develops and evolves in the future, this Section will be revised. /V Peer Review Mechanisms Mechanisms for accomplishing external peer review include, but are not limited to, the following: i. Using an ad hoc technical panel of at least three scientists; ii. Using an established external peer review mechanism such as the Science Advisory Board or Scientific Advisory Panel; or iii. Holding a technical workshop. Qualifications for peer reviewers will likely van/ for model development versus model application. For the former, the emphasis may be toward modelers, while for the latter, the emphasis may be toward scientists with technical expertise in other disciplines (e.g., statistics, field monitoring, etc.). ------- 8 New models, or significant modifications and/or new applications of established models, should be considered for publication in refereed journals. However, this step should be supplemental to and not a substitute for the peer review mechanisms presented above. E5. General Criteria General criteria to be considered for determining when and by what mechanism to initiate an external peer review of the development and/or application of environmental regulatory models include: i. Use of model results as a basis for major regulatory or policy/guidance decision making; ii. Significant investment of Agency resources; iii. Inter-Agency or cross-Agency implications/applicability; iv. Treatment of a new scientific issue; or v. Novelty and/or complexity of the science. V. DOCUMENTATION OF THE PEER REVIEW PROCESS It is important to prepare documentation for each peer review conducted and make this available to the decision maker along with the modeling results. The record should include the following information: A. Identity of the peer reviewers, their relevant expertise, and their institutional affiliation; B. Questions/issues posed to the peer reviewers; C. Reviewer comments, either as a collection of individual statements or as a consensus statement; and D. Agency response to peer review comments including rationale. ------- VI. SPECIFIC ELEMENTS OF EXTERNAL PEER REVIEW FOR ENVIRONMENTAL REGULATORY MODELING This Section of the guidance addresses the specific elements that should be considered to provide the Agency with consistent scientific/technical external peer review of environmental regulatory model development and/or application. Such elements could include, but are not limited to: A.. Model Purpose/Objectives The first step in evaluating a model used for environmental regulatory purposes is to clearly understand the broad context in which a model is intended to be used. i. What is the regulatory context in which the model will be used and what broad scientific questions is the model intended to answer (e.g., evaluating the range of human respiratory exposure resulting from air toxics release from a power plant)? ii. What is the model's application niche (i.e., the particular physical, chemical, and/or biological system and set of defining conditions for which the model is expected to be scientifically defensible)? iii. What are the model's strengths and weaknesses? How well does the model fit its intended application niche compared to existing models and available data sets? It is important that the Agency avoid duplicating prior efforts. B.. Major Defining and Limiting Considerations With a clear understanding of the broad purpose and objectives of a model in a regulatory context, the scientific context needs to be addressed. i. Which processes are characterized by the model (e.g., transport, diffusion, chemical reactions, removal mechanisms, etc.)? ii. What are the important temporal and spatial scales? Is the grid resolution appropriate for the problem? iii. What is the level of aggregation? ------- 10 C Theoretical Basis for the Model Once the regulatory and scientific contexts of a model have been defined, the basis for problem solving must be formulated. i. What algorithms are used within the model and how were they derived? What is the mechanistic basis? ii. What is the method of solution (numerical, analytic)? iii. What formulations are used for those processes which are parameterized? iv. How does the basis for problem solving compare to existing models? What is the scientific rationale? v. What are the shortcomings of the modeling approach (e.g., missing or over- simplification of key processes, restrictive dimensionality, etc.)? D. Parameter Estimation Parameter estimation may be based on case specific data or in their absence, on default values. In the latter case, it is important to understand how parameter defaults were established. i. What methods were used for parameter estimation? ii. What data were available for parameter estimation? iii. What methods were used to estimate parameters for which there were no data? iv. What is the reliability of parameter estimates? v. What are the boundary conditions and are they appropriate? E.. Data Quality/Quantity All models require the input of various types and amount of data. Models may also rely on experimental data to help shape their computational algorithms. In large measure, the utility of a model for regulatory purposes depends on the quality, quantity, and spatial and temporal adequacy of data used in its design and in support of its application. ------- F. 11 Questions related to model design include: i. What data were utilized in the design of the model? ii. How can the adequacy of the data be defined in terms of quality, quantity, and spatial and temporal applicability taking into account the regulatory objectives of the model? Questions related to model application include: i. What kinds of data are required to apply the model? ii. To what extent are these data available and what are the key data gaps? iii. Have data quality objectives been defined? If so, are they scientifically defensible? iv. Is the quantity of data sufficient to address the likely variability? What statistical analyses were performed and are they appropriate? v. To what extent are the data suitable with regard to estimating spatial and temporal effects? VI. Do additional data need to be collected and for what purpose? Key Assumptions The applicability of a model depends on the adequacy of its basic underlying assumptions. i. What are the key assumptions? ii. What is the basis for each key assumption and what is the range of possible alternatives? G. iii. How sensitive is the model toward modifying key assumptions? Modet Performance Measures The most basic test of a model's adequacy is to understand how well its results compare with real world measurements. i. What criteria have been used to assess model performance? ------- 12 ii. Did the data bases used in the performance evaluation provide an adequate test of the model in terms of applicability to the modeling niche? iii. How accurate can the model be expected to perform? Does the model exhibit any overall bias throughout the range of its predictions? Bias is an important test of the model's formulation since intrinsic system uncertainty is not present. iv. How well does the model address, distinguish, and report variability and uncertainty in its output? Which parameters and key assumptions are most significant in determining the model's variability and uncertainty? v. How does the model perform relative to other models in this application niche? JhL Model Documentation and Users Guide The utility of model for regulatory purposes depends on the availability of a clear documentation report and a comprehensive users guide. Do these cover: i Model applicability and limitations? ii. Data input? Interpretation of results? and in. IV. Documentation of the model code and other key aspects such as verification testing? JL Retrospective A retrospective analysis of the "big picture" may sometimes reveal insights that an analysis of individual components of a model may miss. i. Does the model satisfy its intended scientific and regulatory objectives? 11. in. Is there any available scientific evidence to suggest changes to either the model design and/or key parameters and assumptions prior to its use for regulatory purposes? How robust (i.e., not overly sensitive toward small changes in modifying key assumptions or input data) are the model predictions? ------- 13 iv. How well does the model output quantify the overall uncertainty resulting from limitations/simplifications in its design; use of standard assumptions; availability of supporting data; etc.? v. What key research is necessary to refine or improve the model and/or the data bases upon which it relies? The elements provided above are not meant to be prescriptive or limit the nature of external peer review. Rather the purpose for their inclusion is to provide modeling- related guidance for peer review protocols which will be consistent with more general guidance being developed by the Council of Science Advisors. Attachment 1: "Peer Review and Peer Involvement at the U.S. Environmental Protection Agency" dated June 7, 1994 ------- ------- PEER REVIEW AND PEER INVOLVEMENT AT THE U. S. ENVIRONMENTAL PROTECTION AGENCY This document establishes the policy of the United States Environmental Protection Agency (EPA) for peer review of scientifically and technically based work products that are intended to support Agency decisions. Peer review is presented in the context of the broader concept, peer involvement. BACKGROUND The report "Safeguarding the Future: Credible Science, Credible Decisions"1 focused on the state of science at EPA. The panel of experts who prepared the report emphasized the importance of peer review, especially external peer review, and the need for broader and more systematic use of it at EPA to evaluate scientific and technical work products. Their specific recommendation regarding peer review reads as follows: "Quality assurance and peer review should be applied to the planning and results of all scientific and technical efforts to obtain data used for guidance and decisions at EPA, including such efforts in the program and regional offices. Such a requirement is essential if EPA is to be perceived as a credible, unbiased source of environmental and health information, both in the United States and throughout the world." In response to this recommendation, then-Administrator Reilly directed staff to develop an EPA-wide policy statement, which he issued in January, 1993. The paragraphs below preserve the core of that earlier statement while updating it to specify the role of the Science Policy Council in guiding further implementation of the policy. Effective use of peer review is indispensable for fulfilling the EPA mission and therefore deserves high-priority attention from program managers and scientists within all pertinent Headquarters and Regional Offices. EPA/600/9-91/050, March 1992. ------- PEER INVOLVEMENT AND PEER REVIEW EPA strives to ensure that the scientific and technical underpinnings of its decisions meet two important criteria: they should be based upon the best current knowledge from science, engineering, and other domains of technical expertise; and they should be judged credible by those who deal with the Agency. EPA staff therefore frequently rely upon peer involvement — that is, they augment their capabilities by inviting relevant subject-matter experts from outside the program to become involved in one or more aspects of the development of the work products that support policies and actions. One particularly important type of peer involvement occurs when scientifically and technically based work products undergo peer review — that is, when they are evaluated by relevant experts from outside the program who are peers of the program staff, consultants, and/or contractor personnel who prepared the product. Properly applied, peer review not only enriches the quality of work products but also adds a degree of credibility that cannot be achieved in any other way. Further, peer review early in the development of work products in some cases may conserve future resources by steering the development along the most efficacious course. Peer review generally takes one of two forms. The review team may consist primarily of relevant experts from within EPA, albeit individuals who have no other involvement with respect to the work product that is to be evaluated (internal peer review) . Or the review team may consist primarily of independent experts from outside EPA (external peer review). POLICY STATEMENT Major scientifically and technically based work products related to Agency decisions normally should be peer-reviewed. Agency managers within Headquarters, Regions, laboratories, and field components determine and are accountable for the decision whether to employ peer review in particu ar instances and, if so, its character, scope, and timing. These decisions are made in conformance with program goals and priorities, resource constraints, and statutory or court-ordered deadlines. For those work products that are intended to support the most important decisions or that have special importance in their own right, external peer review is the procedure of choice. Peer review is not restricted to the penultimate version of work products; in fact, peer review at the planning stage can often be extremely beneficial. ------- SCOPE Agency managers routinely make regulatory and other decisions that necessarily involve many different considerations. This policy applies to major work products that are primarily scientific and technical in nature and may contribute to the basis for policy or regulatory decisions. By contrast, this policy does not apply to nonmajor or nontechnical matters that Agency managers consider as they make decisions. Similarly, this policy does not apply to these ultimate decisions. This policy applies where appropriate, as determined by the National and Regional Program Managers, to major scientifically and technically based work products initiated subsequent to the date of issuance. Peer review should be employed to the extent reasonable to relevant work products that currently are under development. This policy does not apply to the bases for past decisions, unless and until the relevant scientific and technical issues are considered anew in the Agency's decision-making processes. Except where it is required by law, formal peer review (as distinguished from the Agency's normal internal review procedures) should be conducted in a manner that will not cause EPA to miss or need extension of a statutory or court-ordered deadline. Agency managers still may undertake peer review if it can be conducted concurrently with necessary rulemaking steps. LEGAL EFFECT This policy statement does not establish or affect legal rights or obligations. Rather, it confirms the importance of peer review where appropriate, outlines relevant principles, and identifies factors Agency staff should consider in implementing the policy. On a continuing basis, Agency management is expected to evaluate the policy as well as the results of its application throughout the Agency and undertake revisions as necessary. Therefore, the policy does not stand alone; nor does it establish a binding norm that is finally determinative of the issues addressed. Minor variations in its application from one instance to another are appropriate and expected; they thus are not a legitimate basis for delaying or complicating action on otherwise satisfactory scientific, technical, and regulatory products. Except where provided otherwise by law, peer review is not a formal part of or substitute for notice and comment rulemaking or adjudicative procedures. EPA's decision whether to conduct peer review in any particular case is wholly within the Agency's discretion. Similarly, nothing in this policy creates a legal requirement that EPA respond to peer reviewers. However, to the extent that EPA decisions rely on scientific and technical work products that have been subjected to peer review, the remarks of peer reviewers should be included in the record for that decision. ------- IMPLEMENTATION The Science Policy Council is responsible for overseeing Agency-wide implementation. Its responsibilities include promoting consistent interpretation, assessing Agency-wide progress, and develocina recommendations for revisions of the policy as developing necessary The Science Policy Council will oversee a peer-review work group, which will include representatives from program units throughout EPA to effect a consistent, workable implementation of the policy. The work group will assist the programs in (1) formulating and, as necessary, revising standard operating procedures (SOPs) for peer review consistent with this policy; (2) identifying work products that are subject to review; and (3) for each major work product, selecting an appropriate level and timing of peer review. In assisting the programs, the work group will take into account statutory and court deadlines, resource implications, and availability of disinterested peer reviewers. The group will work closely with Headquarters offices and the Regional Offices toward ensuring effective, efficient uses of peer review in supporting their mission objectives. However, the Assistant Administrators and Regional Administrators remain ultimately responsible for developing SOPs, identifying work products subject to peer review, determining the type and timing of such review, documenting the process and outcome of each peer review, and otherwise implementing the policy within their organizational units. Because peer review can be time-consuming and expensive, Agency managers within Headquarters, Regions, laboratories, and field components are expected to plan carefully with respect to its use — taking account of program priorities, resource considerations, and any other relevant constraints as well as the policy goal of achieving high-quality, credible underpinnings for decisions. External peer reviewers should be chosen carefully to ensure an independent and objective evaluation. The affiliations of peer reviewers should be identified on the public record, so as to avoid undercutting the credibility of the peer-review process by conflicts of interest. This policy is effective immediately. The peer-review work group mentioned above will identify the focal point to whom comments and questions should be addressed and, from time to time, will provide further information about implementation activities. APPROVED: DATE CAROL M. BROWNER, ADMINISTRATOR ------- ------- ------- |