ENVIRONMENTAL PROTECTION AGENCY REGION III
STANDARD OPERATING PROCEDURES
FOR PEER REVIEW OF MAJOR TECHNICAL AND
SCIENTIFIC PRODUCTS
OCTOBER 1, 1994 - SEPTEMBER 30, 1995
11
R3 00038
-------
903R94052
U. S. Environmental Protection Agency
TJPPVACK Env'ronmentai.Science Center
trKKfn\^o . ir\ t »i .' _i - -
. 701 Mapes Road :
Ft. Meade, MD 20755-5350
Overview
These" Standard Operating Procedures (SOPs) provide guidance to
all staff and managers in Region III on the organization and conduct
'of peer reviews pursuant to the Administrator's June 7, 1994 Peer
Review Policy statement (Appendix A) .
The guidance provides information and outlines procedures in
several different areas:
.> basic principles and definitions, including
distinctions between peer review and public comment,
and between peer involvement and peer review;
» preparing for peer review, including identifying work
products, identifying appropriate peer review
mechanisms, and identifying qualified experts; and,
*-,' . conducting and completing peer reviews, including
materials required for peer review, creating a peer
review record, and utilizing peer review comments.
\
In addition, appendices provide information on Region III staff and
management responsible for pear review activities ah(l resources for
the upcoming fiscal year, work products selected for peer review
during the upcoming fiscal year, and detailed information on budget,
procurement, and legal considerations.
These SOPs do not address, nor do they supersede, established
peer review practices or procedures of the selected peer review
mechanism (e.g., professional journals for research papers, Science
Advisory Board and FIFRA Scientific Advisory Panel, and research
grant applications).
Background
Peer review is well-established as a mechanism for assuring the
quality, credibility, and acceptability of individual and
institutional work products. Peer review has long been used for
this purpose in academic institutions, professional organizations,
industry, and government.
Peer review at the USEPA takes many different forms depending
on the nature of the work product, relevant statutory requirements,
and office-specific policies and practices, among other things. In
iii
-------
January 1993,.responding to recommendations in the report
Safeguarding the Futures Credible Science. Credible Decisions,
Administrator William Reilly issued an Agency-wide policy for peer
xeview. Administrator Carol Browner confirmed and reissued the
/policy on June 7, 1994 and instituted an Agency-wide implementation
program. These SOPs constitute implementation guidance for EPA
Region "III.
These procedures are based on the central themes set forth in
the Administrator's policy statement:
Major scientifically and technically based work products
related to Agency decisions normally should be peer reviewed.
Agency managers within Headquarters, Regions, laboratories, and
field .components determine and are accountable for the decision
whether to employ peer review in particular instances and, if
so, its character, scope, and timing. These decisions are made
in conformance with program goals and priorities, resource
constraints, and statutory or court-ordered deadlines. For
those work products that are intended to support the most
important decisions or that have special importance in their
own right, external, peer review is the procedure of choice.
Peer review is not restricted to the penultimate version of
work products; in fact, peer review at the planning stage can
often be extremely beneficial.
As seen in the policy statement, one important task in
implementing the Peer Review Policy entails the identification of
"major scientific and technical work products.1* This decision is
based on several considerations and is discussed in Section II.
Another important task is using a peer review mechanism that is
suitable for the work product under review.
The goal of the Peer Review Policy is to implement a
comprehensive Agency-wide program to assure that major scientific
and technical work products receive critical scrutiny from
scientific and technical experts as part of the overall decision
making process. Generally, this technical review will precede the
customary, more broadly based public review of the total decision.
IV
-------
Region III Peer Review Policy Overview
Currently, Region III produces a wide variety of work products.
To date, the Region has relied upon ad hoc procedures for
identifying the appropriate mechanism of peer review that a work
product needs to undergo. The purpose of this SOP is to formalize
the procedures that Region III uses for peer review of scientific
and technical work products. In general, Region III will continue
to utilize peer review as well as other types of peer involvement
procedures as it has in the past. Optimally, much emphasis will be
placed upon peer input from experts and interested parties early and
often in the development of a scientific or technical work product.
This allows an open exchange of data, insights, and ideas throughout
the life of a project, working toward building consensus on the
technical aspects of the work.
Peer review mechanisms are typically selected based upon the
nature of the scientific or technical work product. In general,
scientific or technical work products will undergo an internal peer
review (whether in Region III or other USEPA offices (e.g., ORD)).
However, for work products that are "major", potentially costly,
controversial, novel approaches, or have cross-Agency implications,
external peer review is recommended. In general, the philosophy of
Region III is to have broad-based programs, methodologies, and
guidelines externally peer reviewed but-, the application of these
typically undergo only internal peer review.
-------
TABLE OP CONTENTS \
Preface . . ; iii
I. Guiding Principles of Peer Review 1
, A. General Principles . 1
B. Definitions 3
II. Identifying Work Products for Peer Review 5
A. The Selection Process 5
B. Mechanisms for Peer Review _.... 7
C. Categories. of Office Products . 9
III. Planning and Conducting a Peer Review 11
A. Selecting Peer Reviewers 11
B. Scheduling Peer Reviews < 14
C. Materials . 14
IV. Completing a Peer Review . 16
A. Evaluating Comments and Recommendations ........ 16
B. The Final Work Product .16
C. -Maintaining the Peer Review Record 17
V. Accountability and Responsibility in the Office 19
A. Line Management . . . 19
B. Office Coordination . 20
C. Budget . . . . , 22
D. Annual Reviews 23
VI
-------
TABIiEGf/CONTENTS-(continued) ; -
Appendix A: Agency Peer Review Policy . . . . . . . . . . . . . A-l
Appendix B: Key Personnel . . . . . . . . . . . . . . . ... . B-i
I. Individuals Involved in the Peer Review . , . . . . . B-I
A. Line Management B-I
B. Office Coordination B-l
II. Legal Advice B-2
Appendix C: Listing of Representative Office Products in Each
Category Over the Past 3 Years C-l
Appendix D: Office Selections for Peer Review in Upcoming
Fiscal Year- D-l
Appendix E: Resources for Peer Review Services E-l
I. Acquisition of Peer Review Services ..... E-l
\ ;
A. Voluntary Services ........... E-l
B. Contracts .~"~ E-l
C. Small Purchases . E-7
II. Travel . , E-8
A. Contracts and Purchase Orders E-8
B. Special Government Employees . . E-9
VII
-------
I. GUIDING PRINCIPLES OF PEER REVIEW
This section offers;some guiding principles for peer review,
defining certain terms, with emphasis on distinguishing between
several closely related concepts.' In particular, this section
discusses jthe interrelationship between the broader concept of peer
involvement and its components: peer input and peer review.
A. General Principles , ,
The purpose of peer review is to uncover any technical problems
or unresolved issues for use in revising a preliminary product so
that the final work product will reflect sound technical information
and analyses. It should be noted that peer review is-a process for
enhancing the scientific or technical work product: This means that
responsible^ reviewer recommendations for change in a work product
are a benefit to the Agency, not an embarrassment or something to be
resisted.
Peer involvement occurs at different stages, involving several
related but different activities in the overall decision-making
process (e.g., an initial research protocol or work plan, a
preliminary review of an interim draft, and/or a later review of a
draft final work product). Peer involvement refers to both peer
input (ongoing discussions during the development of the work
product) and peer review (the critical, and usually, final
evaluation of the work product). Peer review can occur during the
early stages of project or methods selection, or as usually defined,
as part of the culmination of the work product, ensuring that the
final product is technically sound.
Subject matter experts who participate in the overall peer
involvement process can be expected to undertake one of three
related -but different roles. First, they may work as paid or unpaid
consultants with a significant role as author or advisor in
developing a work product. Second, the USEPA may ask independent
experts to provide peer input by participating in early
developmental reviews or discussions of well-developed but
unfinished work products. In this case, the agency is inviting and
expecting expert suggestions that could lead to substantial changes
based on reviewer recommendations in the final product. Third,
experts may be asked to serve as peer reviewers, providing critical
evaluation and comments on work products nearing completion.
Serving as an author/advisor or by participating in early
developmental reviews or discussions can preclude later
participation as a peer reviewer for the same work product because
the expert may not be considered independent and unbiased.
Depending on the situation, peer review can be an iterative process
or a single event. Subject matter experts may review several
revisions of the same document, or they may comment only once on the
-------
document.'?VoAgency managers need to be aware of the role a subject
matter expert5has:played in.earlier reviews and make a determination
on a case by case/basis as to that expert's, independence for
any further;-reviews. x ' .
The importance of the peer reviewer's independence and
technical expertise cannot be overemphasized as factors influencing
the value and credibility of any peer review. . Independence --
freedom from institutional, ideological, or technical bias -- as to
the issues under review is necessary for objective, fair, and
responsible evaluation of the work product under review. However,
if reviewers are selected to represent any particular institution or
technical perspective, balancing the review with-representatives of
other institutions or perspectives becomes a critical objective for
the peer review;. Such perspectives and/or potential conflicts-of-
interest (real or perceived) should be fully identified to ensure a
'.balanced and credible peer review.
The importance of scientific and technical expertise in the
subject matter is obvious, but there are several dimensions. Host
importantly, knowledge in the subject matter is not equivalent to
expertise in the subject matter. For agency decisions, a multi-
disciplinary group of experts corresponding to the disciplines that
contribute to complex agency decisions is often necessary for a full
and complete peer review. For example, a risk assessment that
relies on both animal and human data often requires experts in both
areas for a complete review.
It is useful to distinguish between peer review and public
comment. Public comment refers to comment solicited~from the
general public through the Federal Register, usually, but not
always, as required by the Administrative Procedures Act or relevant
statutes or both. The USEPA takes public comment in this manner on
some strictly scientific products (e.g., ambient air quality
criteria documents, many risk assessment guidelines, and the recent
dioxin analysis) and almost all regulatory decisions. Public
commentors usually include a broad array of people with an interest
in the technical analysis or the regulatory decision; some are
scientific experts, some, are experts in other areas, and some are
interested non-experts. The critical distinction is that public
comment does not. necessarily draw the kind of independent, expert
information and analyses expected from the peer review process.
Public comment is open to all issues, whereas, the peer review
process is limited to consideration of technical issues. While an
important component of the review process, public comment does not
substitute for peer review.
As part of each peer review, the Agency must formulate a clear,
focused charge. A well-developed charge identifies all recognized
problem areas and invites comments or even directly asks for help.
This request signals the Agency's awareness of potential problems
and its receptivity to expert recommendations. The charge to peer
-------
reviewers usually makes.two general requests. First, it focuses the
review by presenting specific questions and concerns that the Agency
expects the reviewers to address. .Secondly, it invites general
comments on the entire, work product.. Focused questions also
greatly simplify the task of collating, analyzing and synthesizing
peer review comments on a topical basis. Moreover, written
responses to these questions by peer reviewers help the Agency
create a peer review record.
Careful attention to all of these elements, singly and
together, assures a credible peer review process. Conversely,
inattention can nullify the peer review attempt. A well-planned
peer review applied to a reasonable quality starting work product,
followed by responsible, visible utilization of peer review
suggestions in the final product assures a credible product for use
in Agency decision-making.
B. Definitions
1. Peer Involvement - Peer involvement refers to independent
expert participation in the development and/or review of a
scientific and technical work product. Peer involvement constitutes
active outreach to and participation of the broad scientific,
engineering, and economics communities beyond the USEPA (external)
as well as within the USEPA (internal). Typically, peer involvement
takes two general forms, peer input and peer review:
a. Peer Input - Peer input generally connotes an interaction
during the development of an evolving Agency work product,
providing an open exchange of data, insights, and ideas. Peer
input is characterized by a continued and iterative interaction
with scientific experts during the early stages of peer
involvement. . '
b. Peer Review - Peer review is an objective, critical review
of a specific Agency major scientific and technical work
product by independent peer reviewer or reviewers. Peer review
can occur at several discrete points during the peer
involvement process. It is characterized by a one-time
interaction or a limited number of interactions by independent
peer reviewers.
2. Independent Peer Reviewer - An independent peer reviewer
is an expert who was not associated with the generation of the
specific work product either directly by substantial contribution to
its development or indirectly by consultation during the development
of the specific product. The independent peer reviewer, thus, can
be objectively judgmental. This individual has expertise in the
subject matter required for the review function.
3. Maior Scientific and Technical Work Product - Scientific,
engineering, and economic documents or positions that are used to
-------
support a research agenda, regulatory program,:policy position or
other Agency decision that meet one or more of the criteria found in
Table l (in Section Hi). ,
4. Peer Review Leader A person who organizes and oversees
the peer review process for individual, specific work products..
Sometimes this individual is also the Project Manager for the work
product. ."
i -
5. Peer Review Coordinator - The individual designated by the
AA or RA to coordinate and monitor peer review activities in that
Program Office or Region.
\ '
6. Peer Review Group - The individuals who form the working
group that supports the Peer Review Coordinator. Service on this
group is normally a rotating responsibility.
-------
II. IDENTIFYING WORK PRODUCTS FOR PEER REVIEW
The principle underlying the Peer Review .Policy is that all
major scientific and technical work products (MSTWPs) normally
should be peer reviewed. The process for identifying major products
for review and then determining, the mechanism of review will take
into account various criteria. The decision maker(s) for peer
review should consider the full field of possible work products that
could benefit from peer review as well as the full spectrum of peer
review mechanisms for each product.
A. The Selection Process ,
1. Major Scientific and Technical Work Product (MSTWP1
a) The determination that a scientific or technical product is
a "major" work product is based upon a thorough consideration of a
series of criteria. Scientific, engineering, and economic work
products that are used to support a regulatory program or policy
position and that meet one or more of the criteria found in Table 1
are "major" work products and candidates for peer review:
TABLE 1
SOME CRITERIA FOR IDENTIFYING MAJOR
SCIENTIFIC AND TECHNICAL WORK PRODUCTS
1. Supports major regulatory decisions or policy/guidance of-major impact
2. Establishes a significant precedent, model, or methodology
3. Addresses controversial issues
4. Focuses on significant emerging issues
5. Has significant cross-Agency/inter-agency implications
6. Involves a significant investment of Agency resources
7. Considers an innovative approach for a previously defined problem/
process/methodology
8. Satisfies a statutory or other legal mandate for peer review
b) Scientific or technical work products that do not meet these
criteria for "major" may be also considered candidates for peer
review depending upon the nature of these products, specific program
needs/goals, and potential benefits of the peer review process
(i.e., is value added to the work product?).
c) Most scientific work products prepared by Region III are
based on guidance and methods issued by national program offices at
-------
EPA, Headquarters'; \ ^These guidance arid aeth<>ds: eire?preparied-;f6r the
direct use of Region III in implementing national programs ait the
Regional and State levels. Such guidance and methods':are subject to
peer review by the issuing national program office priorrto
transmittal to the Region for implementation. Therefore, further
peer review of major scientific products adhering to these methods
and guidelines is considered discretionary at the Regional level.
Major scientific and technical products prepared by Region III which
are not based on guidance from the national program offices or which
deviate substantially from national program office guidance are
strong candidates for peer review.
d) The need for and timing of peer review at various stages in
the development of the work product may greatly benefit that
product. The criteria in Table 1 may be used to help determine if
peer review is appropriate at a particular stage of work product
development (e.g.., help an office/region decide which direction/
methodology/technique is most appropriate from two or more choices) .
e) Appendix D lists scientific/technical activities/products
identified in each category by Region III as candidates for peer
review for the upcoming fiscal year. Also listed is the projected
mechanism of peer review for each work product.
2. Special Circumstances
a) There may be circumstances when all options for peer review
have been examined and there does not appear to be a rationale for
improving the work product with peer review. Then a ^decision for no
peer review can be justified. For example, peer review for major
scientific and technical work products is not required in at least
two special circumstances. First, additional peer review is riot
required for a particular product that has a known peer review
record by a recognized expert or expert body. Second, additional
review is not required if a new application of an adequately peer
reviewed work product does not depart significantly from its
scientific or technical approach.
b) Statutory and court ordered deadlines, time constraints
and/or resource availability may limit or preclude peer review.
Region III will evaluate these circumstances on a case by case
basis; decisions will be based on consultations involving line
management and the Peer Review Coordinator. If no peer review of a
major work product is conducted, then written justification to the
peer review archive is needed to explain the circumstance. Also,
written notification by the RA to the Science Policy Council is
needed.
-------
B. Mechanisms for Peer Review V ;
1. Mechanisms List
Peer review takes many different forms. Table 2 lists 13
different types of peer review, ranging from a single Agency expert
to a twenty-person panel of external experts. Familiarity with the
method, advantages, resource requirements, and logistics of the
different forms, and their variations, is an important aspect of
peer review planning.
-------
, '..-V "- ; ' --"- TABLE 2 " ' .-..
SOME MECHANISMS FOR OBTAINING PEER REVIEW
OF SCIENTIFIC AMD TECHNICAL WORK PRODUCTS
INTERNAL PEER REVIEW
1. An independent expert from within the Agency
. e.g., an ORD expert on non-cancer effects lead reviews a draft article
. on benchmark dose. .
2. An ad hoc panel of independent experts from (mostly) within the Agency
e.g., a group is convened to examine the case for the classification
for carcinogenicity for a chemical.
3. Office work group product reviewed for technical merit by scientist in
Agency laboratory
e.g., initial review of regional WTI risk assessment plan by RTP
scientists. - -
EXTERNAL PEER REVIEW
1. Independent expert(s) from outside the Agency
e.g., a peer reviewer' for a journal to whom an Agency scientist has
submitted a paper for publication; a letter review by one or more
reviewers ;
2. An ad hoc panel of independent experts from (mostly) outside the Agency
e.g., a group'is convened to reach consensus on the carcinogenicity of
formaldehyde. ' / - - .
3. Agency (including Risk Assessment Forum)-sponsored Peer Review
Workshops ,
e.g., review of indicators of .ecosystem damage. i._
4. An Agency-based Federal Advisory Committee (other than SAB)
e.g., the Biotechnology Science Advisory Committee (BSAC) meets to
review technical aspects of the release of bioengineered
. organisms.
5. Agency-appointed special Board or Commission
e.g.", review of risk issued by the CAA Commission on Risk Assessment.
6. Agency Science Advisory Board (SAB)
e.g., review of drinking water criteria document for arsenic.
7. Interagency committee
e.g., review of research plans by Committee on the Environment and
Natural Resources (CENR) coordinated by the White House.
8. Committee of another agency
e.g., review of "dioxin" reassessment by HHS Committee to Coordinate
Environmentally Related Programs (CCERP).
9. Non USEPA-based groups
e.g.. Society of Risk Analysis review of cancer guidelines.
10. National Academy of Science/National Research Council .
e.g., review of children and pesticides. '
-------
'-.. ..'-.-. ' . -;. .1 /_ .._'-.
. : , 2. selection of Appropriate Mechanismla) ,
a) Determining an appropriate mechanism of peer review is
generally subjective. Based upon the importance and impact of the
work product, the resources available (staff, budget, reviewers),
.the time investment, and previous review history, the mechanism of
peer review for the product will be selected (Table 2 provides many
options).' The choice of mechanism is based upon several criteria,
as seen in Table 1. Also, the extent of previous peer involvement
and/or peer review helps determine an appropriate mechanism of peer
review.
b) Generally, the more nearly novel or complex the science or
technology, the greater the cost implications of the impending
decision, and the more controversial the issue, then the stronger
the indication for a more extensive and involved peer.review, and for
external peer review in particular. For example, certain work
products will clearly lend themselves to extensive external peer
review; generally these will be products with large impacts. Other
major work products may not need a large scale external peer review
and may utilize a less involved, less resource intensive review.
The peer review of some .products may be better served with some form
of internal peer review or a.combination of internal and external
peer review. Again, the choice is subjective and will generally be
dependent upon the experience and assessment of the decision
maker(s) dealing with peer review issues.
C. Categories of Office Products
Publications in Scientific Journals
1. Isokindunos: Equal Risk Contours Downwind of An Air Toxics
Source
Region III Reports and Documents
1. Sinkhole Brochures
2. Exposure Point Cone, in Groundwater
3. Habitat Requirements for Ches. Bay Living Resources
Regional Scientific Plans
1. Mid-Atlantic Highlands Assessment
Most Region III work products will fall under one of the above
categories. Examples of other categories of major scientific work
products are the following:
- major environmental region survey/study design
- start-up of innovative technological units of equipment or
environmental processes
application of existing modeling to untried substrates/
application of novel modeling to high impact environmental
situations
-------
permits issued under regulatory statutes administered by
the"region may need peer review
multi-media and geographic initiatives due to unique
scientific or technical nature . .
major scientific/technical publications
'--' . major new scientific/technical data collection/analytical
methodologies
- major scientific/technical environmental studies or
, assessments
major new scientific/technical pollution control/site
remediation technology assessments and demonstration
projects
- technical reports oh methods and procedures
financial analyses
- new approaches for the assessment of chemicals or products
- new/novel approaches to economic evaluation/analyses
- scientific database design
technical guidance .
- regulatory impact analyses, cost/benefit assessments
- emission studies '
- public education and outreach
- interagency and international technical consensus products
. /
Appendix C lists representative scientific/technical activities/work
products in each category produced by Region III over the past three
years.
10
-------
CONDUCTING A PI
The success and usefulness of any peer review depends on the
quality of the peer review draft, the care given to the statement of
the issues or "charge11, the match between the peer review draft and
the form of peer review, the match between the peer review draft and
the scientific/technical expertise of the reviewers, and Agency use
of peer review comments in the final product. It is not enough
simply to conduct a peer review; each of the foregoing elements
requires serious attention.
A. Selecting Peer Reviewers
1. sources for Peer Reviewers
a) Recommendations for,potential peer reviewers can be
identified from a number of organizations. These include external
groups such as the affected party(ies); special interest groups;
public interest groups; environmental groups.; trade or business
associations; state organizations or agencies; the National Research
Council; and other Federal agencies with an involvement in or
familiarity with the issue. Internal groups include the staff of
the Science Advisory Board (SAB) or the Scientific Advisory Panel
(SAP); relevant ORD scientific staff; and other Program or Regional
experts.
b) In certain circumstances, existing peer review organizations
such as the SAB or SAP may be used to conduct a peer preview. These
groups establish their own criteria for accepting work and
coordination must be made directly with them. Both conduct formal,
public, external peer reviews. .
c).Occasionally, a member of the scientific community will
offer his/her services for peer review during an ongoing peer
review. Disposition of these offers will be handled on a case by
case basis in consultation with the Peer Review Coordinator and
appropriate decision makers.
2. selecting Peer Reviewers
a) Peer reviewers should be selected for independence and
scientific/technical expertise. The emphasis on independence and
expertise applies equally to government experts and experts from the
larger scientific community.
b) Ideally, peer reviewers should be free of real or perceived
conflicts-of-interest.
11
-------
c) Some peer reviews can be conducted
reviewers; others involve panels of peer reviewers;;^ In either case
each of the peer reviewers should have recognized;technical
expertise that bears on the subject matter under-discussion. Taken
as a, whole, the peer reviewers of a work product should represent a
balanced range of technically legitimate points of view. In the
face of equally qualified experts, cultural diversity and'"address"
(e.g., industrial, academic, or environmental community) are
secondary factors that can play a role.
3. Internal vs. External Peer Reviewers -
Generally, external peer reviewers are preferred. For some
work products, either external or internal peer review may be
appropriate. Selection of internal peer reviewers should be based
upon technical expertise, available time and "address"; that is,
they should not come from the immediate office or group, producing
the product or have any other connection with the product or
document being peer reviewed.
4. Disciplinary Mix
a) A peer review panel or group can number from just a few
individuals to ten or more, depending on the issue, the time and
resources available and the broad spectrum of expertise required to
treat the range of issues/quest ions in the charge (see Section
III.C. below). Naturally, experts whose understanding of the
specific technical area(s) being evaluated are critical;
nevertheless, it is also important to include a broad enough
spectrum of other related experts to completely evaluate the
relevant impacts on other less obvious concerns. For example, for
health related peer reviews, experts in such fields as ecology and
economics may provide very useful insights.
b) -There is usually a continuum of views on any issue. To the
extent possible or practicable, selected experts should have views
that fall to either side of the centrist position along the
continuum, but not too far to either extreme. This will help
maintain a balanced viewpoint, while allowing all views to be
expressed and discussed. A balanced panel will allow consensus
building. As a general rule, experts who have made public
pronouncements on ah issue (e.g., those who have clearly "taken
sides") have difficulty in reaching consensus positions and should
be avoided.
5. Constraints in Selecting Peer Reviewers
a) Peer review is not free. Regardless of the type of peer
review chosen, there is some cost to the Agency, This cost can
range from the time invested by a few Agency staff during an
internal peer review, to the total costs associated with a full,
formal, public forum external peer review; e.g., SAB review.
12
-------
b) Sometimes the need for a peer review is accelerated due to a
court-ordered deadline or other time-sensitive requirements, in
such cases, it is difficult, if not impossible to conduct a full
external peer review. It-may even be impossible to conduct a small
scale internal peer review using just a few individuals. Mechanisms
for identifying and using a small number of peer reviewers should be
included for Region III so that quick, effective-peer review can be
included for even the most rapidly moving products.
c) Care must be taken to reduce the possibility for real or
apparent conflicts of interest between the reviewers and the Agency
work product under review. . Various tools are available to identify
and limit conflicts of interest (e.g., attention to the employment,
financial, .and professional affiliations of the participants; filing
Confidential Financial Disclosure Forms (SF-450) in the case of
members of Federal Advisory Committees; exploring directly the issue
with each of the participants before the review process .takes place;
and disclosing publicly at the beginning of meetings any previous
involvement with the issue. The established peer review groups such
as the Science Advisory Board and the Scientific Advisory Panel
provide useful models for addressing balance and conflict-of-
interest issues. Assistance in determining legal conflicts of
interest can be obtained through the Office of the General Counsel.
In addition, additional advice can be obtained from the Designated
Agency Ethics Officials. Appendix E has further discussion on
conflict of interest.
d) To evaluate Agency-generated studies properly, some peer
reviewers may need access to confidential business.information
(CBI). However, unless, the reviewers are Federal employees, it is
unlikely that the USEPA has the authority to disclose CBI to them.
Therefore, whenever contemplating the use of outside peer reviewers,
Agency staff should determine whether the reviewers will need access
to CBI. If so, the Office of the General Counsel should be
consulted on whether it is practical to obtain the consent of CBI
submitters to disclose the information to peer reviewers.
e) Offices need to be aware of the requirements of the Federal
Advisory Committee Act (FACA) when establishing peer review
mechanisms. Federal advisory committees are subject to chartering
by the General Services Administration, hold meetings that are open
to the public, and have balanced membership requirements. The
Office of the General Counsel should be consulted regarding the
applicability of FACA to peer review panels.
13
-------
-. ..-. . -
: B. Scheduling Peer Reviews
The peer review schedule is a critical feature of the process.
The schedule must-take into account the availability of a peer
review quality draft work product,: availability of appropriate
experts, time available for using peer review comments, deadlines
for the final work product, and logistical aspects of the peer
review (eig., contracting procedures).
The schedule for peer review should take into account the
overall rulemaking (or other decision-making) schedule. Peer review
sometimes leads to new information and analyses, or recommendations
for new research' that would alter the work product and thus modify
the scientific/technical basis for the action. For this reason, it
is usually advisable to complete the peer review before taking
public comment, or at least before the close of the public comment
period. ..",'"".
r ,
C. Materials
1. Information Provided to Peer Reviewers
a) Essential documentation for each peer reviewer includes:
1) A current copy of the work product to be peer reviewed
with associated background material. The work product
needs to be of the best possible scientific/technical
quality to ensure an adequate and useful peer review.
2) A clear charge or statement of work seeking informed
comment on identified issues to properly focus the efforts
of the peer reviewers and ensure that their individual
efforts can be merged.
3) Some information concerning the process that the Agency
is using for the peer review, including due date of
reviewer comments and the format of those responses.
Responses should be written and submitted to the Agency by
an agreed upon deadline. In certain rare cases, oral
commentary may be sufficient. However, in such cases, a
follow-up written response for the record is required.
4) The Agency will make clear to the peer reviewers their
responsibilities. One of the major responsibilities of
peer reviewers is to ensure confidentiality of the peer
reviewed work product. Each peer reviewer must be
informed of the need for confidentiality with regard to
the release of Agency products that are stamped as "DRAFT"
or "DRAFT - Do Not Cite, Quote, or Release." Premature
release of draft Agency products, views, or positions is
inappropriate and can be damaging to the credibility of
14
-------
the Agency or the peer reviewer. While not having legal
authority, such language will be included in the charge to
the peer reviewers. Other mechanisms to use in
discouraging premature release include a disclaimer that
appears in a separate section at the front of the document
and creating the document with watermarks clearly
delineating DRAFT status (or a header or footer that
states DRAFT status) on every page. In addition, in any
solicitation for peer reviewers, the necessity for
1 confidentiality and the non-release of materials shall be
emphasized.
b) Useful, but not critical materials that may be sent to peer
reviewers include:
1) Unless a "Delphi" type process is being undertaken, it
is also useful for each peer reviewer to have the name,
address, and phone, fax, and/or Internet numbers of each
peer reviewer working on the specific review.
2) Any particularly relevant scientific articles from the
literature;
3) It is often more practical to provide comments on a
product that has line numbering added in the margin.
/ c) Peer Reviewers should be given what is needed to complete
their task they should not be overburdened with excess material.
2. Peer Reviewer Responsibilities
The Peer Review Leader is responsible for ensuring that peer
reviewers understand and comply with these responsibilities:
a) Advise the USEPA of any real or perceived conflicts-of-
interest.
b) Provide written comments in specified format by the
specified deadline.
c) Comply with the request for not disclosing draft work
products to the public.
15
-------
IV. COMPLETING A PEER REVIEW
f '-.--- ' . '
Performance of the formal peer review is not the final stage in
the development of the work product. Rather, it is ah important
stage in developing the work product, with the final work product
representing the true end of the peer review. As a result, the peer
review process closes with three major activities: evaluating
comments and recommendations, utilizing peer review comments for
completing the filial work product, and organizing and maintaining a
record of the peer review.
A. Evaluating comments and Recommendations
1. Region III must carefully evaluate and analyze all peer
review comments and recommendations. As discussed in Section III.,
a carefully crafted charge to the peer reviewers simplifies
organizing and analyzing comments. Also, any other issues that are
raised need to be identified and evaluated.
2. The validity and objectivity of the comments need to be
evaluated. Analyses include consultation with other experts/staff
within the Office and/or Agency.
3. Comments that have significant impact on time, budgetary,
and/or resource requirements need to be evaluated in consultation
with management. These comments may lead to allocation of
additional resources and a revised schedule for the completion of
the work product.
B. The Final Work Product .
1. Region III must .utilize technically sound and responsible
peer review comments and recommendations in completing the final
work product. The major issues raised by the peer reviewers need to
be identified in the front matter of the final work product, along
with information on how they were used or not used in the final work
product.
2. In the. final work product, reference the location of the
peer review record where all comments and related peer review
information can be found. The peer review record should be placed
in any associated established public docket in addition to the
Region III archive.
16
-------
c. Maintaining the Peer Review Record >:: . ^
1. The Peer Review Leader will collect and maintain: the
following materials for the peer review record, including at least:
+ the draft work product submitted for peer reviewed
.+ materials and information given to the peer
reviewer(s) ,
> comments,; information, and materials received from
the peer reviewer(s)
+ information about the peer reviewer(s) (e.g., names,
affiliations, etc.
K any logistical information (e.g., times; locations;
duration, etc.)
> the final work product .
2. The peer review record must be indexed and maintained in an
office archive. All peer review comments should be carefully
evaluated and used to revise work products where appropriate. As
discussed in .III. B. (above) it is particularly useful to craft
specific questions for peer reviewers so that subsequent comment
will be focussed. Focussing reviewers attention on issues of
interest will facilitate comment evaluation.
In some cases, preparation of a document that responds to each
comment may be appropriate. In other circumstances, comments may be
addressed in a more general manner. In any case, a clear record
must be maintained of the peer review process employed, as well as
the specific comments received. Furthermore, the product itself
must include some acknowledgement of the peer review process.
The Agency may or may not agree with comments received.
However, it is important to reflect the nature of comments received
and the extent to which these comments affect the scientific or
technical product. In addition to revising the technical product,
if appropriate, it is particularly important to communicate the
results of peer review to others who may not be versed in the
technical discipline at hand. For example, a decision maker might
find a peer review summary very helpful. The summary could present,
in general terms, the scope of the peer review process and the
thrust of the comments received. A brief statement discussing the
kinds of comments received, how the comments were reflected in the
work product, and the impact on the scientific or technical
conclusion should be included. For example, a hypothetical summary
might include language such as:
"Reviewers commented that the exposure analysis assumed no
chemical degradation as a result of treatment in a POTW.
Reviewers felt this was overly conservative and, as a result,
risks may be overestimated. *
17
-------
No information is available on the fate of the chemical in
POTWs. However, based on analogous chemicals for which data
are available, it is likely that less than 10% degradation
would occur. The risk characterization has been changed to
reflect risk as a range, thereby addressing the uncertainty
associated with our lack of information on fate of this
chemical in a POTW."
[Note: these types of discussions need to be in the final document
so-that the public sees what finally happened to the comment. ]
18
-------
Under the June 7, 1994 Peer Review Policy, ,the Administrator
has designated the Assistant Administrators and Regional
Administrators to be accountable for implementing the Policy in
their respective offices. This section provides information on
Region Ill's delegations of responsibility to (a) line management
for individual peer reviews, and (b) an off ice-wide coordination
group for general assistance and advice. It is recommended that the
individuals that perform the peer review functions discussed in this
section include these responsibilities in their performance
agreements.
A. Line Management ,
! Decision malcer(s)
a) The Regional Administrator is- accountable for the decisions
regarding the identification of major scientific and technical work
products and the mechanism(s) of peer review utilized for each of
the products. The RA designates all line managers (division and
program directors) as having the authority to make these decisions
as well. The RA and his designee(s) are also responsible for
ensuring that the peer reviews are performed and fully completed.
See Appendix B for the names of the designated decision maker(s) .
b) Specific responsibilities of the decision maker(s) are:
K Designating a Peer Review Leader to organise the peer
review
* Providing advice, guidance, and support to the Peer Review
Leader in the preparation, conduct, and. completion of the
peer review
* Establishing a realistic peer review schedule (see Section
III.)
> Designating the stage (s) of product development where peer
review is appropriate
> Ensuring that the results of peer review are carried
forward in the final work product
2. Peer Review Leader
a) The Peer Review Leader organizes and oversees the peer
review for a specific work product. This person (s) can be the
decision maker(s), but will usually be someone who is authorized by
the decision maker to prepare, perform, and bring to completion the
19
-------
peer review. The Peer"Review Leader will obtain the assistance and
support of the Peer Review Coordinator and Peer Review Group (see
below) as well as any others within the Agency to help perform the
peer review. The Leader will be chosen on a case by case basis
depending on the work product heeding peer review.
b) Specific responsibilities of the Peer Review Leader are:
* Coordinating the peer review of their assigned work
' product with the Peer Review Coordinator and Peer Review
Group
* Organizing, conducting, and completing the peer review
following the procedures outlined in this SOP,
particularly Sections III. and IV.
> Selecting the peer reviewers in consultation with others
involved with the peer review (e.g., Peer Review
Coordinator)
* Advising peer reviewers of their responsibilities
B. office coordination
1. Peer Review Coordinator
a) Each year, the Regional Administrator for Region III will
designate one person to coordinate or lead coordination of peer
review activities for this office. The Peer Review Coordinator for
FY '95 is listed in Appendix B.
b) Specific responsibilities of the Peer Review Coordinator
are:
» _ Leading the Peer Review Group as well as Region III peer
review activities
* Reporting peer review activities to the RA
* Liaison with the Science Policy Council (SPC) and Science
Advisor:
1) Representing Region III beforetthe SPC
2) Advising the SPC of any changes in SPC-reviewed list of
work products and peer review mechanisms in Appendix D
3) Participating in SPC peer review training, workshops,
etc. as requested
4) Interfacing with the Science Advisor on peer review
matters
20
-------
> Submit information on Region III peer review candidates
for each fiscal year on June 30 of the preceding year via
Appendix D reporting
* Submit any SOP revisions on July 1 of each year for review
before changes for the next fiscal year
> Provide advice, guidance, and support to the various Peer
Review Leaders for the performance of the peer reviews
> Distribute Agency-wide peer review guidance and materials
to appropriate Region III personnel, as requested
2. Peer Review Group
a) Each year, the Regional Administrator or Division/Program
Directors will also designate other persons and support staff to
constitute the Peer Review Group. The Group will assist the Peer
Review Coordinator either as special assignments or as a designated
part of their normal duties. The group members and support staff
for FY '95 are listed in. Appendix B.
b) Specific responsibilities for the Peer Review Group are:
> Assisting Peer Review Coordinator with oversight of peer-
review activities in Region III
+ Assisting with logistical and technical peer review needs
in the office, e.g., acquisition and distribution of
resources
* If needed, providing advice to decision maker(s) and/or
help them with the identification of major work products
and selection of appropriate peer review mechanism
> Periodic review of the SOP and how it is being implemented
> Assisting in the planning and management of Region Ill's
peer review program, e.g., integrating peer review plans
into office workplans
* Monitoring and maintaining Region Ill's peer review
records and* archives
3. Legal Advice
Offices/regions work regularly with individual OGC/ORC staff
assigned to Agency activities. Peer Review Leaders should continue
to initially consult with their customary OGC/ORC advisors for legal
advice or referral. Headquarters attorneys have specialties in
21
-------
specific areas and can be consulted as needed. Appendix B lists
these contacts for Region III.
C. Budget
Resources needed to implement the Peer Review Policy will need
to be requested through the usual Agency processes. The budget
formulation process within the Executive Branch is followed, after
appropriation bills are passed by Congress, by budget execution.
These two processes provide opportunities to secure resources for
activities, including peer review, carried out by Headquarters and
Regional offices. .
1. Budget Formulation (May-August annually)
a) Regional Division Directors should consult with their
Regional Assistant Administrator for Management to ensure that
budget requests have considered resources for peer review.
b) Each AA makes a request to the Administrator for resources.
Regions, through Lead Regions, coordinate with each Headquarters AA
for resource requests for their programs. -
c) Resource decisions are made by each AA for submission to
the Administrator for consideration in preparing the overall Agency
budget.request.
d) The Agency's budget request is submitted to the Office of
Management and Budget for consideration and inclusion in the
President's Budget request to Congress. (September-February)
2. Budget Execution (Aucrust-Novei
a) . Following enactment of the Agency's appropriation by the
Congress, each AA and Region prepares an operating plan based on the
President's Budget, adjusted by the enacted appropriation. Peer
Review Leaders should consult with their respective Regional
Assistant Administrator for Management concerning resources to
include in the operating plan for peer review.
b) AAs and RAs can make discretionary choices on the use of
resources in their operating plan within the structures required by
Congress, appropriation law and the Agency's justification for those
resources.
c) AAs and RAs may raise a specific activity as an Agency-wide
concern to the Administrator for resource redirections as part of
the development of the enacted operating plan.
22
-------
D. Annual Reviews ' _' ,
1. The Peer Review Coordinator will organize an annual preview
of Region III expected work products for the next fiscal year and
submit this information to the SPC by June 30 of each year as
required for Appendix D.
/ .
2. The Peer Review Coordinator will organize an annual review
to assess the function of the SOP in practice and to edit the body
of the SOP and appendices as needed. The Coordinator will consult
with the SPC regarding any proposed changes by June 30 for the next
fiscal year.
23
-------
APPENDIX A :N < . .
U.S. ENVIRONMENTAL.PROTECTION AGENCY PEER REVIEW POLICY
A-l
-------
PEER REVIEW AND" PEER INVOLVEMENT/ :"
AT THE U. S. ENVIRONMENTAL PROTECTION AGENCY
This document establishes the . policy of the United States
Environmental Protection Agency (EPA) for peer review of
scientifically and technically based work products that are
intended to support Agency decisions. Peer review is presented in
the context of the broader concept, peer involvement.
BACKGROUND .
The report "Safeguarding the Future: Credible Science,
Credible Decisions"1 focused on the state of science at EPA.. The
panel of experts who prepared the report emphasized the importance
of peer review, especially external peer review, and the need for
broader and more systematic use of it at EPA to evaluate scientific
and. technical work products. Their specific recommendation
regarding peer review reads as follows:
"Quality assurance and peer review should be applied to
the planning and results of all scientific and technical
efforts to obtain data used for guidance and decisions at
EPA, including such efforts in the program and regional
offices. Such a requirement is essential if EPA is to be
perceived as a credible, unbiased source of environmental
and health information, both in the United States and
throughout the world."
In response to this recommendation, then-Administrator Reilly
directed staff to develop an EPA-wide policy statement, which he
issued in January, 1993. The paragraphs below preserve the core of
that earlier statement while updating it to specify the role of the
Science Policy Council in guiding further implementation of the
policy. Effective use of peer review is indispensable for
fulfilling the EPA mission and therefore deserves high-priority
attention from program managers and scientists within all pertinent
Headquarters and Regional Offices.
EPA/600/9-91/050, March 1992.
-------
PEER INVOLVEMENT AND PEER'REVIEW.
EPA strives to ensure that the" scientific and technical
underpinnings of its decisions meet two important criteria: they
should "be. based upon .the best current knowledge from science,
engineering/ and other/fdomains of technical expertise; and they
should be judged credible by those who deal with the Agency. EPA
staff therefore frequently rely upon peer involvement that is,
they augment their capabilities by inviting relevant subject-matter
experts from outside the program to become involved in one or more
aspects of the development of the work products that support
policies and actions.
One particularly, important type of peer involvement occurs
when scientifically and "technically based work products undergo
peer review that is, when they are evaluated by relevant
experts from outside1 the program who are peers of the program
staff, consultants, and/or contractor personnel who prepared the
product. Properly applied, peer review not only enriches the
quality of work products-but also adds a degree of credibility that
-cannot be achieved in any other way. Further, peer review early ,in
the development of work: products in some cases may conserve future
resources by steering the development along the most efficacious
course. '
Peer review generally takes one of two forms. The review team
may consist primarily of .relevant experts from within^ EPA, albeit
individuals who have no other involvement with respect^to the work
product that is to rbe evaluated (internal peer review) . Or the
review team may consist primarily of independent experts from
outside EPA (external peer review).
POLICY STATEMENT ..
Major scientifically and technically based work products
related to Agency decisions normally should be peer-reviewed.
Agency managers within Headquarters, Regions, laboratories, and
field components determine and are accountable for the decision
whether to employ peer review in particular instances and, if so,
its character, scope, and timing. These decisions are made in
conformance with program goals and priorities, resource
constraints, and statutory or court-ordered deadlines. For those
work products that are intended to support the most important
decisions or that have special importance in their own right,
external peer review-is the .procedure-of choice. Peer review is
not restricted to the penultimate version of work products; in
fact, peer review at the planning stage can often be extremely
beneficial.
-------
SCOPE
Agendy managers routinely make regulatory and other decisions
that necessarily involve many different considerations. This
policy\applies to major work products that are primarily scientific
and technical in nature and may contribute to the basis for policy
or regulatory decisions. By contrast, this policy does not apply
to.nonmajor or nontechnical matters that Agency managers consider
as they make decisions. Similarly, this policy does not apply to
these ultimate decisions.
This policy applies where appropriate, as determined by the
National and Regional Program Managers, to major scientifically and
technically based work products initiated subsequent to the date of
issuance. Peer review should be employed to the extent-reasonable
to relevant work products that currently are-under development.
This policy does not apply to the bases for past decisions, unless
and .until the relevant scientific and technical issues are
considered anew in the Agency's decision-making processes.
Except where it is required by law, formal peer review (as
distinguished from the Agency's normal internal review procedures)
should be conducted in a manner that will not cause EPA to miss or
need extension of a statutory or court-ordered deadline. Agency
managers still may undertake peer review if it can be conducted
concurrently with necessary rulemaking steps.
LEGAL EFFECT
This policy statement does not establish or affect legal
rights or obligations. Rather, it confirms the importance of peer
review where appropriate, outlines relevant principles, and
identifies factors Agency staff should consider in. implement ing the
policy. On a continuing basis, Agency management is expected to
evaluate the policy as well as the results of its application
throughout the Agency and undertake revisions as necessary.
Therefore, the policy do.es not stand alone; nor does it establish
a binding. norm that is finally determinative .of the issues
addressed. Minor variations in its application from one instance
to another are appropriate and expected; they thus are not a
legitimate basis for delaying or complicating action on otherwise
satisfactory scientific, technical, and regulatory products.
Except where provided otherwise by law, peer review is not a
formal part of or substitute for notice and comment rulemaking or
adjudicative procedures. EPA's decision whether to conduct peer
review in any particular case is wholly within the Agency's
discretion. Similarly, nothing in this policy creates a legal
requirement that EPA respond to peer reviewers. However, to the
extent that EPA decisions rely on scientific and technical work
products that have been subjected to peer review, the remarks of
peer reviewers should be included in the record for that decision.
-------
IMPLEMENTATION ,-
The Science Policy Council is responsible for. overseeing
Agency-wide implementation. Its responsibilities include promoting
consistent" interpretation, assessing Agency-wide progress, and
developing recommendations for revisions, of the policy as
necessary.
The Science Policy Council will oversee a peer-review work
group, which will include representatives from program units
throughout EPA to effect a consistent,. workable implementation of
the policy. The work group will assist the programs in (i)
formulating and, as necessary, revising standard operating
procedures (SOPs) for peer review consistent with this policy; (2)
identifying work products that are subject to review; and (3) for
each major work product, selecting an appropriate level and timing
of peer review. .
In assisting, the programs, the work group will take into
account statutory and court deadlines, resource implications, and
availability of disinterested peer reviewers. The group will work
closely with Headquarters offices and the Regional Offices toward
ensuring 'effective, efficient uses of: peer review in supporting
their mission objectives. However, the Assistant Administrators
and Regional Administrators remain ultimately responsible for
developing SOPs, identifying work products subject to peer review,
determining the type and timing of such review, documenting the
process and outcome of each peer review, and otherwise implementing
the policy within their organizational units.
Because peer review can be time-consuming and expensive,
Agency managers within Headquarters, Regions, laboratories, and
field components are expected to plan carefully with respect to its
use taking account of program priorities, " resource
considerations, and any other relevant constraints as well as the
policy goal of achieving high-quality, credible underpinnings for
decisions. External peer reviewers should be chosen carefully to
ensure an independent and objective evaluation. The affiliations
of peer reviewers should be identified on the public record, so as
to avoid undercutting the credibility of the peer-review process by
conflicts of interest.
This policy is effective immediately. The peer-review work
group mentioned above will identify the focal point to whom
comments and questions should be addressed and, from time to time,
will provide further information 'about implementation activities.
APPROVED: r ^ W^-*SPU ^^-jj^^c^^T^^-^ DATE: Jim 7
CAROL M. BROWNER, ADMINISTRATOR J^"
-------
APPENDIX B - : , :
KEY. PERSONNEL
' ' '
I. Individuals Involved.in the Peer Review
A. Line Management
i. Decision maker(si - Division or Program Directors
B. Office Coordination
2. Peer Review coordinator - Rick Kutz
3. Peer Review Group
Individuals serving on Peer Review Group in FY95:
Chair: Greene Jones
Environmental Services Division
215-597-4532
1 year term of service
Members: Henry Brubaker
Office of Policy and Management
215-597-3735
1 year term, of service
Bill Bulman
Water Management Division
215-597-9966
1 year term of service
Charlene Creamer
Chesapeake Bay Program
215-597-7771
1 year term of service
Ken Kryszczun
Hazardous Waste Management Division
215-597-9401
1 year term of service
Ruth Podems
Office of External Affairs
215-597-4164
1 year term of service
Greg Allen
Environmental Services Division
410-573-6841
1 year term of service
B-l
-------
Jim Smith. . : . .
Air, Radiation and Toxics Division .
215-597-6554 . . -
1 year term of service
Individuals providing support for the Peer Review Group in FY95:
Coordination and Facilitation: Greg Allen
Environmental Services Division
410-573-6841
Clerical Support: Natalie Wagner
Environmental Services Division v
. 410-573-6849
II. Legal Advice
.The USEPA offices/regions work regularly .with individual
OGC/ORC staff assigned to Agency activities. Peer Review Leaders
should continue to initially consult with their customary OGC/ORC
advisers for legal advice or referral.
Regional Assistance: Janet Williams
: 215-597-7748
Office of Regional Counsel
The headquarters attorneys listed below specialize in the
identified areas and may be consulted, as appropriate.
Donald Sadowski
Confidential Business Information
Office of General Counsel
Richard Feldman
Conflicts of Interest Disclosure
Office of General Counsel
Hale Hawbecker
FACA Issues
Office of General Counsel
B-2
-------
. APPENDIX C .'-_-.
LISTING OF REPRESENTATIVE OFFICE PRODUCTS IN EACH
CATEGORY OVER THE PAST 3 YEARS
- ' ~ ~" ~
Category/Work Product Number of ; Mechanism^s)
\ Work Products of Peer
Peer Reviewed Review Used
Publications in Scientific Journ. 1 Internal
1. Isokindunos: Equal Risk Contours Downwind of An Air Toxics
Source
Region III Reports and Documents 10 Internal
1. Sinkhole Brochur.es
2. Exposure Point Concentration in Groundwater
3. Habitat Requirements for Ches. Bay Living Resources
Regional Scientific Plans 1 Internal,
1. Mid-Atlantic Highlands Assessment
C-l
-------
s, APPENDIX D -
OFFICE SELECTIONS FOR PEER REVIEW IN FISCAL YEAR '95 .
Category/Work Product
Mechanism of PC»P»T-
Review Projected
Laboratory Workbook for NPDES Tests
Computerized Maintenance Mgt. for POTWs
Effects of Abandoned Wells on Groundwater
Quality
Lead-210 Dating of Sediment From Presque
Isle Bay
Constructed Wetlands Design Handbook
Chester Cumulative Risk^ Study
Characterization and Monitoring Plan .for
the Deleware Estuary Program
Scientific Support Documentation for 404C
Actions
» i
Coastal Eutrophication Survey
Regional Indicators Strategy for
the State of the. Region Report
Statistical Power Analysis of the Ches. Bay
Monitoring Program
c
Riparian Forced Buffer Technical
Emphasis Document
Trends in Phosphorous, Nitrogen, Secchi Depth
and Dissolved Oxygen in Ches. Bay
Comparison of Patuxent River .Nearshore and
Mid-Channel Water Quality Data, 1985 to 1992
EPA HQ/State
EPA HQ/State
EPA HQ/Region III
External-ANS*
EPA HQ/States/PSU+
EPA HQ/Region III
EPA HQ/Region III
EPA HQ/Region III
EPA HQ/Region III
EPA HQ/Region III
External-States
Internal/External
Internal/External
Internal/External
* Academy of Natural Sciences
+ Penn State University
D-l
-------
APPENDIX E
RESOURCES FOR PEER REVIEW SERVICES
f* i ~-
* -
I. ACQUISITION OF PEER REVIEW SERVICES
A range of peer review services are available to the USEPA
including internal, external (voluntary, purchase order, contractor
employee), and Special Government Employee (SGE). The mechanism
selected is generally based on the nature of the scientific or
technical .work product.
/
A. Voluntary Services.
As a general matter, the USEPA can ask outside experts to peer
review Agency products without compensation. Several environmental
statues authorize the USEPA to engage in cooperative activities with
"institutions, organizations, and individuals." Accordingly, such
groups may cooperate with the USEPA in research and informational
activities related to the following statutes: Clean Air Act at 42
U.S.C. §7402; Clean Water Act at 33 U.S.C. §1254; Resource
Conservation and Recovery Act at 42 U.S.C. §6981; and Safe Drinking
Water Act at 42 U.S.C. §300j-l.
However, if a person or organization wishes to cooperate with
the USEPA to perform some sort of voluntary activity, program
officials should ask them to sign a "Visitor/Guest Worker"
agreement, currently used at USEPA laboratories. This is because
the Antir^Deficiency Act at 31 U.S.C. §1342 prohibits so-called
"voluntary" services that could give rise to a claim for
compensation. Moreover, accepting "volunteers" to fill USEPA
employee "slots" might violate the personnel statutes and might give
rise to a valid claim for compensation in violation -of the Anti-
Deficiency Act.
B. Contracts
The USEPA may contract for peer review services. The contract
may be written solely for peer reviews or be included as part of an
umbrella contract, which calls for performance of other tasks as
well.
For assistance in preparing the necessary pre-award documents,
program officials should consult The Cookbook: How to Get Contracts
Awarded in EPA and Chapter 2 of the Contracts Management Manual
(CMM). The following discussion identifies five key elements that
should be considered in contracts for peer reviews: Statements of
Work (SOWs), Advisory and Assistance Services (AAS) or Sensitive
Activities, Management Controls, Identification of Peer Reviewers,
E-l
-------
and Federal Advisory Committees. Special considerations for small
purchases are discussed in section I.e. Small Purchases.
1. Statement of work (SOW)
The SOW must clearly specify that the contractor is responsible
for preparing peer review evaluations and set forth guidelines for
the peer review of scientific or technical documents. The
contractor may perform the peer review with in-house staff,
subcontractors or consultants. Any guidelines for performing peer
reviews to ensure soundness and defensibility must be developed by
the program office and made part of the contract. The contractor
would then ensure that the peer reviews adhered, to the guidelines.
The SOW cannot simply define the role of the prime contractor
as arranging for the services of others to perform peer reviews and
logistics for meetings. Unless the prime contractor is" clearly
tasked with responsibility for performing peer reviews, individual
peer reviewers' fees and associated travel expenses are not payable
under the contract.
The USEPA may pay for the reviewer's comments or evaluation,
and also for attendance at a meeting with the Agency .and other
reviewers to discuss the results of the peer review. If the SOW
calls for the preparation of comments 'or an evaluation,, and
specifies a meeting with the Agency and other peer reviewers to
discuss the results of the peer review, payment is appropriate. The
peer reviewer's attendance at the meeting would then be part of
contract performance.
2. Advisory and assistance services (AAS) or sensitive
activities -
Contracts that provide services that support or improve Agency
decision'-making or policy development are subject to special
management controls. These services include: "...those services
acquired from .non-Governmental sources by contract or by personnel
appointment to support or improve agency development, decision-
making, management, and administration, or to support or improve the
operation of management systems. Such services may take the form of
information, advice, opinions, alternatives, conclusions,
recommendations, training, and direct assistance." For additional
information on advisory and assistance services and sensitive
activities, program officials should review EPA Order 1900.2,
Contracting at EPA and Chapter»2 of the CMM.
New contracts for these services require management approvals
prior to issuance of the solicitation. The following thresholds
have been established for approval of these justifications:
E-2
-------
Contract Amount Approved Concurrence By
$25,000,000 and over -Both the Program Office Program Office
Assistant/Associate Senior
Administrator (AA) or Resource
' equivalent and the Official
Deputy AA for Finance (SRO)
and Acquisition,
Office of Administration
and Resources Management
$5,000,000 to Both the Program Office Program
$24,999,999 AA and Office Director Office SRO
Office of Acquisition
Management
$25,000 to $4,999,999 Program Office AA - Program
Office SRO
(See the June 30, 1993, Interim Procedures for Approval and
Documentation of Procurements Involving AAS.) Note, these
procedures and approval levels will be changed upon implementation
of Office of Federal Procurement Policy (OFPP) Policy Letter 93-1,
Management Oversight of Service Contracting in Chapter 2 of the CMM.
3. Management controls
Contracting for peer review services is permitted. However,
because of the potential for improper use of these contracts,
special management controls are required.
a. Inherently governmental functions (IGFs)
OFPP Policy Letter 92-1, dated September 23, 1992,
describes (a) functions that are inherently governmental and
must be performed only by Government employees and (b)
functions that may be contracted, but so closely support
Government employees in their performance of IGFs that the
contract terms and performance require close scrutiny by
Federal officials.
Peer reviews represent only a contractor's
recommendations, advice or analysis of a document. Agen-cy
officials must make the official Agency decision regarding
acceptability of the document. To ensure that Agency officials
are not improperly influenced by recommendations in the peer
review, management controls must be included in the contract.
One possible control would be to require the peer reviewers to
submit with their evaluations or comments a description of the
procedures used to arrive at their recommendations; a summary
of their findings; a list of sources relied upon; and make
clear the methods and considerations upon which their
E-3
-------
recommendations are based. To the extent possible, the
contract-should set forth any guidelines or criteria for
performance of the- peer review. Agency officials should
document their evaluations of the quality and validity of .the
peer review. - ...
%.
b. Conflict of interest (COI)
Another important factor is that the objectivity of the^
peer review should not be improperly influenced or .undermined
by the contractor performing the review. To. identify and avoid
orv mitigate actual or potential COI, the contract should
include controls. Such controls might require the contractor
to report on prior and current work, and prior clients that
might create COI. Other controls might include Agency review
and placing limits or advance approval on future work. There
should also be procedures implemented to assure that the
contractor does not gain an unfair advantage in future
requirements as a result of their performance of peer.reviews.
Program officials should consult the Contracting Officer ('CO)
for special contract clauses.
The EPA Acquisition Regulations (EPAAR) at 48 CFR Subpart
1509.5 generally mandates conflicts of interest solicitation
provisions and contract clauses for contracts over $25,000, but
makes them optional for small purchases of $25,000 or less.
See 48 CFR 1509.508(b) & (c)
48 CFR 1552.20S-70, -71 & -72 --;
Contract for peer review services; An USEPA contracting
officer will include conflicts of interest.solicitation ... '
provisions and contract clauses as a matter of course without
involvement by the USEPA project officer, if the peer review
services are obtained pursuant to a contract over $25,000. If
the peer review services are subcontracted pursuant to a prime .
contract (over $25,000), then the prime contractor is
ordinarily required to include a conflicts of interest clause
substantially similar to the conflicts of interest clause in
the primary contract in its subcontract to the peer reviewer.
Although the USEPA contracting officer and/or prime
contractor, has the primary responsibility to include the"
required conflicts of interest provisions/clauses, the USEPA
project officer may nevertheless wish to:
1. Highlight the conflicts of interest requirements in the
Scope of Work for the procurement of the peer review
services.
E-4
-------
2. Develop a specific conflicts of interest clause regarding
the peer review, at issue as a substitute to the standard
conflicts of interest clause. .
3. Review the solicitation/contract to make sure that the
. required conflicts of interest clause has been included.
Small purchase order for peer review services; Although
conflicts of interests requirements are optional for small
purchases, they are nevertheless a good idea. Accordingly, an
USEPA project officer obtaining peer review services with a
small purchase order should request the purchasing
agent/contracting officer to include a conflict of interest
solicitation provision and contract clause in the purchase
order.
c. Confidential business information (CBI)/Privacy Act
protected information and other sensitive information
When peer reviewers are not employees of the United States
Government, 'it is unlikely that the USEPA will have authority
to give reviewers access to confidential business information
in the absence of -consent for such disclosure by the CBI
submitter. Therefore, all documents provided to non-Federal
reviewers must be screened for information claimed as CBI.
Even where business information has not been explicitly claimed
as CBI, if it is of a kind where the submitter might be
expected to object to its release, prior to release the
submitter must be asked whether it wishes to assert a claim,
unless the submitter has previously been informed that failure
to assert a CBI claim may result in disclosure without notice.
Language should be included in the contract to clearly identify
any required procedures or processes prior to release of any
protected information, including any requirements for
confidentiality agreements, as well as limits on use and
disclosure of the data by contractor personnel.
d. Personal services
Under contracts, the USEPA may not engage the peer.
reviewers in any improper personal services relationships,
i.e., an arrangement under which contractor personnel are
subject to relatively continuous supervision and direct control
by an Agency official or employee. These relationships are
characterized as one whe^e the contractor, employee interacts
with the Agency in a manner similar to that of a federal
employee.
To avoid these improper relationships, program officials
should write well-defined SOWs. The SOWs should set forth the
requirements in detail for work to be performed independently,
including the manner in which it will be evaluated. The SOW
E-5
-------
must set forth what work is to be performed;not how the work is
to be performed. Technical direction may be/used to clarify
ambiguous technical.requirements to ensure efficient and
effective contractor performance, and is not considered
supervision or assignment of tasks. For additional
information, program officials should consult EPA Order
1901.1A. Use of Contractor Services to Avoid Improper
Contracting Relationships.
'. .
4. Identification of peer reviewers
, /
Program-officials cannot interfere in a contractor's
ability to perform work by "selecting" who will perform the
peer review. The Federal Acquisition Regulation (FAR) governs
the CO's and program officials7 relationship with the
contractor. , .
However, the CO does have the ability to review and
consent to subcontractors and consultants. The contract can
also specify which individuals are key personnel and include
peer reviewers. The FAR and Agency implementing regulations set
forth rules governing the use and replacement of key personnel.
Further, the -contract can require workplans for approval by the
Agency, wherein the contractor will propose the peer reviewers
it is considering for selection. .
Program officials should identify the qualifications
required to perform the review work and the criteria for
technical acceptability. The USEPA may identify., a pool of
qualified subcontractors and consultants to the prime
contractor, but cannot direct the use of any particular
subcontractor or consultant.
5. Federal Advisory Committees
The Federal-Advisory Committee Act (FACA) requires that the
Agency develop a charter, maintain balanced membership, and hold
open meetings when it establishes an "advisory committee." An
advisory committee is any group established by the USEPA for the
purpose of providing advice as a group to the Agency. It does not
include advice coming from individual attendees at a meeting or
groups established by a non-federal entity such' as an EPA
contractor. See, 41 C.F.R. §101-6.1004(i). If a contractor
convened peer review panel will provide advice and recommendations
to the USEPA as a group, then the group is subject to FACA
chartering requirements.
E-6
-------
C. Small Purchases .
The acquisition of supplies or nonpersonal services from the
open market and, on a sole source basis when the aggregate amount
involved in any one transaction does not exceed $25,000 constitutes
a "small purchase". The USEPA has developed a guide entitled Small
Purchases, A Guide for Program Offices, which provides basic
information about small purchases and purchase orders. The guide
should be used in preparing a procurement request (PR) for the
purchase of peer review services under the small purchase
limitation.
The same considerations in the preceding discussion on IGFs,
COI, access to CBI, and personal services apply to small purchases.
Normally, the Government issues a small purchase order directly to
the individual peer reviewer, instead of to a prime contractor who
may subcontract for performance of the peer review.
Approvals
All small purchases for peer reviews are considered AAS. The
approval level for small purchase AAS is at least one,organizational
level above the i-nitiating office. When award is made during the
fourth quarter of the fiscal year, approval must be received from a
program official-at .least two organizational levels above the
initiating office.
Competition
The FAR requires competition for purchases in excess of $2,500.
To accomplish competition, Purchasing Agents will solicit quotes
from three vendors unless a requirement is justified on a sole
source basis. A sole source justification must be detailed and
fully describe the circumstances supporting the justification.
ProgranTofficials should refer to EPAAR Subpart 1513.170-1 for more
information on preparing justifications. Poor planning does not
constitute a valid basis for a sole source justification; CDs will
make a small purchase award to the vendor with the lowest offered
price.
Procurement Requests
Program Officers should include the following in all PRs- for
the purchase of peer reviews:
1. A fixed-price amount at or below the small purchase
limitation.
2. A detailed description of the requested services,
inclusive of:
a. Total quantity per line item;
E-7
-------
b. . Estimated unit price per line item;.
c. Total cost per line item;
d.' Specific deliverables for each line item; and
- e. Total cost of the purchase request.
3. The name, address, and phone number of three
competitive sources if the value of the request is $2,500 or more.
a. Reference FAR Subpart 3.6 and Environmental
Protection Agency Acquisition Regulation (EPAAR) -Subpart 1503.601
regarding sources'from Government employees or organizations owned
and controlled by them.
b. Provide sources from small businesses if available.
4\ - If the request is a sole source purchase,
justification must be provided in accordance with the EPAR Subpart
1513.170-1.
II. TRAVEL
A. Contracts and Purchase Orders
Funds obligated on a contract or purchase order are available
to pay for the costs of producing the peer review including the
travel costs and fee of the peer reviewer.
The USEPA acquires peer reviews through small purchases issued
.directly to peer reviewers or through contracts with companies,
which then acquire the services of peer reviewers. By issuing a
purchase order or awarding a contract for a peer review, the USEPA
may pay not only for the peer reviewer's comments, but also for his
or her attendance at a meeting with the Agency and other reviewers
to discuss his or her comments. The scope of work of the contract
must include the organization of peer reviews and indicate whether
the contractor will be required to discuss a specific peer review
work product with the Agency and/or with other peer reviewers.
Attendance at a meeting to discuss a peer review work product would
then be part of the contract's performance. Thus, the contract may
serve as the mechanism to pay for a peer reviewer's fee and
associated travel expenses to provide.comments to the USEPA.
E-8
-------
B. Special Government Employees ,
Travel and per diem expenses of experts hired as SGEs for peer
review may only be paid through the issuance of invitational travel
orders (5 U.S.C. §5703) . These invitational travel and per diem
expenses should be charged to an appropriate USEPA travel
account. - -
/.
. Members of the SAB, SAP, and other FACA advisory committees are
hired as SGEs. It is not appropriate to reimburse travel or per
diem expenses of advisory committee members or other SGEs through a
contract.
R3 00038
3 EPA Reg. Ill Standard Operating
] Procedures for Peer Review of Major
~\ Technical and Scientific Products
U. S. Environmental Protection Agency
Environmental Science Center
701 Mapes Road
Ft. Meade, MD 20755-5350
s
1
" C 'm
111
co: °
&s
-
E-9
------- |