v>EPA
United States Office of Research
Environmental and Development
Protection Agency Washington, DC 20460
October 1990
Operational Procedures
for Peer Review
First Edition
Environmental Monitoring
and Assessment Program
-------
Preface
This document has been prepared to assist the Steering Committee, the Technical
Directors, and the Techmcal Coordinators in preparing for, organizing, and
conducting scientific reviews of plans and products from the EnVIronmental
Monitoring and Assessment Program (EMAP). As its title indicates, this docu-
ment is the first issue; it is anticipated that these operational procedures will be
refined as the program progresses. Users of these procedures are encouraged to
send their recommendations for revision to
Jay Messer
AREAL
U.S. EPA (MD-75)
Research Triangle Park, NC 27711
October 31,1990
-------
Operational Procedures for Peer Review
First Edition
Background
EMAP is a highly visible research and
monitoring program with an ambitious
agenda. Its concepts, approaches, and
products will be subject to intense
scrutiny by the Office of Research and
Development (ORO), other EPA Pro-
gram Offices and Regions, as well as
other federal agencies and the scien-
tific community. EMAP's success will
be judged by its ability to promote
sound, scientifically based decisions
that lead to more cost-effective envi-
ronmental regulations and manage-
ment EMAP data and interpretive
products must have broad acceptance
by the scientific leadership in this
country if they are to have a significant
influence on the setting of this nation's
environmental policies.
EMAP must therefore adopt and institu-
tionalize the highest standards for
scientific peer review as well as foster
an open forum among non-scientific
parties with a stake in the program's
outcomes, including business and
citizen groups. Peer reviewers must be
leading experts in the scientific com-
munity, and public com mentors should
be leading spokesmen for non-scien-
tific interest groups.
Several levels of peer review are
required to ensure not only that appro-
priate attention is given to the details,
but also that the details, when inte-
grated, provide answers that are fully
relevant and responsive to the deci-
sion-making process. Moreover, peer
review must be timely to ensure that
plans can be modified to incorporate
review comments and still meet imple-
mentation or deliverable deadlines.
Types of ~eviews
EMAP has four levels of reviews.
These include reviews conducted by
the EPA Science Advisory Board (SAB)
and the National Academy of Sciences
(NAS), as well as specialist reviews and
internal reviews.
Levell: Ecological Effects Committee
(SAB)
Purpose: Ensure relevance to ORD's
mission, including consistency with the
goals of the Ecological Risk Assessment
Program and other relevant ORD pro-
grams.
The Office of the Administrator is
responsible for orchestrating all re-
views conducted by the SAB. Prepara-
tion for SAB reviews is coordinated by
the Director of the Office of Modeling,
Monitoring Systems, and Quality
Assurance (OMMSQA); and the Assis-
tant Administrator for ORO is respon-
sible for funding. In fY 1991, the SAB
will review and comment on the
conceptual framework for integrating
EMAP with ORO's Ecological Risk
Assessment Program. The Board will
also provide an overall program review
using available documents as sources
of background information, including
the Indicator Report, draft Design and
Characterization Plans, and the Draft
EMAP five- and Twenty-five-Year
Strategies.
1
October 31,1990
-------
£MAP Peer RerJlew Procedures (First Edltwn)
Level 2: Joint Board of Ecological
Studies and ToxicologylWater Sci-
ences Technology Board Committee
(NAS)
Purpose: Determine overall scientific
merit and integration both internally and
with other government-sponsored moni-
toring programs.
Preparation for these reviews is coordi-
nated by the Director of OMMSQA,
who also is responsible for funding.
During FY 1991-1993, the NAS review
committee will hold approximately
12 meetings and produce two or three
written reports. In addition to review-
ing the documents listed earlier, the
committee will review the National
Research Plans for each Resource
Group and the completion reports for
the FY 1990 and FY 1991 demonstra-
tion projects for Estuaries, Forests, and
Surface Waters.
Leve13: Specialist Peer Review
Panels
Purpose: Determine scientific merit of
research plans for individual pilot studies,
demonstration projects, or implementa-
tion phases, including specific designs
and methods, logistics, quality assurance,
and dilta management and analysis plans;
specialized program components such as
statistical methods, remote sensing, or
information management systems; and
final project reports and dilta bases.
These reviews are the responsibility of
the appropriate Associate Director.
Preparation for the reviews is coordi-
nated by the Technical Director or
Technical Coordinator, who also is
responsible for funding. Specialist
review panels are assembled from such
scientific organizations as the Ameri-
can Statistical Association and Estua-
rine Research Federation; panels also
may be ad hoc, multi-organizational
committees. Examples of specialist
peer reviews conducted In FY 1990
October 31, 1990
include the Estuarine Research Federa-
tion panel review of the Estuarine
Virginian Province demonstration
project; the peer review of the Indica-
tor Report by 22 independent, subject-
matter specialists; the American Statis-
tical Association panel review of the
draft Design Report; and the review of
the draft Landscape Characterization
plan by a multi-organizational peer
review panel.
Leve14: Internal Reviews (EPA
Laboratory Scientists and Agency
Cooperators)
Purpose: Ensure consistency of research
plans and applications guides with overall
project and laboratory missions and
procedures, including scientific, person-
nel, and budgetary issues.
Preparation for internal reviews is
coordinated by the appropriate Techni-
cal Director or Technical Coordinator,
who also is responsible for funding.
Although there are currently no models
for this level of review within EMAP,
participants should include all partici-
pating Laboratory Directors, other line
managers and technical personnel, and
important cooperators from other
agencies.
Review Procedures
Level 3 (specialist peer reviews) and
Level 4 (internal reviews) are likely to
be the most frequently conducted
EMAP reviews. This section describes
detailed procedures demonstrated to
improve the probability of successful
specialist peer reviews and provides
minimal guidelines for conducting
internal reviews.
Specialist Peer Reviews
1. Level 3 reviews should precede
any major activity such as a pilot
or demonstration project or field
Implementation requIring a go/no-
2
-------
£MAP Peer Review Procedures (First EditIOn)
go decision. The review should be
conducted no less than 5 months
(or 150 days) prior to initiation of
sampling so that recommendations
for changes can be received no
less than 4 months (or 120 days)
prior to initiation and incorporated
into the research, project, or
activity plans. A scoping meeting
should be used to initiate the peer
review process.
2. The panel Chair should be selected
about 3 months prior to the peer
review. The Chair should be a
recognized expert in the disciplin-
ary area, be well-organized and
decision-oriented, and have previ-
ous experience, if possible, as a
panel Chair.
3. The Technical Director or Techni-
cal Coordinator and panel Chair
should concur on the format and
length of the peer review. A
typical format includes 1/2-1 day of
presentations by the EMAP Group;
1/2-1 day of questions for the
presenters or Group by the panel;
and 1-2 days for the panel to
complete the review, summarize
the general review comments, and
present the summary to the EMAP
Group.
4. The Technical Director or Techni-
cal Coordinator should work with
the panel Chair to select the review
committee. This procedure might
include submission of a list of
possible reviewers from which the
panel Chair can select candidate
reviewers. Candidates who could
be perceived to have a conflict of
interest, such as employees of EPA
laboratories, personnel who are
closely associated with the
project's planning or implementa-
tion or who receive funding for
participation in the project, should
not be selected. Committee
members should have a broad
range of backgrounds and include
scientists from EPA Regional or
Program Offices, other federal
agencies, the academic commu-
nity, and the private sector. A
typical panel is comprised of 5-
10 reviewers; a larger number of
reviewers can impede the review
process. Reviewers should be
selected and contacted no later
than 2 months prior to peer re-
views.
Th£ EMAP Deputy Director screens
th£ initial list of candidate reviewers
and approves th£ final list of selected
reviewers.
5. An initial mailing should be sent to
each panel member thanking him
for agreeing to serve on the panel.
This mailing also includes
o a preliminary agenda,
o meeting location and dates,
o names and telephone numbers
of other review committee
members, and
o general information on travel,
per diem, hotel, and hono-
rarium (if appropriate).
This package is forwarded immedi-
ately following acceptance by
panel members.
6. A final mailing should be for-
warded so it is received by the
peer reviewers at least 30 days
prior to the peer review meeting or
requested dates for comments
when no review meeting is held.
This mailing includes the
o final agenda,
o charge to the reviewers,
o list of review questions,
o research plan or review docu-
ment, and
o any other pertinent matenal.
3
October 31,1990
-------
EMAP Peer ReuJeW Procedures (First Edition)
The charge to reviewers and re-
view questions should be re-
viewed by the appropriate Associ-
ate Director prior to transmittal.
Documents that are to be re-
viewed should have been subject
to a technical editor, and all fig-
ures and tables should be of draft
quality at a minimum.
7. All presentations should be pre-
sented to the appropriate Associ-
ate Director in a dry run held at
least one week prior to peer panel
presentations; one week should
be sufficient time to modify slides
or overheads prior to the review.
The purpose of the dry run is to
prepare the presentation team for
the questions likely to be asked
by peer reviewers, and challeng-
ing questions must therefore be
posed to the presenters. Ques-
tions asked at previous peer re-
views should be reviewed and
other Technical Directors or
Technical Coordinators that have
undergone peer review should be
asked to contribute questions.
Graphics should be well prepared
and professional and, to the ex-
tent possible, consistent in format
and style.
8. Peer review meetings may be fa-
cilitated by a contractor not associ-
ated with development of the plan,
but no EMAP personnel should be
perceived to be "leading" the pro-
ceedings. The Associate Director
should assist the panel in resolving
issues or adjusting formats and
schedules, but should not direct
the proceedings away from legiti-
mate areas of concern and debate.
9. Interested parties in government
(including appropriate EPA Re-
gional or Program Offices and
other federal agencies), business
October 31,1990
(e.g., timber, fishing, or power
industry groups), and environ-
mental organizations (e.g., Save
the Bay, the Audubon Society, the
Natural Resources Defense Coun-
cil) should be invited to public
peer panel reviews and allowed
time for questions. The invitee
list should be approved by the
appropriate Associate Director
and Technical Director or Techni-
cal Coordinator.
10. Courtesy copies of all documents
should be sent to the EMAP Steer-
ing Committee, all Technical
Directors and Technical Coordi-
nators, involved Laboratory Di-
rectors, and cooperators in other
federal agencies at the time they
are sent to peer reviewers. Cour-
tesy copies of reviewers' com-
ments should be sent as a follow-
up.
11. A follow-up letter of appreciation
should be forwarded to each peer
reviewer within 7 week of the
peer review or receipt of review
comments, if conducted as a writ-
ten review.
12. A detailed response to the review-
ers' comments should be pre-
pared and sent to the reviewers,
EMAP Steering Committee, Tech-
nical Directors, and Technical
Coordinators within 6 weeks of
the review. The revised docu-
ment, upon completion, should
also be forwarded to these indi-
viduals. The general schematic of
the peer review process is shown
in Figure 1.
Interna'Reviews
Internal reviews should be held at least
once per year and at least 45 days
prior to submission of materials for
review by specialist review panels.
4
-------
EMAP Peer Review Procedures (first Edltwn)
Scoping Meeting
. Fonnulate goals and objectives for peer review
. Define desired output (fonnat, content, audience)
. Identify preliminary list of reviewers
. Set tentative dates and locations
. Prepare draft agenda
T
I
I
Technical
Administrative
. Select final reviewers
. Finalize agenda
. Provide presentation guidance
. list review questions
. Hold dry run
. Facilitate meeting
. Record/organize technical
comments
. Finalize meeting location
. Arrange meeting facilities
. Send correspondence
- First mailing
- Second mailing
I
I
~
Synthesize Comments
~
Prepare Reconciliation
Memorandum
t
Send Follow-Up
Correspondence to Reviewers
Figure 1. Schematic of Level 3 peer review process.
5
October 31, 1990
------- |