Approaches for Expressing and Resolving Differing Scientific Opinions
U.S. Environmental Protection Agency
Scientific Integrity Program
October 8, 2020
Preamble: Scientific products and decisions are strengthened by considering all pertinent evidence and
exploring various plausible explanations of that evidence. Vigorous internal discussion of different points
of view helps to anticipate counter-arguments and alternative positions that could arise during public
comment, peer review, and litigation. This process of challenging and improving ideas helps to guard
against inadequate science and flawed analyses. It also creates a stimulating work environment where
employees can develop professionally. Accordingly, EPA expects and encourages all employees to offer
and welcome differing scientific opinions as a legitimate and necessary part of the scientific process. This
document recommends a progression of approaches that employees and managers can use to encourage
the expression and satisfactory resolution of differing scientific opinions.
1. Introduction
1.1 Purpose: This document aids the implementation of EPA's Scientific Integrity Policy1 by
encouraging the expression of differing scientific opinions and suggesting a progression of
approaches for their resolution. The approaches begin with successive stages of internal
deliberation and may involve the possible use of internal or external peer review. The objective is
a timely, satisfactory resolution within the smallest possible organizational unit.
1.2 Policy: EPA's Scientific Integrity Policy "Mandates the Scientific Integrity Official, with input
from the Deputy Scientific Integrity Officials, to develop a transparent mechanism for Agency
employees to express differing scientific opinions [§IV.A.3]." The Scientific Integrity Policy is
based on the Presidential memorandum on scientific integrity,2 and the Office of Science and
Technology Policy memorandum on scientific integrity. 3 In addition, under EPA's Principles of
Scientific Integrity, 4 EPA employees must "Welcome differing views and opinions on scientific
and technical matters as a legitimate and necessary part of the process to provide the best
possible information to regulatory and policy decision-makers."
1.3 Applicability
The approaches suggested in this document apply to differing opinions regarding scientific data,
interpretations, or conclusions. They do not apply to opinions regarding policy options or
decisions. The distinction between science and policy is fundamental, and these approaches are
focused on professional opinions regarding scientific information, methods, models, analyses,
results, and conclusions. These approaches do not address personal opinions about scientific
issues that are not accompanied by scientific arguments.
1 EPA Scientific Integrity Policy: https://www.epa.gov/sites/production/files/2014-
02/documents/scientific integrity policy 2012.pdf
2 Presidential memorandum on scientific integrity (March 9, 2009): https://www.gp0.gov/fdsys/pkg/FR-2009-03-
ii/pdf/Ec)~S443.pdf
3 OSTP memorandum on scientific integrity (December 17, 2010):
https://obamawhitehouse.archives.gov/adininistration/eop/ostp/library/scientificintegrity
4 EPA's Principles of Scientific Integrity (1999): https: //www.epa.gov/osa/epas-principles-scientific-integrity-fact-
sheet
Page 1 of 15
-------
These approaches apply to all EPA employees, including scientists, managers, and political
appointees. These approaches do not apply to external parties, who have other means to express
their opinions to EPA.
When there is overlap with other applicable rules and guidance, this document is not intended to
preempt other authorities, but instead to work in conjunction with and supplement them. This
document is intended to improve the internal management and operation of EPA. It does not
create any obligation, right or benefit for any member of the public, substantive or procedural,
enforceable by law or in equity by any party against the United States, its departments, agencies,
or entities, its officers, employees or agents, or any other person [Scientific Integrity Policy, §111].
1.4 Other EPA procedures that take precedence
(a) Unaddressed Significant Risks to Public Health or the Environment (Elevation Policy):
Multiple EPA Administrators have affirmed the importance of elevating risks to public health
and the environment so EPA can respond in a timely and effective manner. 5 In the case of a
perceived unaddressed significant risk within the scope of EPA authorities, employees should
promptly notify their management or report it at https://workplace.epa.gov/report-an-
issue.html.
(b) Misconduct: In the case of data, interpretations, or conclusions that constitute misconduct
(fabrication, falsification, or plagiarism)6 beyond a difference of opinion, employees should
promptly notify their management or the Office of Inspector General, or in the case of
plagiarism, EPA's Scientific Integrity Official. 7
1.5 Examples of similar procedures at other federal agencies
Several federal agencies have programs to encourage the expression of differing opinions
promptly and without fear of reprisal. Some programs are broader in scope, for example, covering
differing opinions on policies and decisions or differing opinions from contractors.
(a) The U.S. Nuclear Regulatory Commission supports a Differing Professional Opinion program
"an employee or contractor can use when he or she has a conscientious expression of a
judgment or position that differs from an established staff view, disagrees with a management
decision or policy position, or takes issue with an established agency practice involving
technical, legal, or policy issues (including administrative or corporate support issues)."8
5 Elevation Policy: http://r4intra.epa.gov/orc/guidance/policy elevation of critical public health
issues.pdf:https://www.epa.gov/aboutepa/andrew-wheeler-messages-epa-empk>yees and
https://www.epa.gov/sites/pr0ducti0n/files/2018-10/d0cuments/wheeler-message0nelevati0n-0ct0ber04.2018.pdf
6 Research misconduct is defined as "fabrication, falsification, or plagiarism in proposing, performing, or reviewing
research, or in reporting research results" and does not include honest error or differences of opinion [65 FR
76262]. See the Federal Policy on Research Misconduct: https: //www.gp0.gov/fdsys/pkg/FR-2000-12-06/pdf/00-
308.s2.pdf
7 Policy and Procedures for Addressing Research Misconduct (EPA Order 3120.5), section 9(A): htipsr / /
www.epa.gov/sites/production/files/2014-04/documents/epapolicy.pdf
8 https://www.iirc.gov/readiiig-nn/doc-coHectioiis/inaiiageineiTt-directives/volmTies/vol-io.lTtiTil (approved 11 Aug
2015)
Page 2 of 15
-------
(b) The U.S. Department of Energy has a Differing Professional Opinion process "for employees
to raise technical concerns related to environment, safety, and health which cannot be
resolved using routine processes." 9
(c) The U.S. Food and Drug Administration's Center for Drug Evaluation and Research provides
a procedure for staff members to express Differing Professional Opinions concerning
"regulatory actions or policy decisions with significant public health impact in instances when
the normal procedures for resolving internal disputes are not sufficient." The procedure
provides short timeframes for review by qualified staff not directly involved in the decision so
differing opinions can be resolved expeditiously.10
(d) The U.S. Food and Drug Administration's Center for Devices and Radiological Health
provides a policy "for resolving internal differences of professional opinion and provides an
approach for documenting scientific, clinical, and regulatory findings, perspectives, and
opinions."11
1.6 Freedom from retaliation
EPA's Scientific Integrity Policy states that the policy "extends whistleblower protections to all
EPA employees who uncover or report allegations of scientific and research misconduct, or who
express a differing scientific opinion, from retaliation or other punitive actions [§IV.A.3]."12
Differing scientific opinions should be welcomed and encouraged as a legitimate part of the
scientific process. Retaliation, even the threat of retaliation, against employees who express
differing scientific opinions shall not be tolerated and may result in disciplinary action, '-s An
employee who experiences or fears retaliation based on expressing a differing scientific opinion
may contact EPA's Scientific Integrity Official for advice and assistance. *4
In addition, for employees who feel they cannot openly express a differing scientific opinion, a
confidential option is available (section 6).
1.7 Definitions
Deliberative documents: documents prepared to assist in arriving at a decision and reflecting
preliminary or candid internal views or advice. They are internal, pre-decisional, and have
deliberative content. They may express personal ideas, staff opinions, recommendations or
advice, options papers, issue papers, management briefing documents, edits or comments on
draft documents, draft decision and supporting documents.
9 https: //www.directives.doe.gov/directives-documents/400-series/0442.2-BOrder-chg1-pgchg (updated 5 Oct
2016)
10 https: //www.fda.gov/inedia/7i6i4-/dowiiload (effective 16 Sept 2010)
11 https://www.fda.gov/abont-fda/cdrh-oinbndsiTiaii/center-dewces-aiid-rad.iological-liealtli-cdiii-staiidard-
QBSatijigja-o^^ (updated 4 Sept 2012)
12 Information on whistleblower protection is available at: https://www.epa.gov/office-inspeetor-
general/whistleblower-protection
'•'* Information on protection from retaliation is available at: https: //www.eeoc.gov/laws/types/retaliation.cfm
'4 Other resources include EPA's Whistleblower Protection Coordinator (https://www.epa.gov/offiee-inspector-
general/whistleblower-protection#wbp ombudsman"), the Office of Inspector General Hotline
("https: //www.cpa.gOv/officc-iiispcetor-gciicral/cpa-oig-liotliiic#filc now"), and the Office of Special Counsel
(https: / / osc. gov/) ¦
Administrative Records Guidance: https://www3.epa.gov/ogc/adminrecordsguidaneeo9-oo-11.pdf (Sept 2011)
Page 3 of 15
-------
Differing scientific opinion: a differing opinion of an EPA employee who is substantively engaged
in the science that may inform an EPA decision. It generally contrasts with a prevailing staff
opinion included in a scientific product under development. The differing opinion must
concern scientific data, interpretations, or conclusions, not policy options or decisions. These
approaches do not address personal opinions about scientific issues that are not accompanied
by scientific arguments, are not part of a scientific product, and are not made in the context of
an EPA decision.
Policy: A high-level statement of principles that defines a course of action for a specific purpose
and establishes broad requirements that govern EPA's decision making16.
Prevailing staff opinion: the opinion of the team developing a scientific product. As with differing
scientific opinions, prevailing staff opinions are statements about scientific data,
interpretations, or conclusions, not to include policy options or decisions.
Science, scientist: This document adopts the expansive view of science articulated in the Scientific
Integrity Policy [§II] as including "the full spectrum of scientific endeavors, e.g., basic science,
applied science, engineering, technology, economics, social sciences, and statistics." Similarly,
a scientist is "anyone who collects, generates, uses, or evaluates scientific data, analyses, or
products."
Scientific information: "factual inputs, data, models, analyses, technical information, or scientific
assessments related to such disciplines as the behavioral and social sciences, public health
and medical sciences, life and earth sciences, engineering, or physical sciences. This includes
any communication or representation of knowledge such as facts or data, in any medium or
form, including textual, numerical, graphic, cartographic, narrative, or audiovisual forms.
This definition includes information that an agency disseminates from a web page but does
not include the provision of hyperlinks on a web page to information that others disseminate.
This definition excludes opinions, where the agency's presentation makes clear that an
individual's opinion, rather than a statement of fact or of the agency's findings and
conclusions, is being offered." *7
Scientific Integrity Official; Deputy Scientific Integrity Official: EPA's Scientific Integrity Official
champions scientific integrity throughout EPA and chairs a committee of Deputy Scientific
Integrity Officials representing each program office and region to provide oversight for the
implementation of the Scientific Integrity Policy, to act as liaison for their respective offices,
and to be available to address questions or concerns regarding scientific integrity at EPA
[Scientific Integrity Policy §11].
Scientific product: products that contain scientific information. These include risk assessments,
technical studies and guidance, analytic methods, scientific database designs, technical
models, technical protocols, statistical surveys/studies, technical background materials,
technical guidance, research plans, and research strategies. They can support a research
agenda, regulatory program, policy position, or other EPA position or action.18
16 https://ofmpnb.epa.gov/sor internet/registiy/termreg/searchandretrieve/termsandacronyms/search.do.
17 Final Information Quality Bulletin for Peer Review: https: //georgewbush-whitehouse.archives.gov/omb/
memoranda/fy2005/m05-03.htinl
18 See EPA's Peer Review Handbook (https://www.epa.gov/sites/production/files/2016-
03/documents/epa peer review handbook 4th edition.pdf)
Page 4 of 15
-------
"Substantively engaged in the science": having contributed scientific expertise in an official
capacity to the development or review of a scientific product, beyond presence at meetings or
on mailing lists.
1.8 Effective date: This document is effective when released. After a pilot period of six months, the
Scientific Integrity Program may revise the document based on experience and comments from
EPA employees.
2. Overview
This document recommends a progression of
approaches that employees and managers can use to
encourage the expression and satisfactory resolution of
differing scientific opinions. Recommended steps
begin with discussions (Box l) within the team
developing a scientific product and can proceed to the
engagement of additional subject-matter experts,
managers, then to internal or external peer review of
the scientific merit of the differing opinions,
documenting the differing opinions for the policy
makers (figure l). This progression is in accord with
EPA's Scientific Integrity Policy, which envisages the
use of internal deliberations and scientific peer review
(Box 2.).
BOX i.
Hallmarks of a constructive, free and open
discussion are:
— Being respectful of others
and their views
— Listening to others without
interrupting
— Including persons with
different points of view,
including potential
dissenters
— Involving everyone in the
discussion
— Linking to and building on
what has already been said
Discuss within the team
1
Consult subject-matter
experts
I
Involve management,
initially first-level
I
Seek advice from the
DScIO or ScIO
The Deputy Scientific Integrity Officials (DScIO) and Scientific Integrity Official
(ScIO) are available to advise and facilitate the process, not to decide the science.
Figure l. Paths towards resolution of a differing scientific opinion
The ability to express differing scientific opinions exists in the context of a decision-making agency,
where scientific information supports decision making in a broad policy context that includes many
considerations other than science.
If no resolution:
Document the
differing opinion
If no
peer
> /
Internal review
- or -
Ad-hoc external review
- or -
External review as part of
the larger scientific
product
Peer-review
report
Page 5 of 15
-------
Any employee within the team developing a
scientific product may initiate discussion of a
differing scientific opinion. If the scientific product
or a subsequent decision is time-sensitive or if
other circumstances warrant, these approaches
may be expedited or otherwise adapted, ideally in
consultation with management. The employee
initiating a differing scientific opinion may
voluntarily terminate the process at any time.
A satisfactory resolution does not necessarily entail
choosing one of the opinions, especially when there
are multiple plausible methods, models, analyses,
or conclusions. Many scientific products do not
require that there be a single opinion, and differing
scientific options are often supported by the
uncertainty or variability in the underlying data. It
may be appropriate to present the pros and cons of
several alternatives in a scientific product or
decision document. It also may be appropriate to
recommend additional technical work to better
inform or resolve the differing opinions, in which
case management should be consulted to work through the implications for timing.
In the interests of fostering the expression of differing scientific opinions, policy makers are
encouraged to communicate final decisions and their basis back to the team and anyone who has
formally expressed a differing scientific opinion on that particular matter.
3. Resolving differing scientific opinions: internal deliberations among employees
EPA's Scientific Integrity Policy expresses the expectation that differing scientific opinions will be
resolved during internal deliberations. To the extent possible, it is desirable to resolve differing
scientific opinions through internal deliberations. These deliberations will likely begin among
employees on a team developing a scientific product and can progress to include (1) additional
scientists who are subject-matter experts, (2) managers responsible for the scientific product, (3) the
organization's Deputy Scientific Integrity Official, and (4) EPA's Scientific Integrity Official (figure 1).
3.1 Discussions within a team developing a scientific product
All employees should welcome differing scientific opinions as a legitimate and necessary part of
the scientific process. An employee (or group of employees) with a differing opinion should make
reasonable attempts to engage in scientific discussions with their colleagues and managers. The
ability to discuss differing opinions constructively and respectfully is a hallmark of a healthy
scientific organization.
The objective of these discussions is to resolve the differing opinions, or if that cannot be
achieved, to agree on how to present the alternative opinions in the scientific product so that the
policy makers have the necessary information to make a well-informed policy decision. Possible
forms of presentation may include:
BOX 2.
"When an Agency employee substantively
engaged in the science informing an Agency
policy decision disagrees with the scientific data,
scientific interpretations, or scientific conclusions
that will be relied upon for said Agency decision,
the employee is encouraged to express that
opinion, complete with rationale, preferably in
writing. It is expected that any differing scientific
opinions will be resolved during internal
deliberations and if not, will be addressed during
scientific peer review. The report from the peer
review panel will be made available for the policy
makers' consideration. When no peer review
occurs, differing scientific opinions will be
reflected in the Agency's deliberative documents
for the policy makers' consideration."
- Scientific Integrity Policy §IV.A.s
Page 6 of 15
-------
— Presenting each alternative opinion with its underlying rationale for later resolution
through peer review and possibly by senior EPA officials.
— Developing a new opinion that synthesizes elements from the original opinions.
— Agreeing that one opinion is preferable. In this case, alternative opinions with scientific
merit also may be discussed as having been considered.
In each case, the discussions ensuing from the differing scientific opinion should lead to a
stronger position than before or should result in stronger, more nuanced supporting arguments.
3.2 Discussions augmented by additional subject-matter experts
A team developing a scientific product may invite additional scientists who are subject-matter
experts to enrich the discussion in the hope of resolving the differing opinions. Project teams are
often small, and this can make it difficult to have in-depth scientific discussions when teams lack
a breadth of perspectives and include only one or two people in an area of expertise.
The careful addition of subject-matter experts can promote confidence in the results of the
augmented discussion. The ideal candidate would be a nationally recognized expert who is widely
respected and trusted to be impartial. If the team cannot agree on impartial experts, another
approach is for each group of proponents to invite one or two additional experts, then ask these
experts to bring in others if they think it would be helpful.
An organization might consider creating standing discipline-specific groups of experts or
communities-of-practice who can convene to discuss scientific issues in their area of expertise.
Where such groups exist, a differing scientific opinion could be referred to a readily available
group of subject-matter experts without the need to choose ad-hoc groups.
3.3 Involving office management responsible for the project
If the employees developing a scientific product are unable to resolve a differing scientific opinion
through internal deliberations, the first-level manager responsible for the scientific product may
intervene or be invited to help. If the first-level manager is already a proponent of one of the
differing opinions, then it may be appropriate to ask another manager or the organization's
Deputy Scientific Integrity Official to suggest additional approaches for resolution or additional
subject-matter experts. The manager also might choose to become personally involved as a
moderator between the employees with the differing opinions.
If another manager in the organization has substantive knowledge or experience pertinent to the
issue or has the means of resolving this disagreement, the manager responsible for the scientific
product may ask him or her to assume this role.
The responsible manager also might decide to seek advice from the next level of management.
If there is an impasse, the responsible manager would consult with the policy maker about
whether to proceed to peer review (consistent with EPA's Peer Review Handbook) or to
document the differing scientific opinion as described in section 4.1 and include it in the
deliberative documents provided to policy makers. Although the Scientific Integrity Policy
expresses the expectation that differing scientific opinions not resolved during internal
deliberations will be addressed during scientific peer review, this document does not establish a
right to peer review.
Page 7 of 15
-------
3.4 Seeking advice from the organization's Deputy Scientific Integrity Official
Each EPA program office and region has a Deputy
Scientific Integrity Official who is available to assist in
situations where there are differing scientific opinions.
The Deputy may suggest that the employees resume
their discussions, may suggest additional approaches
for resolution or additional subject-matter experts, or
may choose to become personally involved as a
moderator between the employees with the differing
opinions. Alternatively, the Deputy may consider engaging the services of a professional
mediator.
If there is an impasse, the Deputy Scientific Integrity Official would consult with the manager
responsible for the scientific product to determine, based on the nature of the issue and the
timing of the decision, whether to proceed to peer review or to document the differing scientific
opinion as described in section 4.1 and include it in the deliberative documents provided to policy
makers.
3.5 Seeking advice from EPA's Scientific Integrity Official
Alternatively, EPA's Scientific Integrity Official is available to provide advice and assistance.
Employees who feel that they cannot pursue a differing scientific opinion within their
organization or with a Deputy Scientific Integrity Official may invite EPA's Scientific Integrity
Official to assist in the activities described in section 3.4.
3.6 Discussions initiated from outside a team
Although differing scientific opinions ordinarily would arise within a team developing a scientific
product, there may be other subject-matter experts at EPA whose advice would improve the
scientific quality of the product. Discussions initiated from outside the team occur when an
internal expert learns of a scientific product under development, believes an important issue is
not being adequately addressed, and wants to give advice to the team.
The most collegial approach would be for the internal expert to call the leader of the team
developing the scientific product to discuss the team's approach and the expert's perspectives.
Together the expert and team leader would determine whether there is mutual interest in inviting
the expert to team discussions.
A more formal approach would be for the internal expert to document his or her perspective as
described in section 4.1, then to transmit it to the leader of the team developing the scientific
product, bringing in scientific integrity officials and managers as needed. Consideration should be
given to the potential for the expert's contribution to improve the scientific quality of the product,
as well as schedule and resource constraints. In addition, the team leader or responsible manager
may invite the expert to a team meeting. This discussion may expand as necessary to include
additional subject-matter experts (section 3.2) or managers (section 3.3). The Deputy Scientific
Integrity Official or EPA's Scientific Integrity Official may offer advice and assistance if necessary
(sections 3-4-3-5)-
When acting in this capacity, Deputy
Scientific Integrity Officials serve to
uphold scientific integrity and not to
advocate a scientific position. Their
role is to advise and facilitate the
process, not to decide the science.
Page 8 of 15
-------
In these ways, an internal expert not engaged in the development of a scientific product (though
engaged in the science pertinent to the product) may initiate a differing scientific opinion. Such
experts should have scientific publications, job experience, or specialized training pertinent to the
specific issue, for example, publications on modeling the environmental fate of particulate matter
or job experience assessing reproductive toxicity. This fulfills the mandate that EPA employees
must welcome differing views and opinions and address them, with due consideration to
management concerns about scheduling and resources.
4. Addressing differing scientific opinions through peer review
Although the Scientific Integrity Policy expresses the expectation that differing scientific opinions will
be resolved during internal deliberations, it recognizes that this might not happen and that differing
opinions maybe addressed during scientific peer review1^.
This document does not establish a right to peer review. The managers responsible for the scientific
product have key roles in decisions about peer review. If there is no peer review, the differing
scientific opinions are documented and included in the deliberative documents provided to policy
makers (section 5).
The peer review process, managed in accordance with the program's peer-review procedures, should
engender trust so all parties can respect the outcome. The elements of peer review include:
— Selecting a process for peer review
— Documenting the differing scientific opinions so they can be reviewed
— Developing an unbiased charge to assist the peer reviewers in their task
— Selecting scientifically qualified peer reviewers, with due consideration of impartiality and
conflicting interests, and protecting the panel from interference
— Writing the peer review report
The peer reviewers write joint or individual reports that evaluate the scientific merit of the differing
opinion and respond to the charge. The manager responsible for the scientific product receives the
report and transmits it to everyone involved in the internal deliberations and to the organization's
Deputy Scientific Integrity Official. The manager also makes the report available for consideration by
the policy makers.
4.1 Documenting a differing scientific opinion
If there will be a peer review of a differing scientific opinion, the employee(s) with the differing
opinion should prepare a summary position, similar in coverage to the prevailing staff opinion.
The summary should be factual, impartial, clear, concise, and not burdensome to develop or
review. It may be read by peer reviewers or policy makers with various backgrounds. There is not
a single format appropriate for all scientific disciplines and issues. One best practice might be to
emulate scientific journal editors who have developed a common practice that works for
expressing opinions across many fields. Readers of a published manuscript can express their
opinions through a short letter-to-the-editor, commonly with a limit of up to 500 words, plus a
table or figure if needed, and references
"» See EPA's Peer Review Handbook (https://www.epa.gov/sites/production/files/2016-
03/documents/epa peer review handbook 4th edition.pdf)
Page 9 of 15
-------
One option would be for employees to adapt the practice of a prominent journal in their field and
document their differing opinions in a similarly concise manner. Alternatively, they could agree to
adopt an ad-hoc approach tailored to the specific issue.
4.2 Developing a charge for the peer review
All sides guided by the manager responsible for the scientific product or the organization's Deputy
Scientific Integrity Official work together to formulate a charge to guide the peer review. The
charge should not steer the peer reviewers towards a conclusion.
4.3 Peer review as part of a larger scientific product
If the differing scientific opinion is central to a larger scientific product that will undergo peer
review, an option is to present the differing opinions to the peer reviewers of the scientific
product. This option is possible if several peer reviewers have expertise pertinent to the issue. For
example, in a toxicological review of adverse health effects associated with exposure to a chemical
substance, a differing scientific opinion on whether a toxic effect has a threshold is central to the
assessment and should be within the expertise of the peer reviewers. On the other hand, if the
differing scientific opinion concerns a highly specialized modeling issue, there may not be
adequate expertise on the peer review panel to address this issue. In the latter case, the panel
could be augmented with the required expertise, or an ad-hoc peer review limited to the differing
scientific opinion could be conducted prior to the peer review of the full scientific product (section
4-4)-
There are several options for presenting a differing scientific opinion for peer review
accompanying a larger scientific product. For example, the scientific product could be written to
present the differing opinions consecutively as options under consideration. Alternatively, the
scientific product could include a prevailing staff opinion, with the differing opinions presented in
an appendix or separate document. Either way, the charge would ask peer reviewers to evaluate or
compare the scientific merits of the differing opinions.
4.4 External peer review
An option for peer review limited to the differing scientific opinion is to use an ad-hoc panel of
external subject-matter experts. This might be considered when few employees have the required
expertise or when it would be useful to solicit the opinions of external experts.
Aside from the time and expense of a procurement process, external peer review includes the
same elements as internal peer review: developing a charge, selecting scientifically qualified peer
reviewers, and writing a peer review report.
4.5 Internal peer review
Another option for peer review limited to the differing scientific opinion is to use an ad-hoc panel
of internal subject-matter experts. Internal peer review is an extension of the reviews employees
regularly perform for each other in scientific organizations. This might be considered as a means
of achieving a strong and defensible resolution of the differing opinions before public release.
Page 10 of 15
-------
Because differing scientific opinions by their nature involve scientific judgment about contentious
or intractable issues, an impartial panel is essential. The ideal would be for the proponents of the
differing opinions to mutually agree on the appropriate internal experts.
Alternatively, peer reviewers could be drawn from an established discipline-specific expertise
group (section 3.2), if the organization has established one and if it is considered by each side to
be impartial. An advantage of using a standing group of subject-matter experts is that it skips the
step of identifying and selecting available subject-matter experts.
5. Presenting differing scientific opinions to policy makers
The Scientific Integrity Policy specifies that reports from peer reviews of differing scientific opinions
will be made available for consideration by the policy makers, who are generally office directors or
assistant administrators. It further specifies that if no peer review occurs, then differing scientific
opinions will be included in the deliberative documents considered by the policy makers. These
provisions ensure that policy makers are aware of all differing scientific opinions not resolved through
internal scientist-to-scientist deliberations.
Some options for documenting differing scientific opinions appear in section 4.1.
5.1 Advance notice for policy makers
When it appears that internal deliberations will not resolve a differing scientific opinion, it is
important to notify the policy makers. This is particularly true for issues that are important,
contentious, precedent-setting, or highly visible. Such notification alerts the policy makers to a
potential delay in completing the scientific product while the issue is addressed through peer
review or other means.
5.2 Providing a peer review report to policy makers
The Scientific Integrity Policy specifies that reports from peer reviews of differing scientific
opinions be made available for consideration by the policy makers. This is to ensure that policy
makers are aware of the scientific merit of all differing opinions not resolved through internal
scientist-to-scientist deliberations.
5.3 Presenting a differing scientific opinion to policy makers
If a differing scientific opinion is not resolved through internal deliberations or addressed
through peer review, it should be reflected in the deliberative documents considered by the policy
makers. This is to ensure that policy makers are aware of the differing opinions. If all issues were
neither resolved nor peer reviewed, policy makers may need to consider seeking additional
advice. Options include consulting their organization's science advisors or EPA's Science Advisor.
Page 11 of 15
-------
6. Option for confidential expression
In some cases, an employee may feel it necessary to seek confidentiality when expressing alternative
scientific views. In these situations, the individual may contact EPA's Scientific Integrity Official for
advice and assistance.
6.1 A proxy selected by the Scientific Integrity Official will meet with the employee to become familiar
with the issue and, if necessary, may act in lieu of the employee in subsequent internal
deliberations (section 3) or in preparing for peer review (section 4).
6.2 To the extent possible and as allowed by law, knowledge about the identity of the employee
should be limited to those who need to know. Confidentiality cannot be guaranteed, however,
especially in small teams. For concerns about confidentiality and freedom from retaliation, EPA's
Scientific Integrity Official may be consulted for advice and assistance.
6.3 While confidential differing opinions will be accepted, anonymous differing opinions will not be
accepted. An anonymous opinion could not be verified to meet the criteria for submitting a
differing opinion under this document.
7. Appeal and Redress20
An employee who believes these approaches are not being applied fairly may contact EPA's Scientific
Integrity Official for advice and assistance.
7.1 The findings of a properly conducted peer review generally cannot be appealed. This is in
accordance with the Scientific Integrity Policy, where differing scientific opinions "will be resolved
during internal deliberations and if not, will be addressed during scientific peer review [§IV.A.3]."
7.2 An employee who believes these approaches are not being followed may seek redress from EPA's
Scientific Integrity Official.
8. Recordkeeping
8.1 Differing scientific opinions addressed through peer review require the peer reviewers to create a
peer review report that should be made available for consideration by the policy makers. In
addition, differing scientific opinions presented to policy makers without being addressed
through peer review should be reflected in the deliberative documents considered by the policy
makers [Scientific Integrity Policy §IV.A.3].
8.2 The organization's Deputy Scientific Integrity Official should transmit a summary of each DSO
including date, office, subject area, and resolution to EPA's Scientific Integrity Official for use in
evaluating the implementation of these approaches and in preparing the annual report on
scientific integrity.
8.2 Free and open discussion should be frequent and routine in a healthy scientific organization. If
the participants choose to document the outcome of the discussion for future reference, this need
not be transmitted to EPA's Scientific Integrity Official.
20 An appeal is a request for reconsideration when the proper procedures were followed. Redress is the term used
when the proper procedures were not followed.
Page 12 of 15
-------
g. Examples
The following examples are illustrative but not exhaustive. Similar situations could be approached and
resolved differently depending on the specifics of each case.
(a) Situation: A team of employees is completing a draft document for release for public comment
and peer review. A team member believes that the data are inadequate and too uncertain for
reaching a conclusion.
Possible approach: This is a legitimate topic for a differing scientific opinion, which need not be
framed as Conclusion A versus Conclusion B. Conclusion A versus No Conclusion is equally valid.
By raising the opinion that the data are inadequate for reaching any conclusion, the subsequent
internal deliberations should force the team to reexamine their conclusion, and sharpen the
supporting arguments. In this way, differing scientific opinions can improve the quality of the
scientific product.
(b) Situation: An office has started to update a scientific product that will go to external peer review.
One team member, the only modeler on the project, believes that there are flaws in a model that is
central to the analysis but that were not discovered when the prior version was approved. The
modeler would like to develop a better model, but the project manager rejected this proposed
activity, believing the prior model to be adequate and new model development to be time-
consuming. The modeler raises the modeling approach as a differing scientific opinion.
Possible approach: As there are no other modelers on the project, the first step could be to seek
the views of other EPA modelers. If this does not resolve the issue, the next step could be an
expedited peer review by an internal panel or by an external panel of qualified modelers. If the
timeline becomes tight, the issue could be documented and presented to the peer review panel of
the larger scientific product or to the policy maker.
(c) Situation: Five years ago, the Office of Research and Development developed an assessment of a
chemical substance and, two years later, a headquarters program office used that assessment to
set a national standard. A regional office is applying the national standard to evaluate chemical
risks at a local site in the region. The regional employee responsible for the site assessment
disagrees with the scientific conclusions of the ORD assessment and is considering initiating a
differing scientific opinion.
Possible approach: The ORD assessment is not an appropriate topic for a differing scientific
opinion, as the science in the ORD assessment is not what is under development or under review.
The regional employee may, however, inform regional managers of the disagreement with the
ORD assessment so that managers in the regional office, the headquarters program office, and
ORD can jointly determine how to proceed.
(d) Situation: A team of employees has developed a new scientific product. Before it is released for
external peer review, a senior manager who also is an expert in the subject matter changes a key
scientific conclusion. The team members protest and ask that their previous scientific opinion be
evaluated by the approaches in this document. The senior manager warns that this borders on
insubordination.
Possible approach: A differing scientific opinion is never insubordination, and the team members
in this case are expressing a scientific opinion and not refusing a lawful order. This is a situation
where the manager responsible for the scientific product would quickly become involved. If the
Page 13 of 15
-------
team members believe the manager is not effective as a moderator, then they should invite the
organization's Deputy Scientific Integrity Official to become involved. One outcome would be for
the draft scientific product to include both conclusions as alternatives, with scientific rationales
and a charge to ask the peer reviewers to evaluate the scientific merit of each conclusion.
(e) Situation: An external peer review panel reviewed a draft scientific product, and the staff believes
a key peer review recommendation reflects a misunderstanding of the underlying data. They
would like to reject the peer review recommendation and are considering pursuing this as a
differing scientific opinion.
Possible approach: This should not be considered a differing scientific opinion, as EPA is not
required to accept peer review recommendations with which it disagrees. The staff should
carefully consider the recommendation and document the rationale for its decision. If the staff is
split on whether to accept or reject the recommendation, the approaches to internal deliberations
in section 3 might prove helpful.
(f) Situation: An external peer review panel reviewed a draft scientific product, and a key peer review
recommendation is deliberately ignored when the final product is developed. A concerned
employee protests and asks that the final product reflect the peer review recommendation.
Possible approach: Again, this should not be considered a differing scientific opinion, as EPA is
not required to accept peer review recommendations (although all recommendations should be
considered, not ignored). If the staff is split on whether to accept or reject the recommendation,
the approaches to internal deliberations in section 3 might prove helpful. It would be up to the
managers to decide who else needs to be involved.
(g) Situation: A team of employees has completed a peer reviewed assessment that showed the health
hazards of a chemical are greater than previously believed. The program officials considered the
new assessment and decided not to develop a stricter standard for allowable concentrations. The
employees who developed the assessment believe a stricter standard is warranted and file a
differing scientific opinion.
Possible approach: Differing scientific opinions apply only to scientific data, interpretations, or
conclusions, not to policy options or decisions. Scientific integrity is concerned with the accurate
representation of science to the policy makers, not with their policy decisions. A differing
scientific opinion cannot be used to challenge subsequent policy decisions.
(h) Situation: A scientist and supervisor are coauthors of a manuscript being readied for submission
to a scientific journal. They are having difficulty reaching agreement on the conclusions.
Possible approach: Only some of the approaches in this document apply to journal articles. For
example, while products disseminated by EPA are subject to final approval by a policy maker,
journal articles are written by a team of coauthors with peer review managed by the journal
editor. Many journal editors require all coauthors of a manuscript to agree with the final version
and be accountable for all its aspects. Thus, coauthors cannot be forced to adopt a conclusion that
they cannot accept, including EPA's position, if there is one. (The manuscript should carry the
disclaimer about not necessarily representing EPA's views and policies.) This highlights the
importance of planning journal manuscripts and choosing coauthors. If the coauthors reach an
impasse, they may draw on the approaches for internal deliberation in section 3 to seek advice
from other subject-matter experts or from management, and such contributions should be
recognized with an acknowledgment in the published manuscript.
Page 14 of 15
-------
Contents
1. Introduction l
1.1 Purpose 1
1.2 Authority l
1.3 Applicability l
1.4 Other EPA procedures that take precedence 2
1.5 Examples of similar procedures at other federal agencies 2
1.6 Freedom from retaliation 3
1.7 Definitions 3
1.8 Effective date 4
2. Overview 5
3. Resolving differing scientific opinions: internal deliberations among employees 6
3.1 Discussions within a team developing a scientific product 6
3.2 Discussions augmented by additional subject-matter experts 7
3.3 Involving office management responsible for the project 7
3.4 Seeking advice from the organization's Deputy Scientific Integrity Official 8
3.5 Seeking advice from EPA's Scientific Integrity Official 8
3.6 Discussions initiated from outside a team 8
4. Addressing differing scientific opinions through peer review 9
4.1 Documenting a differing scientific opinion 9
4.2 Developing a charge for the peer review 10
4.3 Peer review as part of a larger scientific product 10
4.4 Ad-hoc external peer review 10
4.5 Internal peer review 10
5. Presenting differing scientific opinions to policy makers 11
5.1 Advance notice for policy makers 11
5.2 Providing a peer review report to policy makers 11
5.3 Presenting a differing scientific opinion to policy makers 11
6. Option for confidential expression 12
7. Appeal and Redress 12
8. Recordkeeping 12
9. Examples 13
Page 15 of 15
------- |