OFFICE OF INSPECTOR GENERAL
Special Report
Review of Conflict of Interest Allegations
Pertaining to the Peer Review of EPA's Draft
Report, "Exposure and Human Health
Evaluation of Airborne Pollution from the
World Trade Center Disaster"
Report No. 2005-S-00003
November 4, 2004
-------
Report Contributors: Rick Beusse
James Hatfield
Chris Dunlap
Steve Schanamann
Eric Hanger
Abbreviations:
COI Conflict of Interest
EPA Environmental Protection Agency
FAR Federal Acquisition Regulation
GAO Government Accountability Office (formerly General Accounting Office)
NCEA National Center for Environmental Assessment
NCP National Contingency Plan
OIG Office of Inspector General
SAB Science Advisory Board
WTC World Trade Center
Cover photo: EPA map of ambient air monitoring locations in lower Manhattan; developed
with EPA's EnviroMapper. Source: http://www.epa.gov/wtc/em/
-------
'"*'"< J>B0t<-C>
U.S. Environmental Protection Agency
Office of Inspector General
At a Glance
2005-S-00003
November 4, 2004
Why W« DW THts
We iwrforajed t&U
(a.) ditemdtte the
thst ttees p«er tevww
conflicts
b«y« JIB paired their jttdgments
is ifieir peer review of EPA's
disf t assessment of the health
oftSw World Trsd* Ceater
(WTC) towers, aad riorto BJTOwci>*«Wfcnt of tfc*
j)eeT review, regarding health
hazards AttHs WTC **
someJimes be addiessed is by
using abalaticed panel.
contact our Qfftwof
CongrassJonal antf Pobltc
Liaison at (202) 566-2391.
To view a» full report,
cftck on 4*9 following link;
Review of Conflict of Interest Allegations Pertaining to
the Peer Review of EPA's Draft Report, "Exposure and
Human Health Evaluation of Airborne Pollution from the
World Trade Center Disaster"
What We Found
The seven-member peer review panel selected to review EPA's draft report
included one panelist with an extensive history ofproviding expert testimony and
similar services for defendants in asbestos lawsuits, and two panelists who had
made prior public statements regarding the safety of the air around the WTC site,
These circumstances provided a basis for the perception that one panelist had a
potential conflict of interest and tw o panelists had potential biases that would
prevent them from providing impartial input. We found that EPA's Contractor did
not inquire whether the three panelists had received funding from industry or had
publically expressed viewpoints on the issues to be reviewed. However, our
examination ofthe peer review record did not find that the panel's input was biased,
nor that perceived biases and conflicts were so "direct and substantial" that any of
the panelists should have been excluded from the panel.
In our view additional measures should have been taken during the peer review
selection process to disclose the information about panel members upon which the
allegations of conflicts of interest or bias were later made. Specifically, had EPA's
Contractor inquired about industry funding, learned whether panelists had expressed
opinions publically about the peer review issues, and taken other actions to identify
the bases for these perceived conflicts and biases prior to the selection ofthe panel,
EPA would have been better able to consider the need for additional actions to
balance the panel or otherwise resolve these perceived, potential conflicts and biases
prior to conducting the peer review.
What We Recommend
We made a number of recommendations to better ensure that guidance in EPA's
Peer Review Handbook will be fully followed, including that EPA provide:
• better oversight of peer review contracts to ensure that potential panelists are
asked about industry financing and their relationship with clients; and
• supplemental guidance and training of peer review leaders regarding the types of
information they may need to obtain about potential panelists' opinions and
viewpoints when they assess whether panels are independent and balanced.
EPA agreed with our conclusions and recommendations and has either taken
actions, or set milestones for completing actions, to address our concerns.
-------
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
!>. WASHINGTON, D.C. 20460
f .9-1
\ ••-• '•'' /
'*** •i»r,
-------
If you or your staff have any questions regarding this report, please contact me at (202) 566-
0827 or Rick Beusse, Director for Program Evaluation - Air Issues, at (919) 541-5747.
Attachment
cc: Cheryl Varkalis, Audit Liaison, Office of Research and Development
Matt Lorber, National Center for Environmental Assessment/ORD
Elizabeth Grossman, Office of Program Evaluation/OIG
-2-
-------
Table of Contents
At a Glance
1 Introduction 1
Purpose 1
Background 1
EPA's Guidance for Conducting Peer Reviews 2
Scope and Methodology 3
Prior OIG Report 4
Results in Brief 4
Recommendations 6
Agency Comments and OIG Evaluation 6
2 Results of Review of the Six Allegations 7
Question 1: Did the NCR Require Public Participation in Peer Review
Panel Selection? 7
Question 2: Was NCEA Obligated by Procurement Policy Notice 95-04
to Evaluate the Sensitivity of the Work Being Performed? 8
Question 3: Was NCEA Obligated to Ensure that Its Peer Review Contractor
Required Potential Panelists to Disclose All Substantial Public
Statements and All Business Relationships? 9
Question 4: Did the Asbestos Expert Have a Conflct of Interest? 13
Question 5: Did Two Panelists Have Biases Based on Their Prior Stated
Conclusions Regarding WTC Health Hazards? 17
Question 6: Did EPA's Contractor Adequately Consider Balance in
Selecting the Panel? 21
Conclusions 22
Recommendation 23
Agency Comments and OIG Evaluation 23
3 Observations on Panel Formation and Balance 24
Handbook Guidance on Panel Balance 24
Experts' Comments on EPA's Peer Review Handbook 24
Prior OIG Report Discussed Concerns With EPA Efforts
to Assess Peer Reviewers' Independence 25
Increased Federal Attention to Peer Review Panel Balance 26
Conclusions 27
Recommendations 27
Agency Comments and OIG Evaluation 28
-------
Appendices
A Complainant's Allegation Memorandum ..
B Peer Review Panelists and Backgrounds
C Agency Response to Draft Report
D ORD Corrective Action Plan
E NCEA Policy Announcement
F Distribution
30
37
38
42
44
51
-ii-
-------
Chapter 1
Introduction
Purpose
The Office of Inspector General (OIG) conducted a review of six allegations
concerning the peer review that was performed on a draft report, issued by the
Environmental Protection Agency's (EPA) National Center for Environmental
Assessment (NCEA), entitled: "Exposure and Human Health Evaluation of
Airborne Pollution from the World Trade Center Disaster" (Draft Assessment).
Based on the allegations made by the Complainant, our objectives were to answer
the following six questions:
Does the National Contingency Plan require that the public be given an
opportunity to comment on the choice of peer review panelists, or suggest
other experts who could represent the views of the community?
Was NCEA obligated to evaluate the sensitivity of the work being
performed as allegedly required by its guidance for handling conflict of
interest situations that are identified after contract award?
• Was NCEA obligated to ensure that its peer review Contractor required
potential panelists to disclose all substantial public statements they had made
about the World Trade Center (WTC) disaster and all business relationships
with industries that could be effected by NCEA's findings on WTC exposure
hazards?
• Did panelist Dr. Eric Chatfield have a conflict of interest and did his resume
lack information which would have underscored this conflict of interest?
• Did panelists Dr. Alison Geyh and Dr. Patrick Kinney have a bias because
they had publically expressed prejudicial opinions before the peer review
process began that "... there were no hazards from WTC exposures"?
What steps were taken to ensure that the panel was balanced?
Appendix A provides more information on the allegations.
Background
In the aftermath of the collapse of the WTC towers on September 11,2001,
various government entities initiated numerous air monitoring activities to better
understand the ongoing impact of emissions at the disaster site. At the request of
EPA Region 2 in New York City, NCEA officials used the data collected from
-------
these monitoring activities to conduct an inhalation exposure and human health
risk assessment for airborne pollution from the WTC disaster. The issue of
potential health impacts from exposure to airborne pollutants resulting from the
collapse of the towers and the continuing fires at Ground Zero was a highly
sensitive and controversial subject. Accordingly, most government
pronouncements and data regarding this subject were and continue to be highly
scrutinized.
In December 2002, NCEA issued its Draft Assessment for public comment.
NCEA officials first obtained a proposal from the National Academy of Sciences
for the peer review of this Assessment; however, they subsequently decided the
National Academy of Sciences proposal would take too long and cost too much.
As such, they decided to obtain the peer review by issuing a task order to Versar,
Inc. under an existing 5-year contract for peer teview and risk assessment support
(Contract).1 Versar, in turn, obtained the services of seven peer reviewers by
issuing seven Work Assignment Authorizations. Versar held a public peer review
meeting on July 14-15,2003. A summary of the peer review panel's comments
and meeting is available on EPA's public website.
EPA's Guidance For Conducting Peer Reviews
The Agency's guidance concerning peer reviews is contained in EPA's Peer
Review Handbook, Second Edition, December 2000 (Handbook), which was
issued by EPA's Science Policy Council. This guidance applies whether EPA
obtains peer reviewers directly, or contracts for another entity to obtain
appropriate peer reviewers. According to the Handbook, the principal reason for
having a peer review is to ensure the scientific integrity and technical credibility
of a work product which supports a policy or decision, through a complete,
independent, objective, and competent review. The peer review process is
intended to uncover any technical problems or unresolved issues in a preliminary
work product through the use of independent experts. Peer review panelists do
not formulate EPA policy; instead, they provide expert input on technical issues
which underpin EPA conclusions or policies, such as whether a particular
sampling procedure is an acceptable method for obtaining data used by EPA
officials to reach conclusions or formulate policy.
The Handbook states that selecting an independent peer review panel is crucial to
an effective peer review, and therefore, EPA should make every effort during
panel formation to use peer reviewers who do not have any real or perceived bias
or conflicts of interest and who are completely independent.
The Contract, entitled NCEA Support for Peer Review and Risk Assessment Guidelines Activities, is a task
order contract which provides forservices from August 1, 2002 through Juty 31, 2007.
-------
Scope and Methodology
To ensure that we accurately and comprehensively addressed the Complainant's
concerns, we interviewed the Complainant. We subsequently discussed in detail
our understanding of toe Complainant's allegations, and reached agreement that
we had stated the allegations fully and accurately. We requested any additional
documentation in the Complainant's possession which provided support for the
allegations, and we reviewed the documentation subsequently provided that
appeared relevant to the allegations.
To identify EPA's efforts to ensure that an adequate peer review was carried out,
we interviewed NCEA's Work Assignment Manager for the subject peer review
task order and Versar's Senior Project Manager for this order. We requested and
reviewed all documentation concerning the peer review process and screened
these documents for applicability to the allegations. We reviewed those
documents relating to the establishment and selection of the peer review panel
including, but not limited to, the work assignment authorizations for the peer
review services, personal conflict of interest statements, resumes, records of
discussions with panelists, and e-mail correspondence.
We reviewed EPA's Peer Review Handbook to determine Agency guidance for
identifying and addressing potential conflicts of interest and bias issues. We also
reviewed other government related peer review guidance (such as that used by the
Science Advisory Board, National Academy of Sciences, or recommended by the
Office of Management and Budget) to determine how these documents addressed
the issue of balance in peer review panels. To enhance our understanding of the
history and development of the Peer Review Handbook, we interviewed one of
the two principal authors of EPA's Handbook. Similarly, to enhance our
understanding of the peer review balance and panel formation processes and
procedures used by the Science Advisory Board, we interviewed the Science
Advisory Board's Director and an Ethics Official. We also reviewed prior
General Accounting Office (GAO) and EPA OIG reports to identify any
previously reported issues relevant to the questions we addressed in this review.
To assess whether the panel's input to the NCEA Assessment was biased, we
obtained and reviewed 0istened to) the entire audio tape of the two-day peer
review meeting. Further, we had our OIG Certified Industrial Hygienist review
the draft and final Summary Peer Review Report, which included all written
comments from the peer review panelists. In addition, we obtained the responses
of all 7 panelists to 13 questions about their perception of the peer review process,
as well as detailed responses of 3 panelists to additional questions about the
specific allegations directed at them. Also, at the request of panelist Dr. Eric
Chatfield, we met with him and discussed his responses to the specific allegations
made against him. In addition, our Certified Industrial Hygienist interviewed
NCEA's Work Assignment Manager concerning the Manager's efforts to
-------
incorporate the recommendations made by the peer review panel about asbestos in
NCEA's Final Assessment. However, since this work had not been completed at
the time our field woik ended, our Certified Industrial Hygienist was not able to
confirm the extent to which the panel's recommendations about asbestos may or
may not be included in NCEA's Final Assessment. Subsequent to the completion
of our field work, theNCEA Work Assignment Manager indicated that all
changes recommended by the panel were incorporated into the working draft of
the Risk Assessment.
To ensure that we accurately interpreted key criteria regarding these issues, our
OIG Office of Counsel advised us on the applicability of the National
Contingency Plan (NCP), Federal Acquisition Regulation, and Agency guidance
on the peer review process. We performed our review and analysis from October
2003 through June 2004 in accordance with Government Auditing Standards
issued by the Comptroller General of the United States.
Prior OIG Report
EPA's Selection of Peer Reviewers, EPA/OIG, Report No. 1999-P-217, September
1999. The applicable findings of this report are discussed in Chapter 3.
Results in Brief
The seven-member peer review panel selected to review NCEA's draft report
included one panelist with an extensive history of providing expert testimony and
similar services for defendants in asbestos lawsuits, and two panelists who had
made prior public statements regarding the safety of the air around the World
Trade Center (WTC) site. These circumstances provided a basis for the
perception that one panelist had a potential conflict of interest and two panelists
had potential biases that would prevent them from providing impartial input.
However, we did not find that the panel's input was biased, nor that these
perceived biases and conflicts were so "direct and substantial" that any of the
panelists should have been excluded from the panel. None of the seven panelists
had concerns about the way the peer review process was conducted.
Nonetheless, given the highly sensitive and controversial nature of the subject, in
our view additional measures should have been taken during the peer review
selection process to disclose the information about panel members upon which the
allegations of bias and conflicts were later made. Specifically, had EPA's
Contractor taken additional actions - such as inquiring about publicly expressed
viewpoints on the subject matter - to disclose this information prior to the
selection of the panel, EPA's Contractor could have considered the need to
balance the panel or otherwise resolve these perceived, potential conflicts and
-------
biases prior to conducting the July 2003 peer review. Regarding the six
allegations, we found that:
The National Contingency Plan does not require, as alleged, that the pUblic
be given an opportunity to comment on the choice of peer review panelists,
or the opportunity to suggest other experts who could represent the views of
the community.
Neither EPA nor its Contractor was required by Procurement Policy Notice
95-04, as alleged, to evaluate the sensitivity of the work being performed
when evaluating potential post-award conflicts of interest.
• EPA's Contractor did not obtain as much information on prospective
panelists as would be needed to adequately assess potential conflicts of
interest and biases. Specifically, the Contractor did not inquire as to whether
the three prospective panelists had received funding from industry, or had
expressed viewpoints on the issues to be reviewed.
The asbestos expert did not have a "direct and substantial" conflict of
interest, as alleged, but there was a basis for the perception of a potential
conflict of interest.
The two panelists who had publicly expressed opinions regarding certain
WTC hazards did not appear to have significant biases, as alleged, because
the opinions expressed were either sufficiently qualified or not directly
related to the subject matter to be peer reviewed.
EPA or its Contractor could have taken additional measures to obtain more
information on prospective panelists' work experience and viewpoints,
which would have necessitated additional actions on the part of EPA or the
Contractor to assess the balance of the panel.
EPA's Peer Review Handbook provides significant flexibility to the peer review
leader in forming a panel considering the range and variety of EPA scientific and
technical work products that are peer reviewed. The Handbook also
acknowledges that obtaining appropriate expertise may mean that some panelists
with potential conflicts and biases are nonetheless needed. We noted that the
Handbook did not always give sufficient instructions on how to define and treat
issues such as bias and panel balance, and additional guidance and training of
peer review leaders on handling sensitive or controversial peer reviews could be
helpful.
-------
Recommendations
We are making a number of recommendations to EPA to better ensure that
guidance in EPA's Peer Review Handbook is Hilly followed, including (1) better
oversight of peer review contracts to ensure that potential panelists are asked
about the industry financing they have received, and their relationship with any
clients whose interests might be affected by the subject being peer reviewed; and
(2) supplemental guidance and training of peer review leaders regarding the types
of information they may need to obtain about potential panelists' opinions and
viewpoints when they assess whether panels are independent and balanced.
Agency Comments and OIG Evaluation
The Agency agreed with our draft report findings, conclusions, and
recommendations. See Appendix C for the Office of Research and
Development's (ORD) response to our draft report. Additionally, in response to
our draft report, on October 7, 2004, ORD issued Policy Announcement #NCEA-
EM-05-01 which requires that specific conflict of interest (COI) provisions be
included in task order statements of work for EPA peer reviews, unless these
provisions are waived by the NCEA Director (see Appendix E for this Policy
Announcement). These provisions adequately address many of our concerns.
ORD also identified milestone dates for (a) developing additional guidance to
supplement the existing EPA Peer Review Handbook, (b) revising existing
training materials, and (c) developing a Peer Review Website. Further, ORD
stated it would ask the Science Policy Council to adopt an Agency-wide policy
directing its contractors to use the information-collection forms that are currently
being used by the EPA Science Advisory Board and the National Academy of
Sciences to aid staff in assessing peer review panel balance.
We commend the Office of Research and Development and the National Center
for Environmental Assessment for the actions already completed, ongoing, and
planned to ensure that scientific and technical work products consistently receive
adequate peer reviews. ORD's completed and planned actions are summarized in
the Corrective Action Plan (see Appendix D). Agency officials also made a
number of technical comments and clarifications to our draft report, which we
have included as appropriate.
-------
Chapter 2
Results of Review of the Six Allegations
A discussion of each allegation and the results of our review follows.
Question 1:,- Did the NCP Require Public Participation In Peer Review
Pane) Selection?
The Complainant alleged that the public was not given an opportunity to
comment on the choice of the seven panelists, or an opportunity to suggest other
members who could represent the views of the community, as the complainant
claimed was required by the National Contingency Plan (NCP). We found that
neither the NCP nor the Peer Review Handbook require public participation in the
selection of a peer review panel. Although the public did not participate in the
selection of the panel, the peer review meeting was held in public and the public
was given the opportunity to comment on the panel composition and the Draft
Assessment during this meeting. Although the extent of this participation may
have been less than the Complainant or others may have desired, the extent of the
participation was consistent with Agency peer review guidelines and prior
practices.
The Handbook acknowledges the general principle that obtaining a peer review
should be a transparent process: "Remember, the Agency is committed to
working 'as if in a fishbowP and most of its activities are transparent to the public
(except where confidential business information is concerned)." The Director of
EPA's National Center for Environmental Research and Quality Assurance,
Versar's Senior Project Manager for the peer review of NCEA's Draft
Assessment, and NCEA's Work Assignment Manager for the peer review told us
there was no requirement for public participation in the peer review process. We
confirmed that mis was the case during our examination of the NCP. The
officials further stated that EPA has conducted numerous peer reviews in the past
without allowing the public to suggest panelists or otherwise participate in the
selection of panelists. The Director estimated that for every thousand panels EPA
has convened, "fewer than ten would have public involvement" in the process of
selecting panelists.
Although the Handbook does not require the Agency to involve the public in the
panel selection process, the public was involved in the peer review meeting.
NCEA's task order to Versar contained the following requirements with respect
to public participation:
• the peer review meeting would be announced to the public though a Federal
Register Notice;
-------
• the deliberations of the panel would be open to the public;
• there would be a 2 to 3 hour period during which the public could address the
panel with specific comments;
• each person who asked to address the panel would be granted 5 minutes to
address the meeting; and
• the public could also submit comments in writing.
Based on our review of the audio tape and other documentation of the peer review
meeting, these requirements were fulfilled. For example, several individuals from
the general public accepted the opportunity to communicate directly with the
panelists. In addition, NCEA's Draft Assessment was made available to the
public prior to the meeting and comments were accepted from the public.
Additionally, the public was allowed to submit comments after the peer review
meeting to be included in the peer review summary report. The summary report
has also been made available to the public via EPA's public website.
Question 2: Was NCEA Obligated by Procurement Policy Notice
95-04 to Evaluate the Sensitivity of the Work Being
Performed?
The Complainant alleged that EPA did not follow the procedures outlined in
"EPA Procurement Policy Notice No. 95-04, Procedures for Handling Post
Award Organizational Conflicts of Interest" to address potential conflicts of
interest as defined in Federal Acquisition Regulation (FAR) Section 9.501. The
Complainant alleged that the Procurement Policy Notice provides specific
questions that EPA should ask contractors to determine whether potential
conflicts of interest exist, and requires EPA to evaluate the sensitivity of the work
being performed by tiie Contractor.
EPA's Procurement Policy Notice No. 95-04 provides guidance for evaluating
potential organizational conflicts of interest and does not apply to individual
potential conflicts of interest pertaining to a peer review panelist. The
requirements of the Procurement Policy Notice apply only to the government-
contractor relationship and to potential organizational conflicts of interest that a
contractor may have related to the work being performed for EPA.
We also reviewed the Peer Review Handbook for guidance on how potential
conflicts of interest discovered after the selection of a panel should be addressed.
The Handbook provides considerable guidance on how to address potential
conflicts of interest in selecting the panel; however, it does not include guidance
for handling potential conflicts of interest discovered after the panel had been
selected.
-------
Question 3: Was NCEA Obligated ID insure that Us Peer Review
Contractor Required Potential Panelists to Disclose All
Substantial Public Statements end AH Business
Relationships?
The Complainant alleged that EPA did not require its Contractor to obtain all
relevant information from potential panelists, including asking potential panelists
whether they had made prior public statements about WTC hazards, or to require
full disclosure of business relationships with industries that would be affected by
EPA's findings on WTC exposure hazards. We found that EPA's Contractor did
not obtain all the necessary information on prospective panelists that would be
needed to adequately assess potential conflicts of interest and biases. For
example, the Contractor did not inquire as to whether the three prospective
panelists had received funding from industry, or had expressed viewpoints on the
issues to be reviewed.
EPA Guidance for Using Contractors to Arrange for Peer Review
Services
The Handbook includes specific requirements and procedures to be followed
when EPA uses a contractor to arrange for peer review services. The Handbook
states that:
The Statement of Work (SOW) must clearly specify that the
contractor is responsible for preparing peer review evaluations
and set forth guidelines for the peer review of scientific or
technical documents. The contractor may perform the peer
review with in-house staff, subcontractors, or consultants. Any
guidelines for performing peer reviews to ensure soundness and
defensibility must be developed by the program office and made
pan of the contract. The contractor would then ensure that peer
reviews adhere to the guidelines.
According to the Handbook, the Statement of Work should direct the contractor
"... to inquire whether prospective peer reviewers have any actual or potential
organizational or personnel conflicts of interest or other matters that would create
the appearance of a lack of impartiality, including whether they have had or
presently have a financial relationship with EPA." According to the Handbook,
When the peer review process is being conducted by a
contractor, the requirement for addressing peer reviewer's
possible conflicts of interest should be highlighted in the
Statement of Work of the work ordering instrument (e.g., Work
Assignment, Delivery Order, Task Order, etc.) and is a matter
that is bound by contractual clauses with the Contracting
Officer as the final Decision Maker in contracting matters.
-------
When contracting for peer review services, EPA officials cannot select the peer
reviewers. Such action would interfere with the contractor's authority and
responsibility to perform the work under contract. EPA officials can establish the
criteria for the type of individual that might participate on a peer review panel,
but they should avoid commenting on the contractor's selection of peer review
panelists other than to determine whether the panel, once selected, meets the
criteria established.
EPA's Peer Review Guidance does not specifically require EPA - or its
contractor when conducting the peer review for EPA - to specifically ask
panelists about their prior public statements, if any, related to the matters to be
peer reviewed, and no attempt was made to obtain prospective panelists'
viewpoints on the subject matter for this peer review. However, the Peer Review
Handbook states that a panel should represent a balanced range of technically
legitimate viewpoints, and in general, panelists who have publicly commented on
the subject should be avoided. Accordingly, it would be difficult to adequately
carry- out the intent of EPA's Peer Review Handbook without obtaining
information on prospective panelists' stated positions or viewpoints.
NCEA's Actions to Ensure that Potential Conflicts of Interest or
Biases Were Identified
NCEA's contract with Versar identified the actions that Versar was to take to
ensure that potential conflicts of interest or biases were identified prior to
selecting the peer review panel. Among other things, this contract defined a
conflict of interest and required Versar to include a substantially similar definition
in any subcontract or consultant agreement which provided services for the
contract. NCEA's contract with Versar required Versar to obtain:
recognized, independent peer-reviewers who will provide
impartial evaluations of NCEA products . . .[and] peer
reviewers ., . shall be free of real or perceived conflicts of
interest, and shall represent a balanced range of technically
legitimate points of view and disciplinary mix.
NCEA's task order to Versar for the peer review of NCEA's Draft Assessment
contained a statement of work which identified areas of expertise that were
desired for the peer review panelists. However, NCEA's task order did not
specifically discuss conflicts of interest or biases but contained a requirement that
NCEA may, after receiving the list of potential panelists, provide feedback to
Versar concerning "any known conflict of interest, or concern about the
appropriate expertise."
10
-------
Contractor's Actions to Identify Potential Conflicts of Interest or
Biases
Versar took actions to identify potential conflicts of interest but did not take all
of the actions that were required by its Contract with NCEA or the actions that
were prescribed in its own internal guidance documents. Versar took the
following principal actions to identify conflicts of interest:
• screened potential panelists to obtain diversity;
• discussed conflicts of interest with seven candidates over the telephone
before their selection (according to Versar's Senior Project Manager);
• obtained personal conflict of interest statements from each selected panelist
before the peer review meeting was held on July 14, 2004;
obtained resumes from each of the panelists, which should have disclosed
important working relationships; and
• required each panelist to orally discuss or confirm the absence of a conflict of
interest at the beginning of the peer review meeting.
Versar identified at least 45 potential panelists before selecting 7 to serve as peer
reviewers. Six of the seven panelists had performed some work related to the
collapse of the WTC towers. In accordance with the requirements inNCEA's
task order, Versar sought experts with different backgrounds and different areas
of expertise. Versar's intent to get different viewpoints on the panel is illustrated
by the fact that one candidate panelist was "not called" because he was known to
collaborate with someone who had already been selected, or was likely to be
selected, for the panel. Further, Versar excluded at least three other prospective
panelists from consideration because of potential conflicts of interest or their
work relationships.
Contractor Did Not Fully Follow Guidance for Identifying Conflicts
The work assignment authorizations used by Versar to obtain the services of peer
review panelists did not define conflicts of interest or require panelists to report
conflicts of interest as required in its contract with NCEA. Further, Versar did
not take all of the actions which were prescribed in its own guidance documents
for identifying potential conflicts of interest. In particular, Versar did not inquire
specifically about industry funding of each panelist's work or each panelist's
working relationships with companies that might have a vested interest in the
results of NCEA's Draft Assessment. Such questions are important in uncovering
potential conflicts of interest.
Versar's generic Conflict of Interest Management Plan required that
subcontractors and consultants (selected panelists) perform "screenings" for
conflicts of interest and report the results of such screenings to Versar. The
screenings were to include, at a minimum, the identification of former and present
clients at the site where work is or has been performed, the type of work
performed, and the period of the performance. However, Versar did not follow
11
-------
this guidance. Versar did not ask panelists to identify their clients whose interests
might be affected by conclusions that could be drawn about WTC issues. When
they submitted their resumes and personal conflict of interest statements to
Versar, several panelists identified work they had completed related to the
conditions at the WTC site. However, panelists were selected without specific
questions being asked concerning the possible interest of the panelists' present or
prior clients.
We reviewed a written script that Versar reportedly used to interview potential
panelists over the telephone. The script contained a definition of conflict of
interest which, according to Versar's Senior Project Manager for the peer review,
was read over the phone to each panelist and discussed with them. This definition
embodied the principal concept of objectivity and impartiality that is contained in
EPA's Handbook and NCSA's Contract with Versar. The script also contained
the following questions:
Are you doing other work at the World Trade Center currently?
For whom? Other work at the WTC is not necessarily a COI
(conflict of interest), we just need to be aware and disclose this
information. In fact we see it as a potential asset.
Are you receiving funding from EPA currently? For what types
of projects? Anything else to disclose that might be related to
this review?
Versar's Senior Project Manager's notes, which summarize his telephone
discussions with the seven panelists who were ultimately selected, indicate that he
specifically discussed conflicts of interest with potential panelists. For example,
one panelist, who was not cited in any of the allegations, expressed concern that
he might have a conflict of interest because he mistakenly thought a paper he had
co-authored concerning the WTC may have been partially funded by EPA. Aside
from the documentation that indicates conflicts of interest were discussed orally
with potential panelists, each panelist signed a personal conflict of interest
statement that affirmed they did not have a conflict of interest.
The script did not include the following question that is contained in Versar's
standard operating procedures document entitled: Summary of Versar's Conflict
of Interest Screening and Certification for Peer Reviewers Under NCEA Peer
Review Contracts:
Are you receiving funding from industry for efforts that might
be related to this document!
We contacted the Versar Project Manager to determine why this question was not
asked but had not received a response at the time this report was prepared. The
NCEA work assignment manager told us he was satisfied that the panel met the
criteria established tea- this panel.
12
-------
Question 4^ Did IN As bestos E Xpert Have a Conflict of Interest?
The Complainant alleged that one panelist, Dr. Eric Chatfield "... is a paid
expert witness for W.R. Grace & Co., the firm responsible for the asbestos
contamination in homes in Libby, Montana..." and "In an expert report dated
8/30/02, Chatfield strongly opposed EPA's analytical evaluation of risks in homes
in Libby." The Complainant further alleged that the resume that Dr. Chatfield
submitted to Versar did not disclose this working relationship. The Complainant
alleged that this relationship represented a conflict of interest since W.R. Grace &
Co. was involved in litigation with the U.S. Department of Justice over
responsibility for removing asbestos contamination in Libby. We determined that
the asbestos expert did not have a "direct and substantial" conflict of interest, as
alleged, but there was a basis for the perception of a potential conflict of interest.
Handbook Definition of Conflict of Interest and Procedures for
Handling
EPA's Peer Review Handbook defines a conflict of interest as follows:
Conflict of interest is a situation in which, because of other activities or
relationships with other persons, an individual is unable or potentially
unable to render impartial assistance or advice to the Agency, or the
person's objectivity in performing the work is or might be otherwise
impaired, or a person has an unfair competitive advantage.
Further, the Handbook states that a conflict of interest may exist when a panelist's
"professional standing and status or the significance of their principal area of
work might be effected by the outcome of the peer review." The Handbook does
not specifically define "bias," but notes that the desire to select individuals with
experience on a panel often comes from those who are considered as having a
potential bias.
The Handbook discusses techniques to help ensure disclosure and appropriate
resolution of potential conflicts of interest Section 3.4.6 states:
One way of identifying conflicts of interest is to ask potential reviewers
about current and prior work, and prior clients that might create conflicts
or the appearance of a lack of impartiality in carrying out peer review
activities... .
The Handbook also states that each potential conflict of interest situation is
unique and must be treated on a case-by-case basis. The Handbook lists certain
factors that should be considered in evaluating potential conflicts of interest.
These factors include:
• attention to employment, financial, and professional affiliations of the
participants;
13
-------
• exploring directly the issue with each of the participants before the review
process takes place;
• disclosing publicly at the beginning of meetings any previous involvement
with the issue; and
• providing non-Federal peer reviewers with a copy of the peer review inquiry
conflict of interest form, and conducting a follow-up discussion to discuss
any relevant issues.
The Handbook offers additional guidance on when a person with a potential
conflict of interest can be used on a panel and when such a potential conflict of
interest would result in a person's exclusion from the panel. According to the
Handbook,
In fact, experts "with a stake in the outcome - - and therefore a potential
conflict - - may be some of the most knowledgeable and up-to-date experts
because they have concrete reasons to maintain their expertise. Such
experts could be used provided the potential conflicts of interest are
disclosed and the peer review panel or group being used as a whole is
balanced. In some cases, however, the conflict may be so direct and
substantial as to rule out a particular expert...
The Handbook's general definition of conflict of interest that would likely result
in a prospective panelist's exclusion from a panel is as follows:
Generally, a conflict of interest arises when the person is affected by
his/her private interests, when he/she or his/her associates would derive
benefit from incorporation of their point of view in an Agency product, or
when their professional standing and status or the significance of their
principal area of work might be affected by the outcome of the peer
review.
Those in charge of panel formation are encouraged to consult with EPA's Office
of General Counsel "whenever there are questions about conflicts of interest."
Disclosure of Dr. Chatfield's Work Experience
In regard to the alleged incompleteness of Dr. Chatfield's resume, the resume he
provided to Versar did not specifically disclose his working relationship with
W.R. Grace & Co. In our view, this relationship should have been disclosed to
the contractor during the screening process. However, Dr. Chatfield's 30-page,
single-lined resume identified approximately 75 instances where he had acted on
behalf of one or more defendants in litigation by providing "depositions,"
"certifications," "affidavits," "declarations," or "expert testimony." While not all
of these instances may have involved asbestos, his resume disclosed that he had
been an "expert witness on behalf of several defendants in asbestos property
14
-------
damage suits" and "asbestos personal injury suits." Accordingly, it should have
been clear to NCEA and Versar officials, as well as other panelists, that Dr.
Chatfield had extensive working relationships with asbestos companies, such as
W.R. Grace & Co., and other defendants in asbestos cases.
According to the Complainant, Dr. Chatfield's relationship with W.R. Grace &
Co. was significant because as part of its legal defense, W.R. Grace & Co. was
using comparisons of the cleanup standards required in Libby, Montana, to the
cleanup standards being used around the WTC. Specifically, W.R. Grace & Co.
was asserting that it was being held to a higher standard in removing asbestos
contamination from buildings in Libby, even though they contended that the
contamination levels in Libby were not as bad as the contamination levels in
buildings around the WTC.
In regard to the extent of Dr. Chatfield's relationship with W.R. Grace & Co., he
maintains that his work for W.R. Grace & Co. involved minimal time and effort,
involving less than 2 weeks total effort, and was not significant in relation to his
many other activities, especially since he was never called to testify. A brief
chronology of his work for W.R. Grace & Co. follows:
• Hired by W.R. Grace & Co. on August 6, 2002, forthe purpose of being an
expert rebuttal witness concerning data that were obtained by EPA using an
analytical method which Dr. Chatfield developed.
• Submitted a report dated August 30, 2002, rebutting the analytical work done
by EPA's Contractor concerning air samples at Libby by transmission
electron microscopy. The report documented cases of misidentification of
fibers and misuse of the analytical method.
• Deposed by a U. S. Department of Justice attorney on September 29,2002.
In anticipation of a January 2003 trial, he made one visit to the attorney
representing W.R. Grace & Co. to discuss this matter.
• Contractual services for W.R. Grace & Co. terminated in January 2003, when
the parties settled out of court.
Dr. Chatfield stated that as a rebuttal witness for W.R. Grace & Co., his work was
limited in that he could only rebut the evidence proffered by others in this matter,
and could not introduce any unsolicited views. Dr. Chatfield also explained that
he had never been a consultant for W.R. Grace & Co., but only an expert witness
hired to provide testimony on data gathered by an analytical method which he
helped to develop in the late 1970s. We did not identify any documentation that
challenged Dr. Chatfield's statements that he did "... less than two weeks work"
or that he has "... never been a consultant for W.R. Grace either before or after
this limited assignment." In addition, the allegations pertaining to Dr.
Chatfield's relationship with W.R. Grace & Co. were raised at the public peer
15
-------
review meeting, and were fully disclosed to other panelists at the meeting. The
Peer Review Handbook notes that full disclosure and discussion of potential
conflicts of interest before the entire panel is one method for resolving such
conflicts.
Our review of Dr. Chatfield's past work as detailed in his resume did not identify
any other work relationships between Dr. Chatfield and W.R. Grace & Co.
Additionally, the work relationship between Dr. Chatfield and W.R. Grace & Co.
did not appear to involve significant time and effort. Thus, this potential conflict
of interest was not sufficiently "direct and substantial," as discussed in EPA's
Peer Review Handbook, to warrant Dr. Chatfield's exclusion from the panel.
Notwithstanding Dr. Chatfield's limited relationship with W.R. Grace & Co., the
fact that he had many working relationships with many asbestos companies over
the years is a basis for a perception that he may have a potential conflict of
interest. Accordingly, we evaluated his input to the peer review (both oral and
written comments) for any indication of bias. We did not identify any biased
input. During the peer review meeting and through his written peer review
comments, he called for treating information cautiously. For example, he
recommended that:
NCEA's Final Assessment state specifically that there are a number of
potential contaminants which have not been measured and that uncertainty
remains, and he encouraged governmental officials to identify the limitations
of their work; and
• all dust samples be treated as if they contained one half percent to 3 percent
asbestos, even when a dust sample was reported to be free of asbestos.
Further, Dr. Chatfield did not comment on the health implications of the data. He
told us that his area of expertise is asbestos sampling and analytical methodology,
and that he did not feel qualified to address health issues. He told us that he does
not make statements about health effects, but instead evaluates considerations
such as sampling methodology and data quality.
Also, in a report on asbestos concentrations measured in two apartments near the
WTC site, Dr. Chatfield recommended that nearby residents obtain professional
cleaning of their residences at a time when the government was advising the
public to clean their own residences with wet rags, wet mops, and vacuums with
HEPA (high efficiency paniculate air) filters as long as there was less than
1 percent asbestos in dust samples. The positions taken by Dr. Chatfield in his
peer review of NCEA's Draft Assessment did not reflect biases on behalf of any
asbestos company or an asbestos industry that retained his services in the past. In
summary, Dr. Chatfield's relationship with W.R. Grace & Co. did not present a
"direct and substantial" conflict of interest as alleged, but there was a basis for the
perception of a potential conflict of interest
16
-------
5: Did Two Panelists Have Biases Based ait Their Prior
Stated Conclusions Regarding WTC Health Hazards?
The Complainant alleged that two panelists, Dr. Alison Geyh and Dr. Patrick
Kinney, "... are known to have advised the public through the press that there
were no hazards from WTC exposures. Their statements are prejudicial, making
it difficult for them to change their opinions at this later date to agree with any
EPA conclusions ... that there are potential or real risks from WTC exposures. "
The Complainant asserted that potential liability issues for these individuals or
their parent institutions would make it "difficult if not impossible" to agree with
any of the draft report's conclusions that differed from their prior public
statements about the health risks. We evaluated the public statements attributed
to Dr. Geyh and Dr. Kirmey to determine whether they demonstrated that these
panelists had "clearly taken a side" and whether these statements represented a
"direct and substantial" conflict as discussed in EPA's Handbook. In our opinion,
these two panelists did not appear to have significant biases because the opinions
expressed were either sufficiently qualified or not directly related to the subject
matter to be peer reviewed.
EPA Guidance on Bias
The Handbook does not specifically define bias but discusses how a potential bias
could impact panel formation. In discussing considerations for selecting peer
reviewers, the Handbook recognizes that a panel can be comprised of individuals
who are considered to have a potential bias and that this concern can be addressed
by ensuring that the panel is balanced. The Handbook states that:
.., the very need to have experienced individuals on a peer review, along
with the desire to have appropriate technical balance and representation,
can mean that the selection of potential peer reviewers often comes from
those who are considered as having a potential bias. To reduce the
concern that a potential panel may have unnecessary bias, it may be
useful to obtain an informal review of the expertise and balance of
potential peer reviewers from others in your organization, from OGC or
even from outside groups....
The Handbook provides additional discussion on the proper mix of a panel, and
notes that:
As a general rule, experts who have made public pronouncements on an
issue (e.g., those who have clearly 'taken sides') may have difficulty in
being objective and should be avoided.
17
-------
Dr. Geyh's Public Statements
The Complainant provided two news articles that contained quotations attributed
to Dr. Geyh, an assistant professor of environmental health sciences at Johns
Hopkins University Bloomberg School of Public Health. The two news articles
contained her comments about a research study that she directed. The
Complainant has not alleged that Dr. Geyh misinterpreted or misrepresented the
results of her research study. This study concerned WTC cleanup workers, not
the general public; thus, the focus of her study is different than NCEA's Draft
Assessment. The principal quotations attributed to Dr. Geyh by one news article
were:
Many of the workers we assessed reported coughing, wheezing, and sore
throats while working at Ground Zero. These symptoms seemed to
increase the longer they worked at the site. The gyod news is that we did
not find unhealthy levels of asbestos but we don't know what the long term
health risks may be regarding exposure to other airborne contaminants at
the site.
A second news article reported that Dr. Geyh said there was no sign of lung
obstruction, and quoted her as saying, "For the general community this is a very
good story..."
We provided Dr. Geyh with a copy of the Complainant's allegations about her
prior public statements. Dr. Geyh specifically denied stating there were no
hazards associated with exposures at the WTC site, as alleged. Further, she noted
that her statement about asbestos levels was "based on results of my own airborne
asbestos monitoring data collected directly on the disaster site," and that none of
her research involved residents, office workers, or first responders. She told us
that:
/ have no data regarding these groups and have made no definitive
statements or come to any definitive conclusions about the relationship
between asbestos levels and these groups.
We found that Dr. Geyh's statements reflected the factual results of her research
study. Additionally, during the peer review, she noted the limitations of existing
test data and the need for more data about the potential long-term health risks
associated with exposures at the WTC site. As such, her statements did not
demonstrate that she had "taken sides" with respect to the issues being peer
reviewed. We concluded that any potential biases arising from these statements
were not so material as to warrant her exclusion from the panel. However, the
fact that such opinions were expressed provided a basis for the perception that
there may have been a bias, and supports the need for EPA or its Contractor to
improve their efforts to ensure that peer review panels as a whole are balanced.
18
-------
Dr. Kinney's Public Statements
The Complainant provided several news articles that contained quotations
attributed to Dr. Kinney. The principal quotations attributed to him by the news
articles were:
The data that has been collected so far do not show concentrations of
pollutants that are of concern . . . Based on what we know so far, there
is no cause for alarm. .. . For the general public, as long as we are
careful, I think the risks are quite low. ... I do not think it was unwise to
have people start repopulating the area [near the World Trade Center].
However, not all of the pollutants that you might want to look at have
been monitored or, at least, reported yet. So the database is incomplete.
We provided Dr. Kinney with a copy of the Complainant's allegations about his
prior public statements. He agreed that he had made these statements, but denied
that it biased his input during the peer review. His written response to us included
the following statement:
/ never feel at all 'prejudiced' by a previous view I have held if new
information points in a different direction. . .. I'm perfectly willing to
change my mind on any issue based on new evidence. This is generally
the way scientists behave.
Our review of Dr. Kinney's written peer review comments indicated that he, in
fact, had advocated alerting the public to potential asbestos risks. He noted:
// would be good to append a concluding sentence to the effect that these
data support the conclusion that the WTC collapse resulted in increased
indoor asbestos levels in lower manhattan.
Dr. Kinney also advocated that NCEA provide more information in its assessment
and clearly communicate the potential risks to the public. For example, Dr.
Kinney's written comments expressed concern that the public may obtain an
unwarranted sense of comfort from the wording in NCEA's Draft Assessment
regarding occupational benchmarks. He noted that:
... the use of a health benchmark in a screening exercise is only valid if
the health benchmark is appropriate for the population at risk, which I
think is somewhat problematic here where occupational benchmarks are
being used to screen risks in the general population. ... somewhere early
in the executive summary there needs to be a short paragraph explaining
why occupational standards may not be directly relevant to general
population health risk assessment, including issues of voluntary and
compensated risks vs. involuntary risks, healthy workers vs. widely
varying susceptibilities in general population. This very important point
will not be obvious to the general reader.
19
-------
Further, Dr. Kinney's oral comments during the peer review meeting July 14-15,
2003, showed his concern about the need for a comprehensive review of indoor
air data and the need for EPA to earn the public's trust by advocating the use of
the National Academy of Science's panel model to obtain an unimpeachable
review of indoor air issues:
I would like to encourage them [EPA] to consider the advantages of
having an independent scientific group take the lead on organizing it [an
indoor air study or compilation of indoor air studies] mainly from the point
of view that - the credibility of the results would probably be enhanced
substantially over a purely sort of EPA or government- agency sponsored
or centered effort. So, given the controversies that have existed to date, I
think one way to sort of resolve the issues — these really important issues
about indoor air and exposure of health — in a way that everybody is
satisfied with, would be to get a bunch of external people .. . I am sort of
thinking of the National Academy of Science's panel model, where on
controversial issues, often government agencies will commission the
National Academy of Science to form an independent group of scientists to
go over an issue , . .
We also noted that Dr. Kinney's detailed written peer review comments reiterated
his concern about the incomplete database. He stated that:
/ believe that the overall conclusions of the evaluation are reasonable
given the available data. .. The report [NCEA's Draft Assessment] is
careful to point out that these are preliminary conclusions based on
available data and that conclusions could change when new information
becomes available.
Considering the qualification that was contained in the news article, Dr. Kinney's
advocacy of full disclosure, his concern that the public understand the potential
health risks, and the overall professional and constructive nature of the comments
that Dr. Kinney made during the peer review process, his statements did not
demonstrate that he had "taken sides" with respect to the issues being peer
reviewed. We concluded that any potential biases arising from the statements
cited by the Complainant were not so material as to warrant Dr. Kinney's
exclusion from the panel. However, the fact that such opinions were expressed
provided a basis for the perception that there may have been a bias, and supports
the need for EPA or its Contractor to improve their efforts to ensure fcat peer
review panels as a whole are balanced.
ORD's Perspective
ORD agreed that Versar could have done more to ensure balance regarding the
viewpoints held by prospective panelists, but ORD emphasized that Versar sought
balance regarding "scientific expertise," and ORD further stated that:
20
-------
For this highly visible World Trade Center Assessment, the contractor, at
EPA's direction, sought panelists who were familiar with the complex
issues presented by this unique environmental disaster, even to the point
of seeking out scientists who had conducted related research of their own
on World Trade Center exposure and health impacts. The two scientists
who had made the public comments noted in the complaint have both
conducted World Trade Center exposure and health impact studies that
ORD considers to be of high quality, and their input to the review spoke of
their familiarity and expertise with the issues. In cases such as this, a
strict adherence to a policy of avoiding scientists who have spoken on the
topic being reviewed may not be warranted.
Question 6; Did EPA's Contractor Adequate ly Consider Balance in
Selecting the Panel?
In expressing concerns about potential conflicts and biases with the panel, the
Complainant noted that a wide divergence of opinions could be accommodated in
a large balanced panel of expert scientists, but had concerns about a panel that
only consisted of seven members for this peer review. In light of the allegations
that certain panel members may have been biased in their points of view, we
reviewed Versar's efforts to ensure that the panel was balanced. We believe EPA
or its Contractor could have taken additional measures to obtain more information
on prospective panelists' work experience and viewpoints, which would have
necessitated additional actions on the part of EPA or the Contractor to assess the
balance of the panel. For example, as noted in Section 4, Versar did not ask
panelists to identify their present or prior clients whose interests might be affected
by conclusions that could be drawn about WTC issues.
EPA Guidance Concerning Panel Balance
According to EPA's Peer Review Handbook,"the peer reviewers of a work
product should represent a balanced range of technically legitimate points of
view." The Handbook also points out that "cultural diversity and 'address' (e.g.,
industrial, academic, or environmental community) are other factors that can play
a role in selecting peer reviewers."
The Handbook does not provide specific examples of inquiries that should be
made to ascertain whether a panel "as a whole" is balanced with respect to
viewpoints, but notes lhat established peer review groups such as EPA's Science
Advisory Board (SAB) provide useful models for addressing balance and conflict
of interest issues. As noted in a recent GAO report,2 when selecting SAB
panelists EPA specifically asks prospective panelists for information about their
points of view related to the issue being reviewed.
7Federal Advisory Committees.'Additional Guidance Could Help Agencies Better Ensure Independence
and Balance, U.S. General Accounting Office, Rpt. No. GAO-04-328, April 2004.
21
-------
Contractor's Efforts To Balance The Panel
The statement of work in NCEA's's task order to Versar identified areas of
expertise that were "desired" for the peer review panel. As required by EPA,
Versar selected the 7 panelists from these areas. Although Versar took several
steps to balance the panel, we saw no evidence of any special actions or
considerations by Versar to ensure that the peer review panel for the NCEA Draft
Assessment was balanced with respect to the specific allegations of the
Complainant. As noted earlier, Versar did not inquire as to whether prospective
panelists had publicly expressed viewpoints on the peer review subject matter.
Accordingly, no consideration was given to the need to balance the panel with
respect to publicly expressed viewpoints on the subject matter. We contacted the
Versar Project Manager to determine whether other actions were taken to balance
the panel, but we had not received a response at the time this report was prepared.
Panelists' Perspective on the Need To Balance WTC Draft
Assessment Panel
We contacted each of the panelists to obtain their opinions on the balance of the
WTC Draft Assessment Panel as a whole. Each panelist expressed his or her
opinion that the panel, as a whole:
• contained the right mix of expertise,
had an appropriate balance of ideas and opinions,
• examined the issues with an appropriate degree of balance, and
• allowed the expression of minority views.
None of the seven panelists had concerns about the way the peer review process
was conducted. As shown in Appendix B, the peer review panelists' backgrounds
included a variety of employers and technical expertise.
Conclusions
Additional measures could have been taken to identify potential biases and
conflicts with prospective panel members. For example, EPA could have had its
Contractor ask prospective peer review panelists if they had publicly expressed
viewpoints - a procedure used in selecting EPA Science Advisory Board
committees - regarding the issues to be peer reviewed. Further, EPA's
Contractor did not take all the actions required by its contract with EPA, nor did
the Contractor take all the actions prescribed in its own internal guidance
documents. Had such measures been taken, it is likely that EPA's Contractor
would have disclosed potential conflicts and perceived biases prior to formulation
of the panel. As such, the Contractor could have identified the need to assess the
22
-------
balance of the panel and taken steps to mitigate these perceived conflicts or
biases.
Neither the statements made by the two panelists nor the work relationships of
one panelist were so "direct and substantial" as to warrant their exclusion from
the panel. Further, ourreview of the panel's deliberations, comments, and peer
review record did not disclose any biased input in the peer review process, and
none of the panelists had concerns about the way the peer review process was
conducted. However, in selecting future panels for sensitive or controversial
issues, EPA and its contractors should take steps to identify prospective panelists'
viewpoints on the subject being reviewed, including prior public statements, and
evaluate the need to resolve any potential biases disclosed through these actions,
including whether the viewpoints of the panel should be balanced as a whole.
Recommendation
2-1 We recommend that the Director, National Center for Environmental
Assessment, implement procedures to ensure that guidance in EPA's Peer
Review Handbook is followed in carrying outNCEA peer reviews,
specifically that:
a. NCEA's peer review contracts, as well as any EPA peer review
Contractors' consultant agreements and subcontracts, contain written
guidance concerning information to consider when evaluating
potential conflicts of interest and biases, and
b. NCEA's potential panelists are asked about the industry financing they
have received, and their relationship with any clients whose interests
might be affected by the subject being peer reviewed.
Agency Comments and OIG Evaluation
The Agency agreed that the contractor should have gathered more information
from the prospective panelists and that ORD "should provide written guidance
and detailed instructions to its peer review contractors concerning potential COI
and biases." On October 7, 2004, ORD issued Policy Announcement #NCEA-
EM-05-01, which requires that specific conflict of interest (COI) provisions be
included in task order statements of work for peer reviews, unless these
provisions are waived by the NCEA Director (see Appendix E for this Policy
Announcement). These provisions adequately address our concerns. ORD did
not request any changes in the draft report wording of Chapter 2 except for the
addition of a few woids to our first recommendation. We made the requested
change to reflect our agreement with ORD.
23
-------
Chapters
Observations on Panel Formation and Balance
In addressing the six questions discussed in Chapter 2, we relied heavily upon
EPA's Peer Review Handbook for guidance and criteria in evaluating EPA's and
its Contractor's actions in conducting the peer review. The Handbook was
viewed as the key guidance on how to conduct an EPA peer review. We noted
that the Handbook did not always give sufficient instructions on how to define
and treat issues such as bias and panel balance. Accordingly, we contacted EPA
officials who helped develop the Handbook to better understand the intent of the
Agency's guidance. Based on our review of the Handbook, our discussion with
these EPA officials, and the issues discussed in Chapter 2, we believe EPA's
Handbook should be supplemented with additional instructions on how EPA and
its contractors should select panels for highly controversial and sensitive issues,
such as the risk assessment conducted for the WTC Draft Assessment.
Handbook Guidance on Panel Balance
EPA's Peer Review Handbook says that "As a general rule, experts who have
made public pronouncements on an issue (e.g., those who have 'taken sides') may
have difficulty in being objective and should be avoided," and that EPA is
"strongly encouraged to obtain peer reviewers who do not have a legal or
perceived conflict of interest." Further, EPA is to exercise an extra degree of care
in selecting panelists on sensitive or controversial issues, which in our view
includes EPA's Draft Assessment on exposure and health risks from air pollution
after the collapse of the WTC towers. However, EPA's Peer Review Handbook
also notes that obtaining appropriate expertise may mean that some panelists with
potential conflicts and biases are nonetheless needed. For example, the
Handbook notes that key methods for handling such concerns involve, among
other things, full disclosure of potential conflicts of interest and ensuring that the
panel is balanced as a whole.
Experts' Comments on EPA's Peer Review Handbook
EPA's Peer Review Handbook states that established peer review groips, such as
EPA's SAB, provide useful models for addressing balance and potential conflict
of interest issues. As such, we discussed these issues with four knowledgeable
experts on peer review issues - one of the principal authors of EPA's Handbook,
the Director of EPA's SAB, an SAB Ethics Official, and a former EPA Office of
General Counsel Attorney who advised the authors of the Handbook.
The principal author of the Handbook we contacted told us that, in creating the
Handbook, the authors had not intended to provide a "cook book" but had,
instead, intended to provide a generic document that provided significant
24
-------
flexibility to the peer review leader in forming a panel considering the range and
variety of EPA scientific and technical work products that are peer reviewed. He
and the SAB Director noted that the Handbook allows the peer review leader to
consider the sensitivity or controversial nature of the work product, and the
importance of the document for Agency decision-making, as well as the cost and
time required. They noted both cost and time can increase dramatically with
more elaborate peer reviews.
In summary, they believed that the existing Peer Review Handbook, if properly
implemented, generally provides sufficient guidance and flexibility for carrying
out an effective peer review. The Handbook's principal author also noted that
revising the Handbook would require a great deal of effort. Nonetheless, the
principal author acknowledged that additional guidance for dealing with
particularly complex issues - such as potential bias and panel balance - could be
helpful. The Handbook author noted that he would prefer to do this by providing
peer review leaders with supplemental guidance regarding the types of
information they may need to obtain about potential panelists' opinions and
viewpoints when they assess whether panels are independent and balanced. They
could also issue a supplemental package of training materials to peer review
leaders, and ensure mat EPA project managers and EPA contractors receive this
training.
We also discussed ways to mitigate the potential for public perceptions that a
panelist might have a conflict of interest or a bias. The SAB Ethics Official told
us that, generally, "the best disinfectant is public involvement." Noting that the
SAB now publicizes potential candidates on its website, we discussed the
advantages and disadvantages of this practice for EPA's other peer reviews. The
SAB Director and the Ethics Official noted that publicizing relevant biographical
information about potential panelists on EPA's website for a specific time, such
as 2 weeks, and soliciting and considering public input concerning the proposed
composition of a panel during the panel formation process, would be possible.
However, it would cost more and take longer, and some prospective panelists may
exclude themselves from the candidate pool under this approach, making it even
more difficult to find qualified peer reviewers. Both the SAB Director and the
Ethics Official noted that considering public input on panel formation could be
worthwhile for the more sensitive and controversial peer reviews.
Prior OIG Report Discussed Concerns With EPA Efforts to Assess
Peer Reviewers' Independence
The OIG last reported3 on EPA's efforts to ensure proper implementation of peer
reviews in 1999. The OIG reviewed 32 work products that were peer reviewed -
14 peer reviews arranged by EPA program offices, 12 by EPA contractors, and 6
by the SAB or the Science Advisory Panel. The OIG found that the SAB and
3£PA 'x Selection of Peer Reviewers, EPA Office of Inspector General, Rpt. No. 1999-P-217, Sept. 1999.
25
-------
Science Advisory Panel's peer review procedures were effective and being
followed. However, the DIG identified instances where peer review leaders in
EPA program offices and contractors did not effectively attempt to determine
whether conditions existed that may have precluded an independent review.
Regarding peer reviews contracted by EPA's program offices, the report
concluded that:
. .. EPA program offices should ensure that contract documents (the basic
contract, work assignments and work plans) consistently include specific
provisions for contractors to address independence concerns with peer
review candidates. ... we are recommending that ORD (Office of
Research and Development) issue supplemental guidance to the
Handbook to help identify and resolve independence concerns.
Increased Federal Attention to Peer Review Panel Balance
The selection of peer review panels for review of government documents has
received considerable attention and scrutiny recently. In September 2003, the
Office of Management and Budget's (OMB's) Office of Information and
Regulatory Affairs published a draft Peer Review Bulletin and, in April 2004,
OMB issued a revised version based on the numerous comments received. Both
versions, if approved, would establish minimum standards for when a peer review
is required for "scientific information" and the types of peer review that should be
considered by Federal agencies in different circumstances. The need for OMB
guidance on conducting peer reviews is based, in part, on a 1999 GAO report
which found variability in both the definition and implementation of peer reviews
across Federal agencies.4 The OMB Bulletin specifically emphasizes
consideration of expertise and balance when selecting peer reviewers. The
Bulletin states that:
Reviewers should also be selected to represent a diversity of scientific
opinions relevant to the subject. On most controversial issues, there exists
a range of respected scientific viewpoints regarding the interpretation of
the literature. Inviting reviewers with competing views on the science may
lead to a sharper, more focused peer review.
With respect to reviewers who are not Federal employees, the draft Bulletin states
that agencies should "adopt or adapt" the prevailing practices of the National
Academy of Sciences regarding committee composition, conflicts, and balance.
Potential candidates for National Academy of Sciences' peer review committees
are required to complete a "Background Information and Confidential Conflict of
Interest Disclosure" form. Among other things, this form asks that prospective
panelists list their relevant articles, testimony, speeches, etc., and the relevant
^Federal Research: Peer Review Practices at Federal Agencies Vary, U.S. General Accounting Office, Rpt.
No. GAO/RCED-99-99, March 1999.
26
-------
positions of any organization or group with which the candidate is closely
identified or associated. The form defines "relevant" as information
. .. related to - and might be reasonably of interest to others concerning
-your knowledge, experience, and personal perspectives regarding the
subject matter and issues to be addressed by the committee activity for
which this form is being prepared.
An April 2004 GAO report5 commented on the lack of guidance to assist agencies
in ensuring better independence and balance in Federal advisory committees, and
provides useful information with respect to assessing conflict of interest and
balance questions in other peer review efforts. For example, GAO reported that
General Services Administration guidance on advisory committee management
does not address the types of information that could be helpful to agencies in
assessing the points of view of potential committee members, nor do agency
procedures identify the information that should be collected about potential
members to make decisions about committee balance.
Conclusions
Given the importance of peer review in maintaining the integrity of EPA's
various scientific and technical work products, EPA should take steps to ensure
that the Handbook is followed when conducting peer reviews of highly sensitive
or controversial subject matters, and that issues of potential bias and overall panel
balance are sufficiently addressed. Two such actions would involve (1) issuing
supplemental guidance for peer review leaders on the types of information they
may need to obtain about potential panelists' opinions and viewpoints when they
assess whether panels are independent and balanced, and (2) ensuring that peer
review leaders receive training on this supplemental guidance.
Recommendations
We recommend that the Assistant Administrator for Research and Development,
as the Agency Science Advisor:
3-1 Revise the Peer Review Handbook or issue supplemental guidance -
applicable to peer review of sensitive or controversial issues - that EPA
and its contractor should follow in identifying and addressing potential
conflicts of interest and biases, and for ensuring that panels are balanced
as a whole. Among other things, this guidance should include procedures
for:
Federal Advisory Committees: Additional Guidance Could Help Agencies Be tier Ensure Independence
and Balance, U.S. General Accounting Office, Rpt. No. GAO-04-328, April 2004.
27
-------
a. explicitly determining whether prospective panelists have viewpoints
on the issues to be peer reviewed and, if so, whether these viewpoints
have been expressed publicly;
b. using general guidelines to assist peer review leaders in determining
whether a panel is balanced as a whole;
c. handling potential conflicts of interests and biases that are identified
after the panel has been selected;
d. ensuring that contracts for peer review services reference the Peer
Review Handbook as criteria to be used by the contractor in selecting
panelists;
e. documenting the basis or rationale for final panel formation prior to
commencement of the peer review, particularly the actions taken to
address any potential conflicts of interest, biases, and to ensure panel
balance; and
f. identifying the extent to which the public should be involved in the
peer review process and, in particular, under what circumstances the
Peer Review Leader should consider soliciting input from the public
on the potential makeup of the peer review panel.
3-2 Revise the training materials for peer review leaders and others involved
in peer reviews for identifying and addressing potential conflicts of
interest and biases, and ensuring panel balance; and ensure that peer
review leaders, EPA project managers, and EPA contractors receive fliis
training.
3-3 For highly sensitive or controversial peer reviews, consider the
development of procedures for using the Agency's website to
electronically post proposed panel designs, and obtain public comments
on proposed panel designs prior to panel formation, in a manner similar to
that presently employed by the Science Advisory Board.
3-4 In regard to the overriding need to obtain as much information as
necessary to provide reasonable assurance that potential panelist will be
impartial, consider the information-collection forms that are used by the
Science Advisory Board and the National Academy of Sciences (these
forms can be found at http://www.nas.edu/coysi-coi FORM-2.pdf and at
http:tfwww.epa.gov/sabybdf/eoafQrrn3l 10-48.pdf).
Agency Comments and OIG Evaluation
The Agency agreed with our recommendations, noting that "... additional steps
are needed to ensure that additional attention is given to the formulation of peer
review panels ...." Agency officials also stated that they plan to develop
28
-------
"supplemental guidance and instructions that articulate the procedure(s) for
identifying and addressing potential COI and biases as well as for ensuring that
panels are balanced as a whole .... " The Agency also agreed that Versar could
have done more to ensure balance regarding the viewpoints held by potential
panelists, but pointed out that NCEA emphasized that Versar sought balance
regarding "scientific expertise." NCEA suggested wording be added to our report
to this effect, which we included in its entirety at the end of our discussion of the
fifth allegation. We also clarified one recommendation in this chapter, as
requested by ORD, to recognize that ORD already provided training and training
materials for peer review leaders and others involved in peer reviews, but that the
Agency would be developing additional training emphasizing identifying and
addressing potential conflicts of interest and biases, and ensuring panel balance.
The actions already completed, ongoing, and planned to ensure that scientific and
technical work products consistently receive adequate peer review, if properly
implemented, address our concerns.
29
-------
Appendix A
Complainant's Allegation Memorandum
•
UW1ED STATEBBWRONBEKTAL PROTECTION MOW
WttHNGTCHDJC.SMW
Jiily 16,2003
•VBJCCTt Cmrillel of Int
Bvtf uM«ii «f Alih
IMWMillTlmd*
TO:
Otatf Kern
funHufidd.GiriiDuilap,
f ifcfckiwlK. 5*P* !
HE EPAIUumd <3™» Cm BB
udMattHida Enh^maf AnteaiBBalUoDtem«M World IMde Center DUamV
DMobcr,ZM2,nilMied to Aepublk inlM Decdnbir, ta (BidMHQbgpMr
pueliih.
(ciccttMi. Bowmw,aiiKpMivri«wf«^kichiibivily7pi(luiptMf. Itc public «• not
ChftB my o^portiaiky b caoEDBitaoftc chadixaftheiG iadindndU, OTIO
»«gt( itotermeiDbtii ivb»«oald Kprwat «h« viow»«f do «nxuoeltytm
(Monti COnihiuuty Pin.
i.tflk*U.&B»fa
i n turn «f tf* ubtr ud A run MMUKOr idM
30
-------
EPA has failed to insure that the government acts with impartiality and provide the public with
full opportunity for participation in die risk assessment process for the WTC disaster. In my
7/4/03 report and memorandum to the EPA Inspector General," evidence was provided of
EPA's similar failure to prevent conflicts of interest in expert panelists chosen for the EPA/IRIS
asbestos reassessment.
EPA did not require their contractor, Versar, to obtain all relevant information from potential
panelists, including asking specific questions whether potential panelists had ever stated
conclusions that there were no hazards from WTC exposures, or to require the submission of full
disclosures of all business relationships with industries that would be affected by EPA findings
on WTC exposure hazards.
The Federal Acquisition Regulations Section 9.501 define conflicts of interest of governmental
contractors as follows:
"... because of other activities or relationships with other persons, a person is unable or
potentially unable to render impartial assistance or advice to the Government, or the
person's objectivity in performing the contract work is or might be otherwise impaired, or a
person has an unfair competitive advantage."
EPA has issued specific guidance on evaluating and handling post contact award conflict of
interest situations in Procurement Policy Notice No. 95 - 04, Proced ures for Handling Post
Award Organizational Conflicts of Interest (COI), dated 9/20/95, posted on the EPA website at
http://www.epa.gov/oam/ptod/COI/ppn95-04.pdf. Specific questions are posed for EPA to
inquire of contractors to determine potential conflicts. COI criteria also include EPA evaluations
of the sensitivity of the work being performed by the contractor. EPA implemented none of
these procedures to control COI in the expert peer review panel for the NCEA risk assessment.
Eric Chatfield, expert witness for W. R. Grace asbestos company
One member of EPA's peer review panel for the NCEA risk assessment is Eric Chatfield. He is
a paid expert witness forW.R. Grace & Co., the firm responsible for the asbestos contamination
in homes in Libby, Montana, the Superfund site. Litigation has been brought against Grace by
the U.S. Department of Justice.
In an expert report dated 8/30/02, Chatfield strongly opposed EPA's analytical evaluation of risks in
homes in Libby. A copy of his expert report is attached to this memorandum, which reads in part as
follows:
Jenkins, C. (1/4/03) Comments on the EPA Office of Inspector General's 1/27/03 interim report titled:
EPA's Response to the World Trade Center Towers Collapse — A DOCUMENTARY BASIS FOR LITIGATION.
This report is available at the following 2 websites:
www.NYenviroLAW.org (see banner at top of page) and http://www.nycosh.org/tfanchorl5430
31
-------
I have reviewed the action memoranda for the Lbby site, the administrative record for the
period may 23, 2000 to June 4, 2002, the expert reports I have also reviewed the
ATSDR Health Consultation dated December 12, 2000, a Report to the Community dated
August 23,2001, and a report entitled "Health Risks from Exposure to Zonolite Home
Insulation: Critical Evaluation of the Scientific Evidence Available from the ATSDR
Studies in Libby, Montana.
• • •
I have determined that there ana significant deficiencies in the underlying analytical data
that form the basis for the risk assessments conducted by EPA. I have also determined
that some analytical methods have not been used h accordance with the specifications of
the methods, and that some of the analytical methods used are not capable of providing a
reliable foundation for the decisions being made.
[Report No. 02C029, Expert Report of Eric J Chatfield, Ph.D., in the matter of: United
States v. WR. Grace & Co., et al., Civil Action No. 01-72-M-DWM (D.Mont.), 8/30/02]
On 7/14/03, Chatfield is reported as telling the participants in the peer review meeting for the
EPA NCEA report that he did not think it was important to mention his work for Grace in the
biography he provided to meeting participants, and that his work for Grace was minimal. He is
incorrect on both points.
First, his work for Grace is not minimal or in the past. Providing an "expert report" to a
defendant in active litigation is just one step in the services provided by an expert witness."* In
order for the expert report to be of any use to Grace, the expert also m ust make himself available
to the litigant, in this case the Dept, of Justice, for deposition. Otherwise, the expert report
would not be admissible as evidence. The expert witness would also need to be available to
testify in court in the event that the case went to trial.
Second, his business connection with Grace is extremely relevant to the WTC investigation.
This is because Grace has been using for its legal defense comparisons between EPA's stringent
cleanup standards for homes in Libby compared to the lenient standards EPA has imposed for
the WTC cleanup. Grace is also using in its defense the fact that homes in Libby are less
contaminated with asbestos than buildings in Manhattan after Ihe WTC collapse.
Thus, Grace has a vested interest in having EPA conclude (by way of the NCEA human health assessment
document) that citizens in NYC are not at risk from asbestos exposures. They can
then directly use these EPA conclusions to escape responsibility for the contamination in Libby.
As a note for the reccrd, on 7/14/03, Chatfield told the meeting participants that it was 1 who was
seeking a copy of his export report through a Freedom of Information Act request, and that he would have gladly
supplied me with a copy if 1 had written. This also is untrue, as it was a concerned parent from (he Brookfield
school system, Kathy Huke, who sought this report and obtained it from EPA through the FO1A. Neither the citizen
nor 1 knew of the existence of the expert report until it was provided by EPA. The FO1A request was broad and
included all writings by Chatfield submitted to EPA regarding WR Grace. Furthermore, on 5/30/02 I wrote to
Chatfield for clarification of claims he made to the same Ms. Hulce regarding the Brookfield school system and
received no reply, so it would not have appeared fruitful for me to have made any new inquiries directly to Chatfield.
32
-------
The following are two letters (attached) from Grace to EPA, written in the same time frame as
Chatfield's expert report, which illustrate this point.
The first letter, below, attempts to gain more lenient cleanup standards forLibby based on the
fact that EPA is imposing less stringent asbestos air and dust standards in New York City after
the WTC collapse:
The purpose of tris letter is to continue Grace's dialogue wth EPA regarding Zonolite
Attic insulation ("ZAI"), a home insulation product consisting of expanded vermiculite. For
your background, ZAI has been insulating homes for over 60 years and there is no
credible reason to believe that ZAI has ever caused an asbestos-related disease in
anyone who has used it in his/her home. Nevertheless, EPA Region 8 is requesting that
EPA declare, for the first time ever, that a "pubic health emergency exists at a
(proposed) Superfund site.
• • •
Contrasted to Region 8's disregard of established norms, EPA's pronouncements and
activities regarding the World Trade Center collapse reaffirm those norms. Thus, EPA's
website reiterates that:
— EPA is using the 1% definition of an asbestos-containing material in evaluating
dust and bulk samples.
— Air samples are the accurate measure of actual exposure potential, whereas the
presence of asbestos in dust is not necessarily a significant health hazard.
— EPA uses the "extremely stringent" standard under AHERA of 70 structures/sq.
mm. to evaluate the risk from asbestos in the outdoor and indoor air. That
standard was expressly chosen because it is the most stringent and protective,
and is based on assumptions of long-term exposure.
— Asbestos exposure becomes a health concern when high concentrations of
asbestos fibers are inhaled over a long period of time. The fact that a small
percentage of samples exceeded the AHERA standard was not considered to be
evidence of a significant health risk.
We believe that sound science dictates that the same peer reviewed methodologies for
assessing risks at the WTC should be applied across the board, including in LJbby,
Montana.
[emphasis in original) (letter to Christine Tcdd Whitman, Administrator, EPA, from William M.
Corcoran, Vice President, W. A. Grace & Co., dated 4/10/021
In the next letter, WR Grace compares asbestos contamination levels inLibby to those in NYC
after the WTC collapse. WR Grace states that residences in Manhattan are more contaminated
than in Libby (which is true). This is also offered as a rationale for not cleaning up Libby to the
more stringent standards imposed by Region 8, since the cleanup in M anhattan is rudimentary
and slipshod in comparison to that being required for Libby.
33
-------
We appreciate the opportunity you gave us on January 31 to present Grace's view of the
data from Libby, Montana that EPA is using to guide its decision regarding removal of
Zonolite Attic Insulation from homes in that community.
I would like to summarize several of the points that were made in the meeting.
Based on the data we have seen, there are very low levels of asbestos in surface dust in
homes in Libby, Montana. There is only chrysotile found on windowsills, which is a key
collection point. In all surfaces In Libby, chrysolite fibers exceed am phi bole fibers by 3X
to 30X. In fact, asbestos concentrations In dust in Libby homes is significantly lower than
that found in New York City before 9/11, is far below that found after 9/11, and is
significantly lower than the concentrations that EPA found in studies of carpets, ceilings,
and lights.
[letter to Michael Shapiro, Prhcipal Deputy Assistant Administrator, Office of Solid Waste
and Emergency Response, EPA, from William M. Corcoran, Vice President, WR. Grace
& Co., dated 2/6/02]
Prejudicial statements by other panelists
Two other panelists are known to have advised die public through the press that there were no
hazards from WTC exposures. Their statements are prejudicial, making it difficult for them to
change their opinions at this later date to agree with any EPA conclusions, no matter what the
data, that there are potential or real risks from WT C exposures.
There would not be the same kind of legal liability problem for a panelist and their parent
institutions if the panelist had earlier advised the public to take precautions over and beyond
those recommended by public officials at the time. Such a person could easily state that new
evidence demonstrated that such precautions were unnecessary. They would not now be in a
liability situation of ha ving failed to warn the public in the past, puffing them at risk.
Prejudicial statements by Patrick Kinney
One panel member, Patrick Kinney, has gone on record to state that tests had conclusively
demonstrated that asbestos concentrations as well as the other measured toxic constituents were
not of concern.
It would be difficult if not impossible for Kinney to now agree with any conclusions by EPA that
there are potential risks, because that would constitute a reversal of earlier advice Kinney offered
to the public through the press on the safety of returning home and being exposed to
uncontrolled WTC dust. Any findings or recommendations on the NCEA draft by Kinney would
always be influenced by his earlier "on record" comments below. There may be many other
places where Kinney made such statements as well.
34
-------
More than six weeks after the September 11 terrorist attack on New York, fires are still
smoldering under the debris of the former World Trade Center complex. The continuing
smoke is worrying some city residents about the quality of the air they are breathing.
City officials have tried to reassure New Yorkers, ordering a series of on-going tests to
monitor air quality. Patrick Wnney specializes in the effects of air quaity on respiratory
systems at Columbia Universitys School of Public Health. He savs the tests show there
are no toxic pollutants, like asbestos, tainting the quality pf air. "The data that has been
collected so far do not show concentrations of pollutants that are of concern." adds
Professor Kinnev. "However, not all of the pollutants that you might want to look at have
been monitored or, at least, reported yet. So the database is incomplete. Based on what
we know so far, there is no cause for alarm."
But Professor Kinney says it is unlikely that new data will show any pollutants that put
people at risk. "I do not think it was unwise to have people start repopulatina the area." he
says. "I think that oir noses are very sensitive and we pick up a lot of things that are not
necessarily toxic. I think it is good to be careful, to minimize exposure as much as
possible. For the general pubic, as long as we are careful, f think the risks are quite low."
[emphasis added] [NY Residents Worry About AirQuatity, Barbara Schoetzau, New York,
10K8/01. http://www.help-for-you.com/news/Oct2001/Ocf28/PRT28-38ArticlB.Mml]
Prejudicial statements by Allison Gcvh
The peer review panelist Alison Geyh has also made definitive conclusions that asbestos testing
showed no harmful levels, and that there were no lung obstructive airway problems for citizens.
As with the panelist Kinney, it would be difficult for Geyh from a liability standpoint to now
agree with any EPA conclusions indicating a hazard or compromised health risk.
Investigators from the Johns Hopkins Bloomberg School of Public Health now believe that
World Trade Center workers' respiratory problems are the result of exposure to dust and
airborne contaminants at Ground Zero. Workers assigned to clear debris have reported
coughing, wheezing and sore throats while working at Ground Zero. The symptoms
seemed to increase the longer they worked at the site, according to Dr. Alison Geyh, the
chief investigator and assistant professor of environmental health sciences at Johns
Hopkins. "The good news is that we dd not find unhealthy levels of asbestos, but we don't
know what the long term health risks may be regarding exposure to other airborne
contaminants at the site."
[emphasis added] World Trade Center Health Update Summarized by Cheryl Runyon from
the Environment News Service AmariScan, Aug. 23, 2002.
http://www.ncsl.org/programs/esnr/wtchealth.htm]
In a study of more than 180 cleanup workers, researchers at Johns Hopkins Bloomberg
School of Public Health found that acute respiratory symptoms such as coughs, phlegm
and wheezing were more prevalent than before they began working at the site. But in most
workers, said Dr. Alison Gevh. the lead researcher, there was no sign of lung obstruction.
35
-------
Tor the general community this is a very good story," Geyh said.
[Researchers tell of health studies tied to WTC disaster; By Caii Gtassman, 11/2/01,
Online News From The Tribeca Trib http.//www.tribecatrib.com/newsnov02/enviro2r html ]
PeterPreuss, Director, EPA ORD NCEA
Lester Grant Director, EPA ORD NCEA-RTP subdivision
Affected parties and other resp onsible o fficials
36
-------
Appendix B
Peer Review Panelists and Backgrounds
P*S*«*t Tttfc Expertise WTC Exp*riW*
Eric J. Chatfield,
Ph.D.
Michael Dourson,
Ph.D.
Alison Geyh,
Ph.D.
Gary Hunt,
M.S.
Patrick Kinney,
Sc.D.
Margaret
MacDonell, Ph.D.
Clifford P. Weisel,
Ph.D.
President,
Chatfield Technical
Consulting Limited
Director,
Toxicology Excellence
for Risk Assessment
Assistant Professor,
John Hopkins
University Bloomberg
School of Public
Health
Vice President of Air
Toxics Programs and
Director of Air Toxics
Monitoring,
TRC Environmental
Corporation
Associate Professor,
Mailman School of
Public Health at
Columbia University
Argonne National
Laboratory,
Department of Energy
Deputy Director,
Exposure
Measurement and
Assessment Division;
Environmental &
Occupational Health
Sciences Institute
Asbestos
measurement methods
Human health risk
assessment
Particulate matter
monitoring and
exposure/ risk
assessment; personal
monitoring and
epidemiology
Characterization,
quantification and
control of toxic air
pollutant emissions
from stationary and
fugitive sources.
Human health rnpacts
of air pollution
Risk assessment
Exposure assessor
(particulates, volatile
organic compounds,
other urban air
pollutants)
Performed limited indoor
dust/asbestos sampling for
Ground Zero Elected Officials
Task Force.
Organized and chaired the
peer review of EPA's
Contaminants of Potential
Concern report.
Principal investigator on project
to assess the health of workers
who were involved in cleanup
and recovery efforts at the
WTC site and Fresh Kills
Landfill.
Consultant to New York City
Law Department relative to
monitoring performed in the
aftermath of the WTC collapse.
Limited involvement in
Columbia University/National
Institute of Environmental
Health Sciences project to
evaluate exposure and
epidemiological data.
None
Co-author on papers about
exposure associated with WTC
disaster based on samples
collected and analyzed wih
funding from National Institute
of Environmental Health
Sciences.
37
-------
Appendix C
Agency Response to Draft Report
October 21,2004
OFFICE OF
RESEARCH AND DEVELOPMENT
MEMORANDUM
SUBJECT: Response to OIG Draft Special Report - World Trade Center Peer Review
FROM:
TO:
Purpose
Paul Oilman /s/ William Farlandfor
Assistant Administrator (8101R)
Kwai-Cheung Chan
Assistant Inspector General for Program Evaluation (2460T)
This memorandum responds to the OIG Draft Special Report: Review of Conflict of
Interest Allegations Pertaining to the Peer Review of EPA's Draft Report, "Exposure and
Human Health Evaluation of Airborne Pollution from the World Trade Center Disaster"
#2003-001585, dated September 15,2004.
Background/Discussion
ORD generally finds flie draft report prepared by the Office of the Inspector General
(OIG) to be reasonable. Although we concur with most of the recommendations, there are
several points which still merit clarification both with respect to the evaluation and specific
recommendations. Detailed comments are attached that we believe will sharpen the quality and
accuracy of the draft report (Attachment 1).
ORD agrees with the primary findings of the draft report that it does not appear the
panel's input was biased and any potential perception of bias or conflict was not so direct and
substantial that any of the panelists should have been excluded from the panel. Specifically, the
draft report found thai the two scientists who had publicly expressed opinions regarding certain
World Trade Center (WTC) hazards do not appear to have had significant biases, and the
specific scientist who was alleged to have a conflict of interest (COI) was found not to have such
a conflict. We agree, however, that a prior business relationship of this latter scientist may have
resulted in a perception of a COI. Further, while it appears that ORD's contractor (Versar) took
reasonable steps to assemble a balanced panel for the review, we agree that we want our
contractors to collect more information relevant to potential COI or bias than was the case here.
ORD's National Center for Environmental Assessment (NCEA) will provide more specific
direction to that effect to its contractors.
38
-------
ORD wants to have the best peer review practices possible for Agency products. With
that end in mind, I will comment on the specific recommendations provided in the draft report.
Recommendation 2-1: Specific Recommendations to the Director, National Center for
Environmental Assessment, concerning peer review procedures and guidance
NCEA agrees with the OIG that the contractor could have gathered more information
about financial ties with parties whose interests might have been affected by the conclusions
presented in the health evaluation document. We agree with the recommendations in Chapter 2
that NCEA should provide written guidance and detailed instructions to its peer review
contractors concerning potential COI and biases.
NCEA has been working on language to articulate more specifically the kinds of information
to be collected by the contractor from potential peer reviewers in regard to employment and
professional affiliations, financial relationships, prior involvement on related issues, and publicly
made statements or a position taken on the subject chemical or topic under review. The Deputy
Director of NCEA recently signed a memorandum with instructions to NCEA staff on this issue
(Attachment 2). This information will provide the contractor the basis to determine whether such
parties might be affected by the outcome of the peer review. Additionally, this information will be
requested for all future NCEA peer review panels unless waived by the Center Director. In the
future, NCEA's contractors will be required to provide a signed certification to the EPA that can be
made public stating there are no unresolved actual or potential COI issues among the panel members
and that the panel is suitably balanced with respect to any actual or apparent bias. We believe this
fulfills Recommendation 2-1.
While not a specific recommendation in the report, I note that public input into the
composition of a peer review panel is not a current requirement. Some bodies such as the EPA
Science Advisory Board (SAB) have adopted that step into their procedures. NCEA may, at
times, provide the public an opportunity for public input to the panel selection. In that
circumstance, the public comments collected by the contractor on the composition of the
proposed panel with respect to expertise, COI and bias may be considered and appropriate
adjustments to the panel may be made.
Recommendation 3-1: Specific Recommendations to the Assistant Administrator for
Research and Development concerning revision of the Peer Review Handbook
We agree that additional steps are needed to ensure that additional attention is given to
the formulation of peer review panels for products anticipated to be highly controversial and/or
address sensitive issues. An overarching recommendation from the OIG draft report is that the
EPA''s Peer Review Handbook be supplemented with additional instructions to address this issue
(see page 22 of draft report). Our plan is to develop additional guidance to address
Recommendation 3-1, "Revise the Peer Review Handbook or issue supplemental guidance -
applicable to peer review of sensitive or controversial issues - that EPA and its contractor should
follow in identifying and addressing potential COI and biases, and for ensuring that panels are
39
-------
balanced as a whole." We do not plan to revise the Peer Review Handbook per se, because we
believe it stands, as written, as a good resource and is not intended to be a "cook book" of
specific types of procedures. Rather, ORD and the Science Advisor believe that, to clarify how
to manage peer reviews involving sensitive or controversial issues, it is more appropriate to use
supplemental guidance and instructions that articulate the procedure(s) for identifying and
addressing potential CO3 and biases as well as for ensuring that panels are balanced as a whole
(e.g., follow and adhere to the guidance and intent of the Handbook and Federal Acquisition
Regulations).
ORD and the Science Advisor view the specific recommendations and detailed
suggestions (particularly Recommendations 3-1 to 3-4) as useful to consider in providing such
supplemental guidance as a means to help ensure "that the Handbook is followed when
conducting peer reviews of highly sensitive or controversial subject matters, and that issues of
potential bias and overall panel balance are sufficiently addressed" (see conclusions page 25 of
draft report). A list of example questions that peer review leaders and contractors might use can
be incorporated into the supplemental instructions. Example questions could be: "What funding
from any present or prior clients (e.g., industry, EPA, others) have you received?" and, "Have
you expressed viewpoints on the issues being reviewed; if so, when and in what forum?" Also,
the instructions can detail specific actions peer review leaders and contractors can take to
address potential CO1, documentation of actions, and public involvement. We believe these
example questions and actions will incorporate the OIG's specific points (Recommendation 3-1,
a.-f.).
Therefore, we request Recommendation 3-1 be revised to read: "EPA should consider
providing further guidance on procedures."
Recommendation 3-2: Specific Recommendations to the Assistant Administrator for
Research and Development for the development of training materials
ORD's Office of Science Policy has conducted peer review training for the Agency based
on the policies and procedures described in the Agency's Peer Review Handbook and guidance
from the Office of Management and Budget (OMB). In addition, there are regular meetings of
the Agency's peer review coordinators to discuss peer review issues. We are currently revising
the training materials and plan to conduct additional training in the future. We request the report
be changed to state that training materials will be "revised" v. "developed."
Recommendation 3-3: Specific Recommendations to the Assistant Administrator for
Research and Development concerning the development of a peer review website
The Agency will develop a peer review website. We agree that the recommended
changes to the existing website information and other changes required to respond to the
proposed OMB guidelines on peer review should be available online.
40
-------
Recommendation 3-4: Consider adoption of forms used by the Science Advisory Board
(SAB) and the National Academy of Sciences (NAS)
The suggestion to use information-collection forms to obtain as much information as
necessary is also very helpful. The Science Policy Council will be asked to adopt an Agency-
wide policy directing its contractors to use the forms developed by the SAB and NAS.
We have also included the ORD corrective action plan to address recommendations made
in the draft report. Since ORD has already completed some of the required actions and identified
milestones for otheis, we believe the final report should state this and be closed upon issuance.
We appreciate the OIG making a careful evaluation of the accusations that were made
about the peer review of the EPA Draft World Trade Center Report. Peer reviews are an
important component of the Agency's quality assurance program, and we take seriously fceir
conduct. Should your staff have questions or require further information, please have them
contact Cheryl Varkalis at 202-564-6688.
Attachments (2)
cc: W. Farland
L. Matthews
K. Dearfield
P. Preuss
M. Lorber
J. Morris
C. Varkalis
41
-------
Appendix D
ORD Corrective Action Plan
OIG Special Report: Review of Conflict of Interest Allegations Pertaining to the Peer
Review of EPA's Draft Report, "Exposure and Human Health Evaluation of Airborne
Pollution from the World Trade Center Disaster"
Assignment No. 2003-001585
September 15, 2004
Rec
#
2-1
3-1
3-2
Recommendation
We recom mend that the Director,
National Center for Environmental
Assessment, implement procedures
to ensure that guidance in EPA's
Peer Review Handbook is followed
in carrying out NCEA peer reviews,
specifically that:
(a) NCEA's peer review contracts,
as well as any EPA peer review
Contractors' consultant agreements
and subcontracts, contain written
guidance concerning potential
conflicts of interest and biases, and
(b) NCEA 's potential panelists are
asked about the industry financing
they have received, and their
relationship with any clients whose
interests might be affected by the
subject being peer reviewed.
Revise the Peer Review Handbook
or issue supplemental guidance -
applicable to peer review of
sensitive or controversial issues -
that EPA and its contractor should
fallow in identifying and addressing
potential conflicts of interest and
biases, and for ensuring that panels
are balanced as a whole.
Develop training materials for peer
review leaders and others involved
in peer reviews for identifying and
addressing potential conflicts of
interest, biases, and ensuring panel
balance, and ensure that peer review
leaders, EPA project managers, and
EPA contractors receive this
training.
Action
Official
Director, NCEA
Director, NCEA
Assistant
Administrator
for Research and
Development
Assistant
Administrator
for Research and
Development
Corrective Action
Memorandum on peer review
conflicts of intere sts distributed to
NCEA staff
Memorandum on peer review
conflicts of interests distributed to
NCEA staff
Develop additional guidance to
supplement the existing Peer
Review Handbook
Revise existing training materials
and augment as necessary to
provide additional information
concerning conflicts of interests.
Due
Date
Completed
10/07/2004
Completed
10/07/2004
6/30/2005
6/30/2005
42
-------
3-3
3-4
For highly sensitive or controversial
peer reviews, consider the
development of procedures for
using the Agency's website to
electronically post proposed panel
designs, and obtain public
comments on proposed panel
designs prior to panel formation, in
a manner similar to that presently
employed by the Science Advisory
Board.
in regard to the overriding need to
obtain as much information as
necessary to provide reasonable
assurance that potential panelists
will be impartial, consider the
information-collection forms that
are used by the Science Advisory
Board and the National Academy of
Sciences.
Assistant
Administrator
for Research and
Development
Assistant
Administrator
for Research and
Development
Develop a Peer Review Website
to post peer review guidance and
information on specific panels
Request Science Policy Council to
adopt an Agency-wide policy
directing its contractors to use the
forms developed by the SAB and
NAS.
6/30/2005
6/30/2005
43
-------
Appendix E
NCEA Policy Announcement
OCX -- 7 2004
Policy Announcement- NCEA-EM-05-O1
Subject: Avoiding Peer Review Conflicts of Interest (COT)
From: George W. Alapas
Deputy Director for Management
National Center For Environmental Assessment
OFFICE OF
RESEARCH AND DEVELOPMENT
To:
Purpose
Staff Directors
Division Directors
This memorandum implements a new approach related to identification and resolution of
Conflict of Interest (COI) for external Peer Review Panelists. The affected extramural vehicles
are NCEA's peer review contracts Support for Peer Review and Risk Assessment Guidelines
Activities and NCEA's Interagency Agreement with the Department of Energy/ORISE IRIS
Technical Support for Human Health Risk Assessment, Peer Review and Related Activities.
Effective Date: Immediately
Background
To assure that peer reviewers are providing impartial advice on the scientific merits of
NCEA products, NCEA is incorporating specific COI provisions into Task Order (TO)
Statements of Work (SOW) for peer reviews. These provisions will serve as a "tool" to aid the
contractor in identifying an actual or potential COI that might impair the objectivity of peer
reviewers.
44
-------
Policy Statement
Panel peer reviews procured through NCEA's peer review contracts Support for Peer
Review and Risk Assessment Guidelines Activities and NCEA's IAG with Department of
Energy/ORlSE IRIS Technical Support for Human Health Risk Assessment, Peer Review and
Related Activities must include the attached COI requirements in the TO SOW, unless the
requirement is waived by the NCEA Director.
Guidance
Each TO procuring peer reviewers shall include the Conflict of Interest Analysis and
Certification (Attachment 1) and Revised Conflict of Interest Section (Attachment 1 a). For
independent Government estimating purposes, 40 labor hours at a Pt3 or comparable rate is
allocated to accomplish this task. This estimate may be adjusted once information on the actual
hours incurred is obtained from the contractor. In addition to the mandatory COI task, standard
language has been developed for purposes of providing Opportunity for Public Input to Panel
Selection (Attachment 2). This is an optional task to be included in the SOW at the discretion of
the TO Project Officer.
Please address questions concerning implementation of this policy to Karen Summers,
NCEA's Extramural Management Specialist, at (202) 564-4453.
cc: Associate Directors
Deputy Division Directors
Branch Chiefs
Attachments (3)
45
-------
Attachment 1
Task #. - Conflict of Interest Analysis and Certification (Mandatory)
a. Prior to selecting expert panelists, the contractor shall perform an evaluation to determine
the existence of an actual or potential COI for each proposed panel member. The
contractor shall incorporate the attached yes/no questions (a.- i.) and requests for
supporting information (j.-r.) into its established process to evaluate and determine the
presence of an actual or potential COI. All information provided by the proposed panel
member shall be verified by the contractor.
b. The contractor shall resolve issues of actual or potential conflicts of interest and panel
composition before assembling the panel. As each situation must be evaluated on a case-
by-case basis after consideration of specific circumstances, the contractor may consult
with the Contract Officer (CO) in carrying out these responsibilities. Consultation
between the contractor and the CO must be documented and provided to the CO to assure
transparency in the process and fill disclosure, if questions arise concerning COI.
c. The contractor shall provide written basis and signed certification that might be made
public for concluding that there are no unresolved actual or potential conflicts of interest
issues among the panel members, and to ensure that the panel is suitably balanced with
respect to any actual or apparent bias.
Deliverable: Written basis
Signed Certification (See TO "Conflict of Interest" Section)
d. The contractor shall require each panel member to provide a signed declaration that the
panel member is not arranging any new professional relationship with, or obtaining new
financial holdings in, an entity which is not yet reported to the contractor or could be
Viewed as related to the topic under discussion and its associated stakeholders.
Deliverable: Panel member signed declaration
e. The contractor shall provide the Project Officer (PO) resumes for all panel members.
These resumes might later be made public.
46
-------
Attachment 1
Deliverable: Resumes
Task #. - Conflict of Interest Analysis and Certification
Questions and Supporting Information
a.
b.
c.
d.
e.
f.
g-
h.
To the best of your knowledge and belief, is there any connection between the
subject chemical or topic and any of your and/or your spouse's compensated or
uncompensated employment, including government service, during the past 24
months? Yes No
To the best of your knowledge and belief, is there any connection between the
subject chemical or topic and any of your and/or your spouse's research support
and project funding, including from any government, during the past 24 months?
Yes No
To the best of your knowledge and belief, is there any connection between the
subject chemical or topic and any consulting by you and/or your spouse, during
the past 24 months? Yes No
To the best of your knowledge and belief, is there any connection between the
subject chemical or topic and any expert witness activity by you and/or your
spouse, during the past 24 months? Yes No
To the best of your knowledge and belief, have you, your spouse, or dependent
child, held in the past 24 months, any financial holdings (excluding well-
diversified mutual funds and holdings, with a value less than $15,000) with any
connection to the subject chemical or topic? Yes No
Have you made any public statements or taken positions on or closely related to
the subject chemical or topic under review? Yes No
Have you had previous involvement with the development of the document (or
review materials) you have been asked to review? Yes No
To the best of your knowledge and belief, is there any other information that
might reasonably raise a question about an actual or potential personal conflict of
interest or bias? Yes No
To the best of your knowledge and belief, is there any financial benefit that might
be gained by you or your spouse as a result of the outcome of this review?
Yes No
47
-------
j- Compensated and non-compensated employment (for panel member and spouse):
list sources of compensated and uncompensated employment, including
government service, for the preceding two years, including a brief description of
Work.
k. Research Funding (for panel member): list sources of research support and project
funding, including from any government, for the preceding two years for which
the panel member sewed as the Principal Investigator, Significant Collaborator,
Project Manager or Director. For panel member's spouse, provide a general
description of research and project activities in the preceding two years.
1. Consulting (for panel member): compensated consulting activities during the
preceding two years, including names of clients if compensation provided 15% or
more of annual compensation. For panel member's spouse, provide a general
description of consulting activities for the preceding two years.
m. Expert witness activities (for panel member): list sources of compensated expert
witness activities and a brief description of each issue and testimony. For panel
member's spouse, provide a genera! description of expert testimony provided in
the preceding 2 years.
n. Assets: Stocks. Bonds. Real Estate. Business. Patents. Trademaiks. and Royalties
(for panel member, spouse and dependent children): specific financial holdings
that collectively had a fair market value greater than $ 15,000 at any time during
the preceding 24-month period (excluding well-diversified mutual funds, money
market funds, treasury bonds and persona] residence).
o. Liabilities (for panel member, spouse and dependent children): liabilities over
$ 10,000 owed at any time in the preceding twelve months (excluding a mortgage
on personal residence, home equity loans, automobile and consumer loans).
p. Public Statements: A brief description of public statement and/or positions on or
closely related to the matter under review by the panel member.
q. Involvement with document under review: A brief description of any previous
involvement of the panel member in the development of the document (or review
materials) the individual has been asked to review.
r. Other potentially relevant information: A brief description of any other
information that might reasonably raise a question about actual or potential
personal conflict of interest or bias.
48
-------
Attachment 1 a
Insert into SOW Under Conflict Of Interest Section
(Mandatory)
1. Certifying describing analysis and conclusion
The contractor shall provide the CO and PO written certification, within the time
specified in the Task # of the SOW after an award, that:
a. The contractor has resolved all conflict of interest issues, either by
eliminating a particular reviewer from the panel or by determining that the
interest will not impair the individual's objectivity nor create an unfair
competitive advantage for any person or organization.
b. The contractor recognizes its continuing obligation to identify and report any
conflicts of interest arising during performance of the task order
c. All personnel who perform work under this task order or relating to this task
order have been informed of their obligation to report any conflict of interest
to the contractor who shall, in turn, report to the contracting officer
2. Ongoing Compliance Review during contract performance
a. The contractor shall require advanced notification from panelist concerning changes
to information disclosed under Task #.a.
b. The contractor shall inform the CO and PO of any change in financial or professional
relationships that may create either an actual or potential conflict of interest or bias
during the period of performance.
c. The contractor shall consult the CO and PO concerning available options in cases
where actual or potential conflict of interest is determined.
3. Disclosure of Information Used in Conflict of Interest Evaluation
The financial and professional information obtained by the contractor as part of the
evaluation to determine existence of actual or potential conflict of interest is considered private
and non-disclosable to EPA or outside entities except as required by law or requested as part of a
formal investigation by the EPA Office of Inspector General, General Accountability Office,
Congressional Committee.
49
-------
Attachment 2
Task # - Opportunity for Public Input to Panel Selection
(Task required only when public comment on a panelist is required)
a. Public opportunity to recommend panel members
As the contractor assembles a proposed panel, Ihe contractor shall review and consider
public recommendations. Public recommendations for panel members will be provided as
identified in the EPA generated Federal Register Notice. This notice will identify how members
of the public can provide information needed to judge the expertise and appropriateness of the
recommended reviewer.
b. Public comment on panel composition
Subsequent to EPA's acceptance of the comprised expertise of the proposed panel
members, the contractor shall make publicly available a description of the proposed panel, the
contractors' statement as to why they judge the panel to be without conflict of interest, or
without significant bias that has been appropriately disclosed and balanced. The contractor shall
invite public comment on the composition of the proposed panel with respect to expertise,
conflict of interest and bias. The contractor shall assemble and consider the comments received
and make any further adjustments appropriate to the panel Copies of any comments received
shall be provided to the EPA project officer.
50
-------
Appendix F
Distribution
EPA Headquarters
Agency Followup Official (2710A)
Agency Followup Coordinator (2724A)
Audit Followup Coordinator, Office of Research and Development (8102A)
Audit Liaison, Office of Research and Development (8102R)
Associate Administrator for Congressional and Intergovernmental Relations (1301 A)
Associate Administrator for Public Affairs (1101 A)
Inspector General (2410)
51
------- |