OFFICE OF INSPECTOR GENERAL
Special Report
         Review of Conflict of Interest Allegations
         Pertaining to the Peer Review of EPA's Draft
         Report, "Exposure and Human Health
         Evaluation of Airborne Pollution from the
         World Trade Center Disaster"
         Report No. 2005-S-00003

         November 4, 2004

-------
Report Contributors:                         Rick Beusse
                                            James Hatfield
                                            Chris Dunlap
                                            Steve Schanamann
                                            Eric Hanger
Abbreviations:
COI         Conflict of Interest
EPA         Environmental Protection Agency
FAR         Federal Acquisition Regulation
GAO         Government Accountability Office (formerly General Accounting Office)
NCEA       National Center for Environmental Assessment
NCP         National Contingency Plan
OIG         Office of Inspector General
SAB         Science Advisory Board
WTC         World Trade Center
Cover photo: EPA map of ambient air monitoring locations in lower Manhattan; developed
             with EPA's EnviroMapper.  Source: http://www.epa.gov/wtc/em/

-------
'"*'"< J>B0t<-C>
                   U.S. Environmental Protection Agency
                   Office of Inspector General

                   At  a  Glance
                                                                                    2005-S-00003
                                                                                 November 4, 2004
Why W« DW THts
We iwrforajed t&U
        (a.) ditemdtte the
thst ttees p«er tevww
           conflicts
b«y« JIB paired their jttdgments
is ifieir peer review of EPA's
disf t assessment of the health
oftSw World Trsd* Ceater
(WTC) towers, aad riorto BJTOwci>*«Wfcnt of tfc*
j)eeT review, regarding health
hazards AttHs WTC **
someJimes be addiessed is by
using abalaticed panel.
contact our Qfftwof
CongrassJonal antf Pobltc
Liaison at (202) 566-2391.

To view a» full report,
cftck on 4*9 following link;
                          Review of Conflict of Interest Allegations Pertaining to
                          the Peer Review of EPA's Draft Report, "Exposure and
                          Human Health Evaluation of Airborne Pollution from the
                          World Trade Center Disaster"
                              What We Found
                          The seven-member peer review panel selected to review EPA's draft report
                          included one panelist with an extensive history ofproviding expert testimony and
                          similar services for defendants in asbestos lawsuits, and two panelists who had
                          made prior public statements regarding the safety of the air around the WTC site,
                          These circumstances provided a basis for the perception that one panelist had a
                          potential conflict of interest and tw o panelists had potential biases that would
                          prevent them from providing impartial input. We found that EPA's Contractor did
                          not inquire whether the three panelists had received funding from industry or had
                          publically expressed viewpoints on the issues to be reviewed. However, our
                          examination ofthe peer review record did not find that the panel's input was biased,
                          nor that perceived biases and conflicts were so "direct and substantial" that any of
                          the panelists should have been excluded from the panel.

                          In our view additional measures should have been taken during the peer review
                          selection process to disclose the information about panel members upon which the
                          allegations of conflicts of interest or bias were later made. Specifically, had EPA's
                          Contractor inquired about  industry funding, learned whether panelists had expressed
                          opinions publically about the peer review issues, and taken other actions to identify
                          the bases for these perceived conflicts and biases prior to the selection ofthe panel,
                          EPA would have been better able to consider the need for additional actions to
                          balance the panel or otherwise resolve these perceived, potential conflicts and biases
                          prior to conducting the peer review.
                              What We Recommend
                          We made a number of recommendations to better ensure that guidance in EPA's
                          Peer Review Handbook will be fully followed, including that EPA provide:

                          •  better oversight of peer review contracts to ensure that potential panelists are
                            asked about industry financing and their relationship with clients; and

                          •  supplemental guidance and training of peer review leaders regarding the types of
                            information they may need to obtain about potential panelists' opinions and
                            viewpoints when they assess whether panels are independent and balanced.

                          EPA agreed with our conclusions and recommendations and has either taken
                          actions, or set milestones for completing actions, to address our concerns.

-------
                  UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
          !>.                        WASHINGTON, D.C.  20460

 f .9-1
 \ ••-• '•'' /
   '*** •i»r,
-------
If you or your staff have any questions regarding this report, please contact me at (202) 566-
0827 or Rick Beusse, Director for Program Evaluation - Air Issues, at (919) 541-5747.

Attachment

cc:     Cheryl Varkalis, Audit Liaison, Office of Research and Development
       Matt Lorber, National Center for Environmental Assessment/ORD
       Elizabeth Grossman, Office of Program Evaluation/OIG
                                          -2-

-------
                         Table of Contents
At a Glance
   1   Introduction	  1

             Purpose 	  1
             Background 	  1
             EPA's Guidance for Conducting Peer Reviews 	 2
             Scope and Methodology	  3
             Prior OIG Report  	  4
             Results in Brief	  4
             Recommendations	   6
             Agency Comments and OIG Evaluation	  6

   2   Results of Review of the Six Allegations	7

             Question 1: Did the NCR Require Public Participation in Peer Review
                        Panel Selection?  	  7
             Question 2: Was  NCEA Obligated by Procurement Policy Notice 95-04
                        to Evaluate the Sensitivity of the Work Being Performed?	  8
             Question 3: Was  NCEA Obligated to Ensure that Its Peer Review Contractor
                        Required Potential Panelists to Disclose All Substantial Public
                        Statements and All Business Relationships?	  9
             Question 4: Did the Asbestos Expert Have a Conflct of Interest?  	 13
             Question 5: Did Two Panelists Have Biases Based on Their Prior Stated
                        Conclusions Regarding WTC Health Hazards?  	 17
             Question 6: Did EPA's Contractor Adequately Consider Balance in
                        Selecting the Panel?  	 21
             Conclusions	 22
             Recommendation	 23
             Agency Comments and OIG Evaluation	 23

   3   Observations on Panel Formation and Balance 	  24

             Handbook Guidance on Panel Balance 	 24
             Experts' Comments on EPA's Peer Review Handbook  	24
             Prior OIG Report Discussed Concerns With EPA Efforts
                 to Assess  Peer Reviewers' Independence 	 25
             Increased Federal Attention to Peer Review Panel  Balance	26
             Conclusions 	 27
             Recommendations	 27
             Agency Comments and OIG Evaluation	 28

-------
Appendices
     A     Complainant's Allegation Memorandum ..
     B     Peer Review Panelists and Backgrounds
     C     Agency Response to Draft Report	
     D     ORD Corrective Action Plan	
     E     NCEA Policy Announcement	
     F     Distribution	
30
37
38
42
44
51
                                        -ii-

-------
                                Chapter 1
                                Introduction
Purpose
             The Office of Inspector General (OIG) conducted a review of six allegations
             concerning the peer review that was performed on a draft report, issued by the
             Environmental Protection Agency's (EPA) National Center for Environmental
             Assessment (NCEA), entitled: "Exposure and Human Health Evaluation of
             Airborne Pollution from the World Trade Center Disaster" (Draft Assessment).
             Based on the allegations made by the Complainant, our objectives were to answer
             the following six questions:

                 Does the National Contingency Plan require that the public be given an
                 opportunity to comment on the choice of peer review panelists, or suggest
                 other experts who could represent the views of the community?

                 Was NCEA obligated to evaluate the sensitivity of the work being
                 performed as allegedly required by its guidance for handling conflict of
                 interest  situations that are identified after contract award?

             •    Was NCEA obligated to ensure that its peer review Contractor required
                 potential panelists to disclose all substantial public statements they had made
                 about the World Trade Center (WTC) disaster and all business relationships
                 with industries that could be effected by NCEA's findings on WTC exposure
                 hazards?

             •    Did panelist Dr. Eric Chatfield have a conflict of interest and did his resume
                 lack information which would have underscored this conflict of interest?

             •    Did panelists Dr. Alison Geyh and Dr. Patrick Kinney have a bias because
                 they had publically expressed prejudicial opinions before the peer review
                 process began that "... there were no hazards from WTC exposures"?

                 What steps were taken to ensure that the panel was balanced?

             Appendix A provides more information on the allegations.
Background
             In the aftermath of the collapse of the WTC towers on September 11,2001,
             various government entities initiated numerous air monitoring activities to better
             understand the ongoing impact of emissions at the disaster site. At the request of
             EPA Region 2 in New York City, NCEA officials used the data collected from

-------
             these monitoring activities to conduct an inhalation exposure and human health
             risk assessment for airborne pollution from the WTC disaster. The issue of
             potential health impacts from exposure to airborne pollutants resulting from the
             collapse of the towers and the continuing fires at Ground Zero was a highly
             sensitive and controversial subject.  Accordingly, most government
             pronouncements and data regarding this subject  were and continue to be highly
             scrutinized.

             In December 2002, NCEA issued its Draft Assessment for public comment.
             NCEA officials first obtained a proposal from the National Academy of Sciences
             for the peer review of this Assessment; however, they subsequently decided the
             National Academy of Sciences proposal would take too long and cost too much.
             As such, they decided to obtain the peer review by issuing a task order to Versar,
             Inc. under an existing 5-year contract for peer teview and risk assessment support
             (Contract).1  Versar, in turn, obtained the services of seven peer reviewers by
             issuing seven Work Assignment Authorizations. Versar held a public peer review
             meeting on July 14-15,2003. A summary of the peer review panel's comments
             and meeting is available on EPA's public website.

EPA's  Guidance For Conducting Peer Reviews

             The Agency's guidance concerning peer reviews is contained in EPA's Peer
             Review Handbook, Second Edition, December 2000 (Handbook), which was
             issued by EPA's Science Policy Council.  This guidance applies whether EPA
             obtains peer reviewers directly, or contracts for another entity to obtain
             appropriate peer reviewers. According to the Handbook, the principal reason for
             having a peer review is to ensure the scientific integrity and technical credibility
             of a work product which supports a policy or decision, through a complete,
             independent, objective, and competent review. The peer review process is
             intended to uncover any technical problems or unresolved issues in a preliminary
             work product through the use of independent experts.  Peer review panelists do
             not formulate EPA policy; instead, they provide expert input on technical issues
             which underpin EPA conclusions or policies, such as whether a particular
             sampling procedure is an acceptable method for obtaining data used by EPA
             officials to reach conclusions or formulate policy.

             The Handbook states that selecting an independent peer review panel is crucial to
             an effective peer review, and therefore, EPA should make every effort during
             panel formation to use peer reviewers who do not have any real or perceived bias
             or conflicts of interest and who are completely independent.
       The Contract, entitled NCEA Support for Peer Review and Risk Assessment Guidelines Activities, is a task
order contract which provides forservices from August 1, 2002 through Juty 31, 2007.

-------
Scope and Methodology
             To ensure that we accurately and comprehensively addressed the Complainant's
             concerns, we interviewed the Complainant. We subsequently discussed in detail
             our understanding of toe Complainant's allegations, and reached agreement that
             we had stated the allegations fully and accurately. We requested any additional
             documentation in the Complainant's possession which provided support for the
             allegations, and we reviewed the documentation subsequently provided that
             appeared relevant to the allegations.

             To identify EPA's efforts to ensure that an adequate peer review was carried out,
             we interviewed NCEA's Work Assignment Manager for the subject peer review
             task order and Versar's Senior Project Manager for this order. We requested and
             reviewed all documentation concerning the peer review process and screened
             these documents for applicability to the allegations. We reviewed those
             documents relating to the establishment and selection of the peer review panel
             including, but not limited to, the work assignment authorizations for the peer
             review services, personal conflict of interest statements, resumes, records of
             discussions with panelists, and e-mail correspondence.

             We reviewed EPA's Peer Review Handbook to determine Agency guidance for
             identifying and addressing potential conflicts of interest and bias issues.  We also
             reviewed other government related peer review guidance (such as that used by the
             Science Advisory Board, National Academy of Sciences, or recommended by the
             Office of Management and Budget) to determine how these documents addressed
             the issue of balance in peer review panels. To enhance our understanding of the
             history and development of the Peer Review Handbook, we interviewed one of
             the two principal authors of EPA's Handbook. Similarly, to enhance our
             understanding of the peer review balance and panel formation processes and
             procedures used by the Science Advisory Board, we interviewed the Science
             Advisory Board's Director and an Ethics Official. We also reviewed prior
             General Accounting Office (GAO) and EPA OIG reports to identify any
             previously reported issues relevant to the questions we addressed in this review.

             To assess whether the panel's input to the NCEA Assessment was biased, we
             obtained and reviewed 0istened to) the entire audio tape of the two-day peer
             review meeting.  Further, we had our OIG Certified Industrial Hygienist review
             the draft and final Summary Peer Review Report, which included all written
             comments from the peer review panelists. In addition, we obtained the responses
             of all 7 panelists to 13 questions about their perception of the peer review process,
             as well as detailed responses of 3 panelists to additional questions about the
             specific allegations directed at them. Also, at the request of panelist Dr. Eric
             Chatfield, we met with him and discussed his responses to the specific allegations
             made against him. In addition, our Certified Industrial Hygienist interviewed
             NCEA's Work Assignment Manager concerning the Manager's efforts to

-------
             incorporate the recommendations made by the peer review panel about asbestos in
             NCEA's Final Assessment. However, since this work had not been completed at
             the time our field woik ended, our Certified Industrial Hygienist was not able to
             confirm the extent to which the panel's recommendations about asbestos may or
             may not be included in NCEA's Final Assessment.  Subsequent to the completion
             of our field work, theNCEA Work Assignment Manager indicated that all
             changes recommended by the panel were incorporated into the working draft of
             the Risk Assessment.

             To ensure that we accurately interpreted key criteria regarding these issues, our
             OIG Office of Counsel advised us on the applicability of the National
             Contingency Plan (NCP), Federal Acquisition Regulation, and Agency guidance
             on the peer review process. We performed our review and analysis from October
             2003 through June 2004 in accordance with Government Auditing Standards
             issued by the Comptroller General of the United States.
Prior OIG Report
             EPA's Selection of Peer Reviewers, EPA/OIG, Report No. 1999-P-217, September
             1999. The applicable findings of this report are discussed in Chapter 3.
Results in Brief
             The seven-member peer review panel selected to review NCEA's draft report
             included one panelist with an extensive history of providing expert testimony and
             similar services for defendants in asbestos lawsuits, and two panelists who had
             made prior public statements regarding the safety of the air around the World
             Trade Center (WTC) site. These circumstances provided a basis for the
             perception that one panelist had a potential conflict of interest and two panelists
             had potential biases that would prevent them from providing impartial input.
             However, we did not find that the panel's input was biased, nor that these
             perceived biases and conflicts were so "direct and substantial" that any of the
             panelists should have been excluded from the panel. None of the seven panelists
             had concerns about the way the peer review process was conducted.

             Nonetheless, given the highly sensitive and controversial nature of the subject, in
             our view additional measures should have been taken during the peer review
             selection process to disclose the information about panel members upon which the
             allegations of bias and conflicts were later made. Specifically, had EPA's
             Contractor taken additional actions - such as inquiring about publicly expressed
             viewpoints on the subject matter - to disclose this information prior to the
             selection of the panel, EPA's Contractor could have considered the need to
             balance the panel or otherwise resolve these perceived, potential conflicts and

-------
biases prior to conducting the July 2003 peer review. Regarding the six
allegations, we found that:

     The National Contingency Plan does not require, as alleged, that the pUblic
     be given an opportunity to comment on the choice of peer review panelists,
     or the opportunity to suggest other experts who could represent the views of
     the community.

     Neither EPA nor its Contractor was required by Procurement Policy Notice
     95-04, as alleged, to evaluate the sensitivity of the work being performed
     when evaluating potential post-award conflicts of interest.

 •    EPA's Contractor did not obtain as much information on prospective
     panelists as would be needed to adequately assess potential conflicts of
     interest and biases. Specifically, the Contractor did not inquire as to whether
     the three prospective panelists had received funding from industry, or had
     expressed viewpoints on the issues to be reviewed.

     The asbestos expert did not have a "direct and substantial" conflict of
     interest, as alleged, but there was a basis for the perception of a potential
     conflict of interest.

     The two panelists who had publicly expressed opinions regarding certain
     WTC hazards did not appear to have significant biases, as alleged, because
     the opinions expressed were either sufficiently qualified or not directly
     related to the subject matter to be peer reviewed.

     EPA or its Contractor could have taken additional measures to obtain more
     information on prospective panelists' work experience and viewpoints,
     which would have necessitated additional actions on the part of EPA or the
     Contractor to assess the balance of the panel.

 EPA's Peer Review Handbook provides significant flexibility to the peer review
 leader in forming a panel considering the range and variety of EPA scientific and
 technical work products that are peer reviewed. The Handbook also
 acknowledges that obtaining appropriate expertise may mean that some panelists
 with potential conflicts and biases are nonetheless needed.  We noted that the
 Handbook did not always give sufficient instructions on how to define and treat
 issues such as bias and panel balance, and additional guidance and training of
 peer review leaders on handling sensitive or controversial peer reviews could be
 helpful.

-------
Recommendations

             We are making a number of recommendations to EPA to better ensure that
             guidance in EPA's Peer Review Handbook is Hilly followed, including (1) better
             oversight of peer review contracts to ensure that potential panelists are asked
             about the industry financing they have received, and their relationship with any
             clients whose interests might be affected by the subject being peer reviewed; and
             (2) supplemental guidance and training of peer review leaders regarding the types
             of information they may need to obtain about potential panelists' opinions and
             viewpoints when they assess whether panels are independent and balanced.

Agency Comments and OIG Evaluation

             The Agency agreed with our draft report findings, conclusions, and
             recommendations.  See Appendix C for the Office of Research and
             Development's (ORD) response to our draft report. Additionally,  in response to
             our draft report, on October 7, 2004, ORD issued Policy Announcement #NCEA-
             EM-05-01 which requires that specific conflict of interest (COI) provisions be
             included in task order statements of work for EPA peer reviews, unless these
             provisions are waived by the NCEA Director (see Appendix E for this Policy
             Announcement). These provisions adequately address many of our concerns.
             ORD also identified milestone dates for (a) developing additional  guidance to
             supplement the existing EPA  Peer Review Handbook, (b) revising existing
             training materials, and (c) developing a Peer Review Website. Further, ORD
             stated it would ask the Science Policy Council to adopt an Agency-wide policy
             directing its contractors to use the information-collection forms that are currently
             being used by the EPA Science Advisory Board and the National Academy of
             Sciences to aid staff in assessing peer review panel balance.

             We commend the Office of Research and Development and the National Center
             for Environmental Assessment for the actions already completed, ongoing, and
             planned to ensure that scientific and technical work products consistently receive
             adequate peer reviews. ORD's completed and planned actions are summarized in
             the Corrective Action Plan (see Appendix D). Agency officials also made a
             number of technical comments and clarifications to our draft report, which we
             have included as appropriate.

-------
                               Chapter 2

           Results of Review of the Six Allegations

             A discussion of each allegation and the results of our review follows.

Question 1:,- Did the NCP Require Public Participation In Peer Review
                Pane) Selection?

             The Complainant alleged that the public was not given an opportunity to
             comment on the choice of the seven panelists, or an opportunity to suggest other
             members who could represent the views of the community, as the complainant
             claimed was required by the National Contingency Plan (NCP). We found that
             neither the NCP nor the Peer Review Handbook require public participation in the
             selection of a peer review panel. Although the public did not participate in the
             selection of the panel, the peer review meeting was held in public and the public
             was given the opportunity to  comment on the panel composition and the Draft
             Assessment during this meeting. Although the extent of this participation may
             have been less than  the Complainant or  others may have desired, the extent of the
             participation was consistent with Agency peer review guidelines and prior
             practices.

             The Handbook acknowledges the general principle that obtaining a peer review
             should be a transparent process: "Remember, the Agency is committed to
             working 'as if in a fishbowP  and most of its activities are transparent to the public
             (except where confidential business information is concerned)." The Director of
             EPA's National Center for Environmental Research and Quality Assurance,
             Versar's Senior Project Manager for the peer review of NCEA's Draft
             Assessment, and NCEA's Work Assignment Manager for the peer review told us
             there was no requirement for public participation in the peer review process. We
             confirmed that mis was the case during  our examination of the NCP.  The
             officials further stated that EPA has conducted numerous peer reviews in the past
             without allowing the public to suggest panelists or otherwise participate in the
             selection of panelists. The Director estimated that for every thousand panels EPA
             has convened, "fewer than ten would have public involvement" in the process of
             selecting panelists.

             Although the Handbook does not require the Agency to involve the public in the
             panel selection process, the public was involved in the peer review meeting.
             NCEA's task order to Versar contained  the following requirements with respect
             to public participation:

             •  the peer review meeting would be announced to the public though a Federal
               Register Notice;

-------
             •  the deliberations of the panel would be open to the public;

             •  there would be a 2 to 3 hour period during which the public could address the
                panel with specific comments;

             •  each person who asked to address the panel would be granted 5 minutes to
                address the meeting; and

             •  the public could also submit comments in writing.

             Based on our review of the audio tape and other documentation of the peer review
             meeting, these requirements were fulfilled. For example, several individuals from
             the general public accepted the opportunity to communicate directly with the
             panelists. In addition, NCEA's Draft Assessment was made available to the
             public prior to the meeting and comments were accepted from the public.
             Additionally,  the public was allowed to submit comments after the peer review
             meeting to be included in the peer review summary report. The summary report
             has also been  made available to the public via EPA's public website.

Question 2: Was NCEA Obligated by Procurement Policy Notice
                95-04 to Evaluate the Sensitivity of the Work Being
                Performed?
             The Complainant alleged that EPA did not follow the procedures outlined in
             "EPA Procurement Policy Notice No. 95-04, Procedures for Handling Post
             Award Organizational Conflicts of Interest" to address potential conflicts of
             interest as defined in Federal Acquisition Regulation (FAR) Section 9.501. The
             Complainant alleged that the Procurement Policy Notice provides specific
             questions that EPA should ask contractors to determine whether potential
             conflicts of interest exist, and requires EPA to evaluate the sensitivity of the work
             being performed by tiie Contractor.

             EPA's Procurement Policy Notice No. 95-04 provides guidance for evaluating
             potential organizational conflicts of interest and does not apply to individual
             potential conflicts of interest pertaining to a peer review panelist.  The
             requirements of the Procurement Policy Notice apply only to the government-
             contractor relationship and to potential organizational conflicts of interest that a
             contractor may have related to the work being performed for EPA.

             We also reviewed the Peer Review Handbook for guidance on how potential
             conflicts of interest discovered after the selection of a panel should be addressed.
             The Handbook provides considerable guidance on how to address potential
             conflicts of interest in selecting the panel; however, it does not include guidance
             for handling potential conflicts of interest discovered after the panel had been
             selected.

-------
Question 3:   Was NCEA Obligated ID insure that Us Peer Review
                Contractor Required Potential Panelists to Disclose All
                Substantial Public Statements end AH Business
                Relationships?

             The Complainant alleged that EPA did not require its Contractor to obtain all
             relevant information from potential panelists,  including asking potential panelists
             whether they had made prior public statements about WTC hazards, or to require
             full disclosure of business relationships with industries that would be affected by
             EPA's findings on WTC exposure hazards. We found that EPA's Contractor did
             not obtain all the necessary information on prospective panelists that would be
             needed to adequately assess potential conflicts of interest and biases. For
             example, the Contractor did not inquire as to whether the three prospective
             panelists had received funding from industry,  or had expressed viewpoints on the
             issues to be reviewed.

             EPA Guidance for Using Contractors to Arrange for Peer Review
             Services

             The Handbook includes specific requirements and procedures to be followed
             when EPA uses a contractor to arrange for peer review services. The Handbook
             states that:

                     The Statement of Work (SOW) must clearly specify that the
                    contractor is responsible for preparing peer review evaluations
                    and set forth guidelines for the peer review of scientific or
                    technical documents.  The contractor may perform the peer
                    review with in-house staff, subcontractors, or consultants. Any
                    guidelines for performing peer reviews to ensure soundness and
                    defensibility must be developed by the program office and made
                    pan of the contract. The contractor would then ensure that peer
                    reviews adhere to the guidelines.

             According to the Handbook, the Statement of Work should direct the contractor
             "... to inquire whether prospective peer reviewers have any actual or potential
             organizational or personnel conflicts of interest or other matters that would create
             the appearance of a lack of impartiality, including whether they have had or
             presently have a financial relationship with EPA." According to the Handbook,

                     When the peer review process is being conducted by a
                    contractor, the requirement for addressing peer reviewer's
                    possible conflicts of interest should be highlighted in the
                    Statement of Work of the work ordering instrument (e.g., Work
                    Assignment, Delivery Order, Task Order, etc.) and is a matter
                    that is bound by contractual clauses  with  the Contracting
                     Officer as the final Decision Maker in contracting matters.

-------
When contracting for peer review services, EPA officials cannot select the peer
reviewers. Such action would interfere with the contractor's authority and
responsibility to perform the work under contract. EPA officials can establish the
criteria for the type of individual that might participate on a peer review panel,
but they should avoid commenting on the contractor's selection of peer review
panelists other than to determine whether the panel, once selected, meets the
criteria established.

EPA's Peer Review Guidance does not specifically require EPA - or its
contractor when conducting the peer review for EPA - to specifically ask
panelists about their prior public statements, if any, related to the matters to be
peer reviewed, and no attempt was made to obtain prospective panelists'
viewpoints on the subject matter for this peer review.  However, the Peer Review
Handbook states that a panel should represent a balanced range of technically
legitimate viewpoints, and in general, panelists who have publicly commented on
the subject should be avoided.  Accordingly, it would be difficult to adequately
carry- out the intent of EPA's Peer Review  Handbook without obtaining
information on prospective panelists' stated positions or viewpoints.

NCEA's Actions to Ensure that Potential Conflicts of Interest or
Biases Were Identified

NCEA's contract with Versar identified the actions that Versar was to take to
ensure that potential conflicts of interest or biases were identified prior to
selecting the peer review panel.  Among other things, this contract defined a
conflict of interest and required Versar to include a substantially similar definition
in any subcontract or consultant agreement which provided services for the
contract. NCEA's contract with Versar required Versar to obtain:

        recognized, independent peer-reviewers who will provide
        impartial evaluations of NCEA products . . .[and] peer
        reviewers ., . shall be free of real or perceived conflicts of
        interest, and shall represent a balanced range of technically
        legitimate points of view and disciplinary mix.

NCEA's task order to Versar for the peer review of NCEA's Draft Assessment
contained a statement of work which identified areas of expertise that were
desired for the peer review panelists. However, NCEA's task order did not
specifically discuss conflicts of interest or biases but contained a requirement that
NCEA may, after receiving the list of potential panelists, provide feedback to
Versar concerning "any known conflict of interest, or concern about the
appropriate expertise."
                             10

-------
Contractor's Actions to Identify Potential Conflicts of Interest or
Biases

Versar took actions to identify potential conflicts of interest but did not take all
of the actions that were required by its Contract with NCEA or the actions that
were prescribed in its own internal guidance documents. Versar took the
following principal actions to identify conflicts of interest:

•   screened potential panelists to obtain diversity;
•   discussed conflicts of interest with seven candidates over the telephone
    before their selection (according to  Versar's Senior Project Manager);
•   obtained personal conflict of interest statements from each selected panelist
    before the peer review meeting was held on July 14, 2004;
    obtained resumes from each of the panelists, which should have disclosed
    important working relationships;  and
•   required each panelist to orally discuss or confirm the absence of a conflict of
    interest at the beginning of the peer review meeting.

Versar identified at least 45 potential  panelists before selecting 7 to serve as peer
reviewers.  Six of the seven panelists had performed some work related to the
collapse of the WTC towers. In accordance with the requirements inNCEA's
task order, Versar sought experts with different backgrounds and different areas
of expertise. Versar's intent to get different viewpoints on the panel is illustrated
by the fact that one candidate panelist was "not called" because he was known to
collaborate with someone who  had already been selected, or was likely to be
selected, for the panel.  Further, Versar excluded at least three other prospective
panelists from consideration because  of potential conflicts of interest or their
work relationships.

Contractor Did Not Fully Follow Guidance for Identifying Conflicts

The work assignment authorizations used by Versar to  obtain  the services of peer
review panelists did not define conflicts of interest or require panelists to report
conflicts of interest as required in its contract with NCEA. Further, Versar did
not take all of the actions which were prescribed in its own guidance documents
for identifying potential conflicts of interest. In particular, Versar did not inquire
specifically about industry funding of each panelist's work or each panelist's
working relationships with companies that might have a vested interest in the
results of NCEA's Draft Assessment. Such questions are important in uncovering
potential conflicts of interest.

Versar's generic Conflict of Interest Management Plan required that
subcontractors and consultants (selected panelists) perform "screenings" for
conflicts of interest and report the results of such screenings to Versar. The
screenings were to include, at a minimum, the identification of former and present
clients at the site where work is or has been performed, the type of work
performed, and the period of the performance. However, Versar did not follow
                              11

-------
this guidance. Versar did not ask panelists to identify their clients whose interests
might be affected by conclusions that could be drawn about WTC issues. When
they submitted their resumes and personal conflict of interest statements to
Versar, several panelists identified work they had completed related to the
conditions at the WTC site. However, panelists were selected without specific
questions being asked concerning the possible interest of the panelists' present or
prior clients.

We reviewed a written script that Versar reportedly used to interview potential
panelists over the telephone. The script contained a definition of conflict of
interest which, according to Versar's Senior Project Manager for the peer review,
was read over the phone to each panelist and discussed with them. This definition
embodied the principal concept of objectivity and impartiality that is contained in
EPA's Handbook and NCSA's Contract with Versar. The script also contained
the following questions:

        Are you doing other work at the World Trade Center currently?
        For whom? Other work at the WTC is not necessarily a COI
         (conflict of interest), we just need to be aware and disclose  this
         information.  In fact we see it as a potential asset.

        Are you receiving funding from EPA currently? For what types
         of projects? Anything else to disclose that might be related to
         this review?

Versar's Senior Project Manager's notes, which summarize his telephone
discussions with the seven panelists who were ultimately selected, indicate that he
specifically discussed conflicts of interest with potential panelists. For example,
one panelist, who was not cited in any of the allegations, expressed concern that
he might have a conflict of interest because he mistakenly thought a  paper he had
co-authored concerning the WTC may have  been partially funded by EPA. Aside
from the documentation that indicates conflicts of interest were discussed orally
with potential panelists,  each panelist signed a personal conflict of interest
statement that affirmed they did not have a conflict of interest.

The script did not include the following question that is contained in Versar's
standard operating procedures document entitled: Summary of Versar's Conflict
of Interest Screening and Certification for Peer Reviewers Under NCEA Peer
Review Contracts:

        Are you receiving funding from industry for efforts that might
         be related to this document!

We contacted the Versar Project Manager to determine why this question was not
asked but had not received a response at the time this report was prepared. The
NCEA work assignment manager told us he was satisfied that the panel met the
criteria established tea- this panel.
                              12

-------
Question 4^   Did IN As bestos E Xpert Have a Conflict of Interest?

             The Complainant alleged that one panelist, Dr. Eric Chatfield "... is a paid
             expert witness for W.R. Grace & Co., the firm responsible for the asbestos
             contamination in homes in Libby, Montana..."  and "In an expert report dated
             8/30/02, Chatfield strongly opposed EPA's analytical evaluation of risks in homes
             in Libby."  The Complainant further alleged that the resume that Dr. Chatfield
             submitted to Versar did not disclose this working relationship.  The Complainant
             alleged  that this relationship represented a conflict of interest since W.R. Grace &
             Co. was involved in litigation with the U.S. Department of Justice over
             responsibility for removing asbestos contamination in Libby. We determined that
             the asbestos expert did not have a "direct and substantial" conflict of interest, as
             alleged, but there was a basis for the perception of a potential conflict of interest.

             Handbook Definition of Conflict of Interest and Procedures for
             Handling

             EPA's Peer Review Handbook defines a conflict of interest as follows:

                 Conflict of interest is a situation in which, because of other activities or
                 relationships with other persons, an  individual is unable or potentially
                 unable to render impartial assistance or advice to the Agency, or the
                person's objectivity in performing the work is or might be otherwise
                 impaired, or a person has an unfair competitive advantage.

             Further, the Handbook states that a conflict of interest may exist when a panelist's
             "professional standing and status or the significance of their principal area of
             work might be effected by the outcome of the peer review." The Handbook does
             not specifically define "bias," but notes that the desire to select individuals with
             experience on  a panel often comes from those who are considered as having a
             potential bias.

             The Handbook discusses techniques to help ensure disclosure and appropriate
             resolution of potential conflicts of interest Section 3.4.6 states:

                 One way of identifying conflicts of interest is to ask potential reviewers
                 about current and prior work, and prior clients that might create conflicts
                 or the appearance of a lack of impartiality in carrying out peer review
                 activities... .

             The Handbook also states that each potential conflict of interest situation is
             unique and must be treated on a case-by-case basis. The Handbook  lists certain
             factors that should be considered in evaluating potential conflicts of interest.
             These factors include:

             •   attention to employment, financial, and professional affiliations  of the
                 participants;

                                          13

-------
•   exploring directly the issue with each of the participants before the review
    process takes place;

•   disclosing publicly at the beginning of meetings any previous involvement
    with the issue; and

•   providing non-Federal peer reviewers with a copy of the peer review inquiry
    conflict of interest form, and conducting a follow-up discussion to discuss
    any relevant issues.

The Handbook offers additional guidance on when a person with a potential
conflict of interest can be used on a panel and when such a potential conflict of
interest would result in a person's exclusion from the panel.  According to the
Handbook,

    In fact, experts "with a stake in the outcome - - and therefore a potential
    conflict - - may be some of the most knowledgeable and up-to-date experts
    because they have concrete reasons to maintain their expertise. Such
    experts could be used provided the potential conflicts of interest are
    disclosed and the peer review panel or group being used as a whole is
    balanced. In some cases, however, the conflict may be so direct and
    substantial as to rule out a particular expert...

The Handbook's general definition of conflict of interest that would likely result
in a prospective panelist's exclusion from a panel is as follows:

    Generally, a conflict of interest arises when the person is affected by
    his/her private interests, when he/she or his/her associates would derive
    benefit from incorporation of their point of view  in an Agency product, or
    when their professional standing and status or the significance of their
    principal area of work might be affected by the outcome of the peer
    review.

Those  in charge of panel formation  are encouraged to consult with EPA's Office
of General Counsel "whenever there are questions about conflicts of interest."

Disclosure  of Dr. Chatfield's Work Experience

In regard to the alleged incompleteness of Dr. Chatfield's resume, the resume he
provided to Versar did not specifically disclose his working relationship with
W.R. Grace & Co.  In our view, this relationship should have been disclosed to
the  contractor during the screening process. However, Dr. Chatfield's 30-page,
single-lined resume identified approximately 75 instances where he had acted on
behalf of one or more defendants in litigation by  providing "depositions,"
"certifications," "affidavits," "declarations," or "expert testimony." While not all
of these instances may have involved asbestos, his resume disclosed that he had
been an "expert witness on behalf of several defendants in asbestos property
                              14

-------
damage suits" and "asbestos personal injury suits." Accordingly, it should have
been clear to NCEA and Versar officials, as well as other panelists, that Dr.
Chatfield had extensive working relationships with asbestos companies, such as
W.R. Grace & Co., and other defendants in asbestos cases.

According to the Complainant, Dr. Chatfield's relationship with W.R. Grace &
Co. was significant because as part of its legal defense, W.R. Grace & Co. was
using comparisons of the cleanup standards required in Libby, Montana, to the
cleanup standards being used around the WTC. Specifically, W.R. Grace & Co.
was asserting that it was being held to a higher standard in removing asbestos
contamination from buildings in Libby, even though they contended that the
contamination levels in Libby were not as bad as the contamination levels in
buildings around the WTC.

In regard to the extent of Dr. Chatfield's relationship with W.R. Grace & Co., he
maintains that his work for W.R. Grace & Co. involved minimal time and effort,
involving less than 2 weeks total effort, and was not significant in relation to his
many other activities, especially since he was never called to testify. A brief
chronology of his work for W.R. Grace & Co. follows:

•   Hired by W.R. Grace & Co. on August 6, 2002, forthe purpose of being an
    expert rebuttal witness concerning data that were obtained by EPA using an
    analytical method which Dr. Chatfield developed.

•   Submitted a report dated August 30, 2002, rebutting the analytical work done
    by EPA's Contractor concerning air samples at Libby by transmission
    electron microscopy.  The report documented cases of misidentification of
    fibers and misuse of the analytical method.

•   Deposed by a U. S. Department of Justice attorney on September 29,2002.

    In anticipation of a January 2003 trial, he made one visit to the attorney
    representing W.R. Grace & Co. to discuss this matter.

•   Contractual services for W.R. Grace & Co. terminated in January 2003, when
    the parties settled out of court.

Dr. Chatfield stated that as a rebuttal witness for W.R. Grace & Co., his work was
limited  in that he could only rebut the evidence proffered by others in this matter,
and could not introduce any unsolicited views. Dr. Chatfield also explained that
he had never been a consultant for W.R. Grace & Co., but only an expert witness
hired to provide testimony on data gathered by an analytical method which he
helped to develop in the late  1970s. We did not identify any documentation that
challenged Dr. Chatfield's statements that he did  "... less than two weeks work"
or that he has "... never been a consultant for W.R. Grace either before or after
this limited assignment."  In addition, the  allegations pertaining to Dr.
Chatfield's relationship with W.R. Grace & Co. were raised at the public peer
                             15

-------
review meeting, and were fully disclosed to other panelists at the meeting. The
Peer Review Handbook notes that full disclosure and discussion of potential
conflicts of interest before the entire panel is one method for resolving such
conflicts.

Our review of Dr. Chatfield's past work as detailed in his resume did not identify
any other work relationships between Dr. Chatfield and W.R. Grace & Co.
Additionally, the work relationship between Dr. Chatfield and W.R. Grace & Co.
did not appear to involve significant time and effort.  Thus, this potential conflict
of interest was not sufficiently "direct and substantial," as discussed in EPA's
Peer Review Handbook, to warrant Dr. Chatfield's exclusion from the panel.

Notwithstanding Dr. Chatfield's limited relationship with W.R. Grace & Co., the
fact that he had many working relationships with many asbestos companies over
the years is a basis for a perception that he may have a potential conflict of
interest.  Accordingly, we evaluated his input to the peer review (both oral and
written comments) for any indication of bias.  We did not identify any biased
input. During the peer review meeting and through his written peer review
comments, he called for treating information cautiously. For example, he
recommended that:

    NCEA's Final Assessment state specifically that there are a number of
    potential contaminants which have not been measured and that uncertainty
    remains, and he encouraged governmental officials to identify the limitations
    of their work; and

•   all dust samples be treated as if they contained one half percent to 3 percent
    asbestos, even when a dust sample was reported to be free of asbestos.

Further, Dr. Chatfield  did not comment on the health implications of the data.  He
told us that his area of expertise is asbestos sampling and analytical methodology,
and that he did not feel qualified to address health issues. He told us that he does
not make statements about health effects, but  instead evaluates considerations
such as sampling methodology and data quality.

Also, in a report on asbestos concentrations measured in two apartments near the
WTC site, Dr. Chatfield recommended that nearby residents obtain professional
cleaning of their residences  at a time when the government was advising the
public to clean their own residences with wet rags, wet mops, and vacuums with
HEPA (high efficiency paniculate air) filters as long as there was less than
1 percent asbestos in dust samples.  The positions taken by Dr. Chatfield in his
peer review of NCEA's Draft Assessment did not reflect biases on behalf of any
asbestos company or an asbestos industry that retained his services in the past. In
summary, Dr. Chatfield's relationship with W.R. Grace & Co. did not present a
"direct and substantial" conflict of interest as  alleged, but there was a basis for the
perception of a potential conflict of interest
                              16

-------
5:   Did Two Panelists Have Biases Based ait Their Prior
     Stated Conclusions Regarding WTC Health Hazards?

 The Complainant alleged that two panelists, Dr. Alison Geyh and Dr. Patrick
 Kinney, "... are known to have advised the public through the press that there
 were no hazards from WTC exposures.  Their statements are prejudicial, making
 it difficult for them to change their opinions at this later date to agree with any
 EPA conclusions ... that there are potential or real risks from WTC exposures. "
 The Complainant asserted that potential liability issues for these individuals or
 their parent institutions would make it "difficult if not impossible" to agree with
 any of the draft report's conclusions that differed from their prior public
 statements about the health risks.  We evaluated the public statements attributed
 to Dr. Geyh and Dr. Kirmey to determine whether they demonstrated that these
 panelists had "clearly taken a side" and whether these statements represented a
 "direct and substantial" conflict as discussed in EPA's Handbook. In our opinion,
 these two panelists did not appear to have significant biases because the opinions
 expressed were either sufficiently qualified or not directly related to the subject
 matter to be peer reviewed.

 EPA Guidance on Bias

 The Handbook does not specifically define bias but discusses how a potential bias
 could impact panel formation. In discussing considerations for selecting peer
 reviewers, the Handbook recognizes that a panel can be comprised of individuals
 who are considered to have a potential bias and that this concern can be addressed
 by ensuring that the panel is balanced. The Handbook states that:

     .., the very need to have experienced individuals on a peer review, along
     with the desire to have appropriate technical balance and representation,
     can mean that the selection of potential peer reviewers often comes from
     those who are considered as having a potential bias.  To reduce the
     concern that a potential panel may have unnecessary bias, it may be
     useful  to obtain an informal review of the expertise and balance of
     potential peer reviewers from others in your organization, from OGC or
     even from outside groups....

 The Handbook provides additional discussion on the proper mix of a panel, and
 notes that:

     As a general rule, experts who have made public pronouncements on an
     issue (e.g., those who have clearly 'taken sides') may have difficulty in
     being objective and should be avoided.
                              17

-------
Dr. Geyh's Public Statements

The Complainant provided two news articles that contained quotations attributed
to Dr. Geyh, an assistant professor of environmental health sciences at Johns
Hopkins University Bloomberg School of Public Health. The two news articles
contained her comments about a research study that she directed. The
Complainant has not alleged that Dr. Geyh misinterpreted or misrepresented the
results of her research study.  This study concerned WTC cleanup workers, not
the general public; thus, the focus of her study is different than NCEA's Draft
Assessment. The principal quotations attributed to Dr. Geyh by one news article
were:

    Many of the workers we assessed reported coughing, wheezing, and sore
    throats while working at Ground Zero. These symptoms seemed to
    increase the longer they worked at the site. The gyod news is that we did
    not find unhealthy levels of asbestos but we don't know what the long term
    health risks may be regarding exposure to other airborne contaminants at
    the site.

A second news article reported that Dr. Geyh said there was no sign of lung
obstruction, and quoted her as saying, "For the general community this is a very
good story..."

We provided Dr. Geyh with a copy of the Complainant's allegations about her
prior public statements. Dr. Geyh specifically denied stating there were no
hazards associated with exposures at the WTC site, as alleged. Further, she noted
that her statement about asbestos levels was "based on results of my own airborne
asbestos monitoring data collected directly on the disaster site," and that none of
her research involved residents, office workers, or first responders.  She told us
that:

    / have no data regarding these groups and have made no definitive
    statements or come to any definitive conclusions about the relationship
    between asbestos levels and these groups.

We found that Dr. Geyh's statements reflected the factual results of her research
study. Additionally, during the peer review, she noted the limitations of existing
test data and the need for more data about the potential long-term health risks
associated with exposures at the WTC site. As such, her statements did not
demonstrate that she had "taken sides" with respect to the issues being peer
reviewed. We concluded that any potential biases arising from these statements
were not so material as to warrant her exclusion from the panel. However, the
fact that such opinions were expressed provided a basis for the perception that
there may have been a bias, and supports the need for EPA or its Contractor to
improve their efforts to ensure that peer review panels as a whole are balanced.
                             18

-------
Dr. Kinney's Public Statements

The Complainant provided several news articles that contained quotations
attributed to Dr. Kinney. The principal quotations attributed to him by the news
articles were:

   The data that has been collected so far do not show concentrations of
   pollutants that are of concern .  .  .  Based on what we know so far, there
   is no cause for alarm.  .. . For the general public, as long as we are
   careful, I think the risks are quite low. ... I do not think it was unwise to
   have people start repopulating the area [near the World Trade Center].
   However, not all of the pollutants that you might want to look at have
   been monitored or, at least, reported yet. So the database is incomplete.

We provided Dr. Kinney with a copy of the Complainant's allegations about his
prior public statements. He agreed that he had made these statements, but denied
that it biased his input during the peer review. His written response to us included
the following statement:

   / never feel at all 'prejudiced' by a previous view I have held if new
   information points in a different direction. . .. I'm perfectly willing to
   change my mind on any issue based on new evidence. This is generally
   the way scientists behave.

Our review of Dr. Kinney's written peer review comments indicated that he, in
fact, had advocated alerting the public to potential asbestos risks. He noted:

   // would be good to append a concluding sentence to the effect that these
   data support the conclusion that the WTC collapse resulted in increased
   indoor asbestos levels in lower manhattan.

Dr. Kinney also advocated  that NCEA provide more information in its assessment
and clearly communicate the potential risks to the public. For example, Dr.
Kinney's written comments expressed concern that the public may obtain an
unwarranted sense of comfort from the wording in NCEA's Draft Assessment
regarding occupational benchmarks. He noted that:

    ... the use of a health benchmark in a screening exercise is only valid if
   the health benchmark is appropriate for the population at risk, which I
   think is somewhat problematic here where occupational benchmarks are
   being used to screen risks in the general population. ... somewhere early
   in the executive summary there needs to be a short paragraph explaining
   why occupational standards may not be directly relevant to general
   population health risk assessment, including issues of voluntary and
   compensated risks vs. involuntary risks, healthy workers vs. widely
   varying susceptibilities in general population.  This very important point
   will not be obvious to the general reader.

                             19

-------
Further, Dr. Kinney's oral comments during the peer review meeting July 14-15,
2003, showed his concern about the need for a comprehensive review of indoor
air data and the need for EPA to earn the public's trust by advocating the use of
the National Academy of Science's panel model to obtain an unimpeachable
review of indoor air issues:

   I would like to encourage them [EPA] to consider the advantages of
   having an independent scientific group take the lead on organizing it [an
   indoor air study or compilation of indoor air studies] mainly from the point
   of view that - the credibility of the results would probably be enhanced
   substantially over a purely sort of EPA or government- agency sponsored
   or centered effort.  So, given the controversies that have existed to date, I
   think one way to sort of resolve the issues — these really important issues
   about indoor air and exposure of health — in a way that everybody is
   satisfied with, would be to get a bunch of external people .. . I am sort of
   thinking of the National Academy of Science's panel model, where on
   controversial issues, often government agencies will commission the
   National Academy of Science to form an independent group of scientists to
   go over an issue , . .

We also noted that Dr. Kinney's detailed written peer review comments reiterated
his concern about the incomplete database.  He stated that:
   / believe that the overall conclusions of the evaluation  are reasonable
   given the available data.  .. The  report [NCEA's Draft Assessment] is
   careful to point out that these are preliminary conclusions based on
   available data and that conclusions could change when new information
   becomes available.

Considering the qualification that was contained in the news article, Dr. Kinney's
advocacy of full disclosure, his concern that the public understand the potential
health risks, and the overall professional and constructive nature of the comments
that Dr. Kinney made during the peer review process, his statements did not
demonstrate that he had "taken sides" with respect to the issues being peer
reviewed. We  concluded that any potential biases arising from the statements
cited by the Complainant were not so material as to warrant Dr. Kinney's
exclusion from the panel. However,  the fact that such opinions were expressed
provided a basis for the perception that there may have been a bias, and supports
the need for EPA or its Contractor to improve their efforts to ensure fcat peer
review panels as a whole are balanced.

ORD's Perspective

ORD agreed that Versar could have done more to ensure balance regarding the
viewpoints held by prospective panelists, but ORD emphasized that Versar sought
balance regarding "scientific  expertise," and ORD further stated that:
                             20

-------
                 For this highly visible World Trade Center Assessment, the contractor, at
                 EPA's direction, sought panelists who were familiar with the complex
                 issues presented by this unique environmental disaster, even to the point
                 of seeking out scientists who had conducted related research of their own
                 on World Trade Center exposure and health impacts. The two scientists
                 who had made the public comments noted in the complaint have both
                 conducted World Trade Center exposure and health impact studies that
                 ORD considers to be of high quality, and their input to the review spoke of
                 their familiarity and expertise with the issues.  In cases such as this, a
                 strict adherence to a policy of avoiding scientists who have spoken on the
                 topic being reviewed may not be warranted.

Question 6;   Did EPA's Contractor Adequate ly Consider Balance in
                 Selecting the Panel?

             In expressing concerns about potential conflicts and biases with the panel, the
             Complainant noted that a wide divergence of opinions could be accommodated in
             a large balanced panel of expert scientists, but had concerns about a panel that
             only consisted of seven members for this peer review. In light of the allegations
             that certain panel members may have been biased in their points of view, we
             reviewed Versar's efforts to ensure that the panel was balanced.  We believe EPA
             or its Contractor could have taken additional measures to obtain more information
             on prospective panelists' work experience and viewpoints, which would have
             necessitated additional actions on the part of EPA or the Contractor to assess the
             balance of the panel. For example, as noted in Section 4, Versar did not ask
             panelists to identify their present or prior clients whose interests might be affected
             by conclusions that could be drawn about WTC issues.

             EPA Guidance Concerning Panel Balance

             According to EPA's Peer Review Handbook,"the peer reviewers of a work
             product should represent a balanced range of technically legitimate points of
             view." The Handbook also points out that "cultural diversity and 'address' (e.g.,
             industrial, academic, or environmental community) are other factors that can play
             a role in selecting peer reviewers."

             The Handbook does not provide specific examples of inquiries that should be
             made to ascertain whether a panel "as a whole" is balanced with respect to
             viewpoints, but notes lhat established peer review groups such as EPA's Science
             Advisory Board (SAB) provide useful models for addressing balance and conflict
             of interest issues. As noted in a recent GAO report,2 when selecting SAB
             panelists EPA specifically asks prospective panelists for information about their
             points of view related to the issue being reviewed.
       7Federal Advisory Committees.'Additional Guidance Could Help Agencies Better Ensure Independence
and Balance, U.S. General Accounting Office, Rpt. No. GAO-04-328, April 2004.

                                          21

-------
             Contractor's Efforts To Balance The Panel

             The statement of work in NCEA's's task order to Versar identified areas of
             expertise that were "desired" for the peer review panel. As required by EPA,
             Versar selected the 7 panelists from these areas.  Although Versar took several
             steps to balance the panel, we saw no evidence of any special actions or
             considerations by Versar to ensure that the peer review panel for the NCEA Draft
             Assessment was balanced with respect to the specific allegations of the
             Complainant. As noted earlier, Versar did not inquire as to whether prospective
             panelists had publicly expressed viewpoints on the peer review subject matter.
             Accordingly, no consideration was given to the need to balance the panel with
             respect to publicly expressed viewpoints on the subject matter. We contacted the
             Versar Project Manager to determine whether other actions were taken to balance
             the panel, but we had not received a response at the time this report was prepared.

             Panelists' Perspective on the Need To Balance WTC Draft
             Assessment Panel

             We contacted each of the panelists to obtain their opinions on the balance of the
             WTC Draft Assessment Panel  as a whole. Each panelist expressed his or her
             opinion that the panel, as a whole:

             •   contained the right mix of expertise,

                 had an appropriate balance of ideas and opinions,

             •   examined the issues with an appropriate degree of balance, and

             •   allowed the expression of minority views.

             None of the seven panelists had concerns  about the way the peer review process
             was conducted.  As shown in Appendix B, the peer review panelists' backgrounds
             included a variety of employers and technical expertise.
Conclusions
             Additional measures could have been taken to identify potential biases and
             conflicts with prospective panel members. For example, EPA could have had its
             Contractor ask prospective peer review panelists if they had publicly expressed
             viewpoints - a procedure used in selecting EPA Science Advisory Board
             committees - regarding the issues to be peer reviewed.  Further, EPA's
             Contractor did not take all the actions required by its contract with EPA, nor did
             the Contractor take all the actions prescribed in its own internal guidance
             documents.  Had such measures been taken, it is likely that EPA's Contractor
             would have disclosed potential conflicts and perceived biases prior to formulation
             of the panel. As such, the Contractor could have identified the need to assess the
                                          22

-------
             balance of the panel and taken steps to mitigate these perceived conflicts or
             biases.

             Neither the statements made by the two panelists nor the work relationships of
             one panelist were so "direct and substantial" as to warrant their exclusion from
             the panel. Further, ourreview of the panel's deliberations, comments, and peer
             review record did not disclose any biased input in the peer review process, and
             none of the panelists had concerns about the way the peer review process was
             conducted. However, in selecting future panels for sensitive or controversial
             issues, EPA and its contractors should take steps to identify prospective panelists'
             viewpoints on the subject being reviewed, including prior public statements, and
             evaluate the need to resolve any potential biases disclosed through these actions,
             including whether the viewpoints of the panel should be balanced as a whole.

Recommendation

             2-1    We recommend that the Director, National Center for Environmental
                    Assessment, implement procedures to ensure that guidance in EPA's Peer
                    Review Handbook is followed in carrying outNCEA peer reviews,
                    specifically that:

                    a. NCEA's peer review contracts, as well as any EPA peer review
                       Contractors' consultant agreements and subcontracts, contain written
                       guidance concerning information to consider when evaluating
                       potential conflicts of interest and biases, and

                    b. NCEA's potential panelists are asked about the industry financing they
                       have received, and their relationship with any clients whose interests
                       might be affected by the subject being peer reviewed.

Agency Comments and OIG Evaluation

             The Agency agreed that the contractor should have gathered more information
             from the  prospective panelists and that ORD "should provide written guidance
             and detailed instructions to its peer review contractors concerning potential COI
             and biases." On October 7, 2004, ORD issued Policy Announcement #NCEA-
             EM-05-01, which requires that specific conflict of interest (COI) provisions be
             included  in task order statements of work for peer reviews, unless these
             provisions are waived by the NCEA Director (see Appendix E for this Policy
             Announcement). These provisions adequately address our concerns. ORD did
             not request any changes in the draft report wording of Chapter 2 except for the
             addition  of a few woids to our first recommendation. We made the requested
             change to reflect our agreement with ORD.
                                          23

-------
                               Chapters

      Observations on Panel Formation and Balance

            In addressing the six questions discussed in Chapter 2, we relied heavily upon
            EPA's Peer Review Handbook for guidance and criteria in evaluating EPA's and
            its Contractor's actions in conducting the peer review. The Handbook was
            viewed as the key guidance on how to conduct an EPA peer review. We noted
            that the Handbook did not always give sufficient instructions on how to define
            and treat issues such as bias and panel balance. Accordingly, we contacted EPA
            officials who helped develop the Handbook to better understand the intent of the
            Agency's guidance. Based on our review of the Handbook, our discussion with
            these EPA officials, and the issues discussed in Chapter 2, we believe EPA's
            Handbook should be supplemented with additional instructions on how EPA and
            its contractors should select panels for highly controversial and sensitive issues,
            such as the risk assessment conducted for the WTC Draft Assessment.

Handbook Guidance on Panel Balance

            EPA's Peer Review Handbook says that "As a general rule, experts who have
            made public pronouncements on an issue (e.g., those who have 'taken sides') may
            have difficulty in being objective and should be avoided," and that EPA is
            "strongly encouraged to obtain peer reviewers who do not have a legal or
            perceived conflict of interest." Further, EPA is to exercise an extra degree of care
            in selecting panelists on sensitive or controversial issues, which in our view
            includes EPA's Draft Assessment on exposure and health risks from air pollution
            after the collapse of the WTC towers.  However, EPA's Peer Review Handbook
            also notes that obtaining appropriate expertise may mean that some panelists with
            potential conflicts and biases are nonetheless needed.  For example, the
            Handbook notes that key methods for handling such concerns involve, among
            other things, full disclosure of potential conflicts of interest and ensuring that the
            panel is balanced as a whole.

Experts' Comments on EPA's Peer Review Handbook

            EPA's Peer Review Handbook states that established peer review groips, such as
            EPA's SAB, provide useful models for addressing balance and potential conflict
            of interest issues. As such, we discussed these issues with four knowledgeable
            experts on peer review issues - one of the principal authors of EPA's Handbook,
            the Director of EPA's SAB, an SAB Ethics Official, and a former EPA Office of
            General Counsel Attorney who advised the authors of the Handbook.

            The principal author of the Handbook we contacted told us that, in creating the
            Handbook, the authors had not intended to provide a "cook book" but had,
            instead, intended to provide a generic document that provided significant
                                        24

-------
             flexibility to the peer review leader in forming a panel considering the range and
             variety of EPA scientific and technical work products that are peer reviewed. He
             and the SAB Director noted that the Handbook allows the peer review leader to
             consider the sensitivity or controversial nature of the work product, and the
             importance of the document for Agency decision-making, as well as the cost and
             time required. They noted both cost and time can increase dramatically with
             more elaborate peer reviews.

             In summary, they believed that the existing Peer Review Handbook, if properly
             implemented, generally provides sufficient guidance and flexibility for carrying
             out an effective peer review. The  Handbook's principal author also noted that
             revising the Handbook would require a great deal of effort. Nonetheless, the
             principal author acknowledged that additional guidance for dealing with
             particularly complex issues - such as potential bias and panel balance - could be
             helpful. The Handbook author noted that he would prefer to do this by providing
             peer review leaders with supplemental guidance regarding the types of
             information they may need to obtain about potential panelists' opinions and
             viewpoints when they assess whether panels are independent and balanced.  They
             could also issue a supplemental package of training materials to peer review
             leaders, and ensure mat EPA project managers and EPA contractors receive  this
             training.

             We also discussed ways to mitigate the potential for public perceptions that a
             panelist might have a conflict of interest or a bias. The SAB Ethics Official told
             us that, generally, "the best disinfectant is public involvement." Noting that the
             SAB now publicizes potential candidates on its website, we discussed the
             advantages and disadvantages of this practice for EPA's other peer reviews.  The
             SAB Director and the Ethics Official noted that publicizing relevant biographical
             information about potential panelists on EPA's website for a specific time, such
             as 2 weeks, and soliciting and considering public input concerning the proposed
             composition of a panel during the  panel formation process, would be possible.
             However, it would cost more and take  longer, and some prospective panelists may
             exclude themselves from the candidate pool under this approach, making it even
             more difficult to find qualified peer reviewers. Both the SAB Director and the
             Ethics Official noted that considering public input on panel formation could be
             worthwhile for the more sensitive  and  controversial peer reviews.

Prior OIG Report Discussed Concerns With EPA Efforts to Assess
Peer Reviewers' Independence

             The OIG last reported3 on EPA's efforts to ensure proper implementation of peer
             reviews in 1999.  The OIG reviewed 32 work products that were peer reviewed -
             14 peer reviews arranged by EPA  program offices, 12 by EPA contractors, and 6
             by the SAB or the Science Advisory Panel. The OIG found that the SAB and
      3£PA 'x Selection of Peer Reviewers, EPA Office of Inspector General, Rpt. No. 1999-P-217, Sept. 1999.

                                          25

-------
             Science Advisory Panel's peer review procedures were effective and being
             followed. However, the DIG identified instances where peer review leaders in
             EPA program offices and contractors did not effectively attempt to determine
             whether conditions existed that may have precluded an independent review.
             Regarding peer reviews contracted by EPA's program offices, the report
             concluded that:

                 . .. EPA program offices should ensure that contract documents  (the basic
                 contract, work assignments and work plans) consistently include specific
                 provisions for contractors to address independence concerns with peer
                 review candidates.  ... we are recommending that ORD (Office of
                 Research and Development) issue supplemental guidance to the
                 Handbook to help identify and resolve independence concerns.

Increased Federal Attention to Peer Review Panel Balance

             The selection of peer review panels for review of government documents has
             received considerable attention and scrutiny recently. In September 2003, the
             Office of Management and Budget's (OMB's) Office of Information and
             Regulatory Affairs published a  draft Peer Review Bulletin and, in April 2004,
             OMB issued a revised version based on the numerous comments received. Both
             versions, if approved, would establish minimum standards for when a peer review
             is required for "scientific information" and the types  of peer review that should be
             considered by Federal agencies in different circumstances. The need for OMB
             guidance on conducting peer reviews is based, in part, on a 1999 GAO report
             which found variability in both the definition and implementation of peer reviews
             across Federal agencies.4  The OMB Bulletin specifically emphasizes
             consideration of expertise and balance when selecting peer reviewers. The
             Bulletin states that:

                 Reviewers should also be selected to represent a  diversity of scientific
                 opinions relevant to the subject. On most controversial issues, there exists
                 a range of respected scientific viewpoints regarding the interpretation of
                 the literature. Inviting reviewers with competing views on the science may
                 lead to a sharper, more focused peer review.

             With respect to reviewers who are not Federal employees, the draft Bulletin states
             that agencies should "adopt or adapt" the prevailing practices of the National
             Academy of Sciences regarding committee composition, conflicts, and balance.
             Potential candidates for National  Academy of Sciences' peer review committees
             are required to complete a "Background Information and Confidential Conflict of
             Interest Disclosure" form. Among other things, this  form asks that prospective
             panelists list their relevant articles, testimony, speeches, etc., and the relevant
       ^Federal Research: Peer Review Practices at Federal Agencies Vary, U.S. General Accounting Office, Rpt.
No. GAO/RCED-99-99, March 1999.

                                          26

-------
             positions of any organization or group with which the candidate is closely
             identified or associated. The form defines "relevant" as information

                 . .. related to - and might be reasonably of interest to others concerning
                 -your knowledge, experience, and personal perspectives regarding the
                 subject matter and issues to be addressed by the committee activity for
                 which this form is being prepared.

             An April 2004 GAO report5 commented on the lack of guidance to assist agencies
             in ensuring better independence and balance in Federal advisory committees, and
             provides useful information with respect to assessing conflict of interest and
             balance questions in other peer review efforts.  For example, GAO reported that
             General Services Administration guidance on advisory committee management
             does not address the types of information that could be helpful to agencies in
             assessing the points of view of potential committee members, nor do agency
             procedures identify the information that should be collected about potential
             members to make decisions about committee balance.
Conclusions
             Given the importance of peer review in maintaining the integrity of EPA's
             various scientific and technical work products, EPA should take steps to ensure
             that the Handbook is followed when conducting peer reviews of highly sensitive
             or controversial subject matters, and that issues of potential bias and overall panel
             balance are sufficiently addressed. Two such actions would involve (1) issuing
             supplemental guidance for peer review leaders on the types of information they
             may need to obtain about potential panelists' opinions and viewpoints when they
             assess whether panels are independent and balanced, and (2) ensuring that peer
             review leaders receive training on this supplemental guidance.
Recommendations
             We recommend that the Assistant Administrator for Research and Development,
             as the Agency Science Advisor:

             3-1    Revise the Peer Review Handbook or issue supplemental guidance -
                    applicable to peer review of sensitive or controversial issues - that EPA
                    and its contractor should follow in identifying and addressing potential
                    conflicts of interest and biases, and for ensuring that panels are balanced
                    as a whole.  Among other things, this guidance should include procedures
                    for:
       Federal Advisory Committees: Additional Guidance Could Help Agencies Be tier Ensure Independence
and Balance, U.S. General Accounting Office, Rpt. No. GAO-04-328, April 2004.

                                          27

-------
                     a.  explicitly determining whether prospective panelists have viewpoints
                        on the issues to be peer reviewed and, if so, whether these viewpoints
                        have been expressed publicly;
                     b.  using general guidelines to assist peer review leaders in determining
                        whether a panel is balanced as a whole;
                     c.  handling potential conflicts of interests and biases that are identified
                        after the panel has been selected;
                     d.  ensuring that contracts for peer review services reference the Peer
                        Review Handbook as criteria to be used by the contractor in selecting
                        panelists;
                     e.  documenting the basis or rationale for final panel formation prior to
                        commencement of the peer review, particularly the actions taken to
                        address any potential conflicts of interest, biases, and to ensure panel
                        balance; and
                     f.  identifying the extent to which the public should be involved in the
                        peer review process and, in particular, under what circumstances the
                        Peer Review Leader should consider soliciting input from the public
                        on the potential makeup of the peer review panel.

              3-2    Revise the training materials for peer review leaders and others involved
                     in peer reviews for identifying and addressing potential conflicts of
                     interest and biases, and ensuring panel balance; and ensure that peer
                     review leaders, EPA project managers, and EPA contractors receive fliis
                     training.

              3-3  For highly sensitive or controversial peer reviews, consider the
                     development of procedures for using the Agency's website to
                     electronically post proposed panel designs, and obtain public comments
                     on proposed panel designs prior to panel formation, in a manner similar to
                     that presently employed by the Science Advisory Board.

              3-4    In regard to the overriding need to obtain as much information as
                     necessary to provide reasonable assurance that potential panelist will be
                     impartial, consider the information-collection forms that are used by the
                     Science Advisory Board and the National Academy of Sciences (these
                     forms can be found at http://www.nas.edu/coysi-coi  FORM-2.pdf and at
                     http:tfwww.epa.gov/sabybdf/eoafQrrn3l 10-48.pdf).
Agency Comments and OIG Evaluation

              The Agency agreed with our recommendations, noting that "... additional steps
              are needed to ensure that additional attention is given to the formulation of peer
              review panels ...."  Agency officials also stated that they plan to develop
                                           28

-------
"supplemental guidance and instructions that articulate the procedure(s) for
identifying and addressing potential COI  and biases as well as for ensuring that
panels are balanced as a whole .... " The Agency also agreed that Versar could
have done more to ensure balance regarding the viewpoints held by potential
panelists, but pointed out that NCEA emphasized that Versar sought balance
regarding "scientific expertise."  NCEA suggested wording be added to our report
to this effect, which we included in its entirety at the end of our discussion of the
fifth allegation. We also clarified one recommendation in this chapter, as
requested by ORD, to recognize  that ORD already provided training and training
materials for peer review leaders and others involved in peer reviews, but that the
Agency would be developing additional training emphasizing identifying and
addressing potential conflicts of interest and biases, and ensuring panel balance.
The actions already completed, ongoing, and planned to ensure that scientific and
technical work products consistently receive adequate peer review, if properly
implemented, address our concerns.
                             29

-------
                                                              Appendix A
Complainant's Allegation Memorandum
      •
            UW1ED STATEBBWRONBEKTAL PROTECTION MOW
                      WttHNGTCHDJC.SMW
               Jiily 16,2003
 •VBJCCTt   Cmrillel of Int
                                    Bvtf uM«ii «f Alih
                IMWMillTlmd*
 TO:
            Otatf Kern

     funHufidd.GiriiDuilap,
f ifcfckiwlK. 5*P* !
 HE EPAIUumd <3™» Cm BB
 udMattHida Enh^maf AnteaiBBalUoDtem«M World IMde Center DUamV
     DMobcr,ZM2,nilMied to Aepublk inlM Decdnbir, ta (BidMHQbgpMr
 pueliih.
 (ciccttMi. Bowmw,aiiKpMivri«wf«^kichiibivily7pi(luiptMf. Itc public «• not
 ChftB my o^portiaiky b caoEDBitaoftc chadixaftheiG iadindndU, OTIO
 »«gt( itotermeiDbtii ivb»«oald Kprwat «h« viow»«f do «nxuoeltytm
 (Monti COnihiuuty Pin.
    i.tflk*U.&B»fa
                               i n turn «f tf* ubtr ud A run MMUKOr idM
                              30

-------
EPA has failed to insure that the government acts with impartiality and provide the public with
full opportunity for participation in die risk assessment process for the WTC disaster. In my
7/4/03 report and memorandum to the EPA Inspector General," evidence was provided of
EPA's similar failure to prevent conflicts of interest in expert panelists chosen for the EPA/IRIS
asbestos reassessment.

EPA did not require their contractor, Versar, to obtain all relevant information from potential
panelists, including asking specific questions whether potential panelists had ever stated
conclusions that there were no hazards from WTC exposures, or to require the submission of full
disclosures of all business relationships with industries that would be affected by EPA findings
on WTC exposure hazards.

The Federal Acquisition Regulations Section 9.501 define conflicts of interest of governmental
contractors as follows:

               "... because of other activities or relationships with other persons, a person is unable or
               potentially unable to render impartial assistance or advice to the Government, or the
               person's objectivity in performing the contract work is or might be otherwise impaired, or a
               person  has an unfair competitive advantage."

EPA has issued specific guidance on evaluating and  handling post contact award conflict of
interest situations in Procurement Policy Notice No.  95 - 04, Proced ures for Handling Post
Award Organizational Conflicts of Interest (COI), dated 9/20/95, posted on the EPA website at
http://www.epa.gov/oam/ptod/COI/ppn95-04.pdf.  Specific questions are posed for EPA to
inquire of contractors to determine potential conflicts. COI criteria also include EPA evaluations
of the sensitivity of the work being performed by the contractor. EPA implemented none of
these procedures to control COI in the expert peer review panel for the NCEA risk assessment.

Eric Chatfield, expert witness for W. R. Grace asbestos  company

One  member of EPA's  peer review panel for the NCEA risk assessment is Eric Chatfield. He  is
a paid expert witness forW.R. Grace & Co., the firm responsible for the asbestos contamination
in homes in Libby, Montana,  the Superfund site. Litigation has been brought against Grace by
the U.S. Department of Justice.

In an expert report dated 8/30/02, Chatfield strongly opposed EPA's analytical evaluation of risks in
homes in Libby. A copy of his expert report is attached to this memorandum, which reads in part as
follows:
          Jenkins, C. (1/4/03) Comments on the EPA Office of Inspector General's 1/27/03 interim report titled:
 EPA's Response to the World Trade Center Towers Collapse — A DOCUMENTARY BASIS FOR LITIGATION.
This report is available at the following 2 websites:
 www.NYenviroLAW.org (see banner at top of page) and http://www.nycosh.org/tfanchorl5430

                                                31

-------
                I have reviewed the action memoranda for the Lbby site, the administrative record for the
                period may 23, 2000 to June 4, 2002, the expert reports I have also reviewed the
                ATSDR Health Consultation dated December 12, 2000, a Report to the Community dated
                August 23,2001, and a report entitled "Health Risks from Exposure to Zonolite Home
                Insulation: Critical Evaluation  of the Scientific Evidence Available from the ATSDR
                Studies in Libby, Montana.
                • • •
                I have determined that there ana significant deficiencies in the underlying analytical data
                that form the basis for the risk assessments conducted by EPA. I have also determined
                that some analytical methods have not been used h accordance with the specifications of
                the methods, and  that some of the analytical methods used are not capable of providing a
                reliable foundation for the decisions being made.

                [Report No.  02C029, Expert Report of Eric J Chatfield, Ph.D., in the matter of: United
                States v. WR. Grace & Co., et al., Civil Action No. 01-72-M-DWM (D.Mont.), 8/30/02]

On 7/14/03, Chatfield is reported as telling the participants in the peer review meeting for the
EPA NCEA report that he did not think it was important to mention his work for Grace in the
biography he provided to meeting participants, and that his work for Grace was minimal. He is
incorrect on both points.

First, his work for Grace is not minimal or  in the past. Providing an "expert report" to a
defendant in active litigation is just one step in the services provided by an expert witness."* In
order for the expert report to be of any  use to Grace, the expert also m ust make himself available
to the litigant, in this case the Dept, of Justice, for deposition. Otherwise, the expert report
would not be admissible as evidence. The expert witness would  also need to  be available to
testify in court in the event  that the case went to trial.

Second, his business connection with Grace is extremely relevant to the WTC investigation.
This is because Grace has been using for its legal defense comparisons between EPA's stringent
cleanup standards for homes in Libby compared to the lenient standards  EPA has imposed for
the WTC  cleanup. Grace is also using in  its defense the fact that homes in Libby are less
contaminated with asbestos than buildings  in Manhattan after Ihe WTC collapse.

Thus, Grace has a vested interest in having EPA conclude (by way of the NCEA human health assessment
document) that citizens in NYC are not at risk from asbestos exposures. They can
then directly use these EPA conclusions to escape responsibility for the contamination in Libby.
            As a note for the reccrd, on 7/14/03, Chatfield told the meeting participants that it was 1 who was
seeking a copy of his export report through a Freedom of Information Act request, and that he would have gladly
supplied me with a copy if 1 had written. This also is untrue, as it was a concerned parent from (he Brookfield
school system, Kathy Huke, who sought this report and obtained it from EPA through the FO1A. Neither the citizen
nor 1 knew of the existence of the expert report until it was provided by EPA. The FO1A request was broad and
included all writings by Chatfield submitted to EPA regarding WR Grace. Furthermore, on 5/30/02 I wrote to
Chatfield for clarification of claims he made to the same Ms. Hulce regarding the Brookfield school system and
received no reply, so it would not have appeared fruitful for me to have made any new inquiries directly to Chatfield.

                                                 32

-------
The following are two letters (attached) from Grace to EPA, written in the same time frame as
Chatfield's expert report, which illustrate this point.

The first letter, below, attempts to gain more lenient cleanup standards forLibby based on the
fact that EPA is imposing less stringent asbestos air and dust standards in New York City after
the WTC collapse:

                The purpose of tris letter is to continue Grace's dialogue wth EPA regarding Zonolite
                Attic insulation ("ZAI"), a home insulation product consisting of expanded vermiculite. For
                your background, ZAI has been insulating homes for over 60 years and there is no
                credible reason to believe that ZAI has ever caused an asbestos-related disease in
                anyone who has used it in his/her home. Nevertheless, EPA Region 8 is requesting that
                EPA declare, for the first time ever, that a "pubic health emergency exists at a
                (proposed) Superfund site.
                • • •
                Contrasted to Region 8's disregard of established norms, EPA's pronouncements and
                activities regarding the World Trade Center collapse reaffirm those norms. Thus, EPA's
                website reiterates that:

                  —  EPA is using the 1% definition of an asbestos-containing material in evaluating
                      dust and bulk samples.

                  —  Air samples are the accurate measure of actual exposure potential, whereas the
                      presence of asbestos in dust is not necessarily a significant health hazard.

                 —  EPA uses the "extremely stringent" standard under AHERA of 70 structures/sq.
                      mm. to evaluate the risk from asbestos in the outdoor and indoor air. That
                      standard was expressly chosen because it is the most stringent and protective,
                      and is based on assumptions of long-term exposure.

                 —   Asbestos exposure becomes a health concern when high concentrations of
                      asbestos fibers are inhaled over a long period of time. The fact that a small
                      percentage of samples exceeded the AHERA standard was not considered to be
                      evidence of a significant health risk.

                We believe that sound science dictates that the same peer reviewed methodologies for
                assessing risks at the WTC should be applied across the board, including in LJbby,
                Montana.

                [emphasis in original) (letter to Christine Tcdd Whitman, Administrator, EPA, from William M.
                Corcoran, Vice President, W. A. Grace & Co., dated 4/10/021


In the next letter, WR Grace compares asbestos contamination levels inLibby to those in NYC
after the WTC collapse. WR Grace states that residences in Manhattan are more contaminated
than in Libby (which is true). This is also offered as a rationale for not cleaning up Libby to the
more stringent standards imposed by Region 8, since the cleanup  in M anhattan is rudimentary
and slipshod in comparison to that being required for Libby.
                                                  33

-------
        We appreciate the opportunity you gave us on January 31 to present Grace's view of the
        data from Libby, Montana that EPA is using to guide its decision regarding removal of
        Zonolite Attic Insulation from homes in that community.

               I would like to summarize several of the points that were made in the meeting.

               Based on the data we have seen, there are very low levels of asbestos in surface dust in
               homes in Libby, Montana. There is only chrysotile found on windowsills, which is a key
               collection point. In all surfaces In Libby, chrysolite fibers exceed am phi bole fibers by 3X
               to 30X. In fact, asbestos concentrations In dust in Libby homes is significantly lower than
               that found in New York City before 9/11, is far below that found after 9/11, and is
               significantly lower than the concentrations that EPA found in studies of carpets, ceilings,
               and lights.

               [letter to Michael Shapiro,  Prhcipal Deputy Assistant Administrator, Office of Solid Waste
               and Emergency Response, EPA, from William M. Corcoran, Vice President, WR. Grace
               & Co., dated 2/6/02]
Prejudicial statements by other panelists


Two other panelists are known to have advised die public through the press that there were no
hazards from WTC exposures. Their statements are prejudicial, making it difficult for them to
change their opinions at this later date to agree with any EPA conclusions, no matter what the
data, that there are potential or real risks from WT C exposures.

There would not be the same kind of legal liability problem for a panelist and their parent
institutions if the panelist had earlier advised the public to take precautions over and beyond
those recommended by public officials at the time. Such a person could easily state that new
evidence demonstrated that such precautions were unnecessary. They would not now be in a
liability situation of ha ving failed to warn the public in the past, puffing them at risk.
Prejudicial statements by Patrick Kinney


One panel member, Patrick Kinney, has gone on record to state that tests had conclusively
demonstrated that asbestos concentrations as well as the other measured toxic constituents were
not of concern.


It would be difficult if not impossible for Kinney to now agree with any conclusions by EPA that
there are potential risks, because that would constitute a reversal of earlier advice Kinney offered
to the public through the press on the safety of returning home and being exposed to
uncontrolled WTC dust. Any findings or recommendations on the NCEA  draft by Kinney would
always be influenced by his earlier "on record" comments below. There may be many other
places where Kinney made such statements as well.
                                                34

-------
                More than six weeks after the September 11 terrorist attack on New York, fires are still
                smoldering under the debris of the former World Trade Center complex. The continuing
                smoke is worrying some city residents about the quality of the air they are breathing.

                City officials have tried to reassure New Yorkers, ordering a series of on-going tests to
                monitor air quality. Patrick Wnney specializes in the effects of air quaity on respiratory
                systems at Columbia Universitys School of Public Health. He savs the tests show there
                are no toxic pollutants, like asbestos, tainting the quality pf air. "The data that has been
                collected so far do not show concentrations of pollutants that are of concern." adds
                Professor Kinnev. "However, not all of the pollutants that you might want to look at have
                been monitored or, at least, reported yet. So the database is incomplete. Based on what
                we know so far, there is no cause for alarm."

                But Professor Kinney says it is unlikely that new data will show any pollutants that put
                people at risk. "I do not think it was unwise to have people start repopulatina the area." he
                says. "I think that oir noses are very sensitive and we pick up a lot of things that are not
                necessarily toxic. I think it is good to be careful, to minimize exposure as much as
                possible. For the general pubic, as long as we  are careful, f think the risks are quite low."

                [emphasis added] [NY Residents Worry About AirQuatity, Barbara Schoetzau, New York,
                10K8/01. http://www.help-for-you.com/news/Oct2001/Ocf28/PRT28-38ArticlB.Mml]
Prejudicial statements by Allison Gcvh

The peer review panelist Alison Geyh has also made definitive conclusions that asbestos testing
showed no harmful levels, and that there were no lung obstructive airway problems for citizens.
As with the panelist Kinney, it would be difficult for Geyh from a liability standpoint to now
agree with any EPA conclusions indicating a hazard or compromised health risk.

                Investigators from the Johns Hopkins Bloomberg School of Public Health now believe that
                World Trade Center workers' respiratory problems are the result of exposure to dust and
                airborne contaminants at Ground Zero. Workers assigned to clear debris have reported
                coughing, wheezing and sore throats while working at Ground Zero. The symptoms
                seemed to increase the longer they worked at the site, according to Dr. Alison Geyh, the
                chief investigator and assistant professor of environmental health sciences at Johns
                Hopkins. "The good news is that we dd not find unhealthy levels of asbestos, but we don't
                know what the long term health risks may be regarding exposure to other airborne
                contaminants at the site."


                [emphasis added] World Trade Center Health Update Summarized by Cheryl Runyon from
                the Environment News Service AmariScan, Aug. 23, 2002.
                http://www.ncsl.org/programs/esnr/wtchealth.htm]
                In a study of more than 180 cleanup workers, researchers at Johns Hopkins Bloomberg
                School of Public Health found that acute respiratory symptoms such as coughs, phlegm
                and wheezing were more prevalent than before they began working at the site. But in most
                workers, said Dr. Alison  Gevh. the lead researcher, there was no sign of lung obstruction.
                                               35

-------
Tor the general community this is a very good story," Geyh said.

       [Researchers tell of health studies tied to WTC disaster; By Caii Gtassman, 11/2/01,
       Online News From The Tribeca Trib http.//www.tribecatrib.com/newsnov02/enviro2r html ]
PeterPreuss, Director, EPA ORD NCEA
Lester Grant Director, EPA ORD NCEA-RTP subdivision
Affected parties and other resp onsible o fficials
                                     36

-------
                                  Appendix B



Peer Review Panelists and Backgrounds
P*S*«*t Tttfc Expertise WTC Exp*riW*
Eric J. Chatfield,
Ph.D.
Michael Dourson,
Ph.D.
Alison Geyh,
Ph.D.
Gary Hunt,
M.S.
Patrick Kinney,
Sc.D.
Margaret
MacDonell, Ph.D.
Clifford P. Weisel,
Ph.D.
President,
Chatfield Technical
Consulting Limited
Director,
Toxicology Excellence
for Risk Assessment
Assistant Professor,
John Hopkins
University Bloomberg
School of Public
Health
Vice President of Air
Toxics Programs and
Director of Air Toxics
Monitoring,
TRC Environmental
Corporation
Associate Professor,
Mailman School of
Public Health at
Columbia University
Argonne National
Laboratory,
Department of Energy
Deputy Director,
Exposure
Measurement and
Assessment Division;
Environmental &
Occupational Health
Sciences Institute
Asbestos
measurement methods
Human health risk
assessment
Particulate matter
monitoring and
exposure/ risk
assessment; personal
monitoring and
epidemiology
Characterization,
quantification and
control of toxic air
pollutant emissions
from stationary and
fugitive sources.
Human health rnpacts
of air pollution
Risk assessment
Exposure assessor
(particulates, volatile
organic compounds,
other urban air
pollutants)
Performed limited indoor
dust/asbestos sampling for
Ground Zero Elected Officials
Task Force.
Organized and chaired the
peer review of EPA's
Contaminants of Potential
Concern report.
Principal investigator on project
to assess the health of workers
who were involved in cleanup
and recovery efforts at the
WTC site and Fresh Kills
Landfill.
Consultant to New York City
Law Department relative to
monitoring performed in the
aftermath of the WTC collapse.
Limited involvement in
Columbia University/National
Institute of Environmental
Health Sciences project to
evaluate exposure and
epidemiological data.
None
Co-author on papers about
exposure associated with WTC
disaster based on samples
collected and analyzed wih
funding from National Institute
of Environmental Health
Sciences.
                  37

-------
                                                                     Appendix C
                Agency Response to Draft Report
                                  October 21,2004
                                                                        OFFICE OF
                                                                  RESEARCH AND DEVELOPMENT
MEMORANDUM

SUBJECT:   Response to OIG Draft Special Report - World Trade Center Peer Review
FROM:


TO:


Purpose
Paul Oilman  /s/ William Farlandfor
Assistant Administrator (8101R)

Kwai-Cheung Chan
Assistant Inspector General for Program Evaluation (2460T)
     This memorandum responds to the OIG Draft Special Report: Review of Conflict of
Interest Allegations Pertaining to the Peer Review of EPA's Draft Report, "Exposure and
Human Health Evaluation of Airborne Pollution from the World Trade Center Disaster"
#2003-001585, dated September 15,2004.

Background/Discussion

       ORD generally finds flie draft report prepared by the Office of the Inspector General
(OIG) to be reasonable. Although we concur with most of the recommendations, there are
several points which still merit clarification both with respect to the evaluation and specific
recommendations. Detailed comments are attached that we believe will sharpen the quality and
accuracy of the draft report (Attachment 1).
       ORD agrees with the primary findings of the draft report that it does not appear the
panel's input was biased and any potential perception of bias or conflict was not so direct and
substantial that any of the panelists should have been excluded from the panel.  Specifically, the
draft report found thai the two scientists who had publicly expressed opinions regarding certain
World Trade Center (WTC) hazards do not appear to have had significant biases, and the
specific scientist who was alleged to have a conflict of interest (COI) was found not to have such
a conflict.  We agree, however, that a prior business relationship of this latter scientist may have
resulted in a perception of a COI. Further, while it appears that ORD's contractor (Versar) took
reasonable steps to assemble a balanced panel for the review, we agree that we want our
contractors to collect more information relevant to potential COI or bias than was the case here.
ORD's National Center for Environmental Assessment (NCEA) will provide more specific
direction to that effect to its contractors.
                                         38

-------
       ORD wants to have the best peer review practices possible for Agency products.  With
that end in mind, I will comment on the specific recommendations provided in the draft report.


Recommendation 2-1: Specific Recommendations to the Director, National Center for
Environmental Assessment, concerning peer review procedures and guidance
       NCEA agrees with the OIG that the contractor could have gathered more information
about financial ties with parties whose interests might have been affected by the conclusions
presented in the health evaluation document. We agree with the recommendations in Chapter 2
that NCEA should provide written guidance and detailed instructions to its peer review
contractors concerning potential COI  and biases.
       NCEA has been working on language to articulate more specifically the kinds of information
to be collected by the contractor from potential peer reviewers in regard to employment and
professional affiliations, financial relationships, prior involvement on related issues, and publicly
made statements or a position taken on the subject chemical or topic under review.  The Deputy
Director of NCEA recently signed a memorandum with instructions to NCEA staff on this issue
(Attachment 2).  This information will provide the contractor the basis to determine whether such
parties might be affected by the outcome of the peer review. Additionally, this  information will be
requested for all future NCEA peer review panels unless waived by the Center Director.  In the
future, NCEA's contractors will be required to provide a signed certification to  the EPA that can be
made public stating there are no unresolved actual or potential COI issues among the panel members
and that the panel is suitably balanced with respect to any actual or apparent bias. We believe this
fulfills Recommendation 2-1.
       While not a specific recommendation in the report, I note that public input into the
composition of a peer review  panel is not a current requirement.  Some bodies such as the EPA
Science Advisory Board (SAB) have  adopted that step into their procedures.  NCEA may, at
times, provide the public an opportunity for public input to the panel selection.  In that
circumstance, the public comments collected by the contractor on the composition of the
proposed panel with respect to expertise, COI and bias may be considered and appropriate
adjustments to the panel may  be made.

Recommendation 3-1: Specific Recommendations to the Assistant Administrator for
Research and Development  concerning  revision of the Peer Review Handbook

       We agree that additional steps are needed to ensure that additional attention is given to
the formulation of peer review panels for products anticipated to be highly controversial and/or
address sensitive issues. An overarching recommendation from the OIG draft report is that the
EPA''s Peer Review Handbook be supplemented with additional instructions to  address this issue
(see page 22 of draft report).  Our plan is to develop additional guidance to address
Recommendation 3-1, "Revise the Peer Review Handbook or issue supplemental guidance -
applicable to peer review of sensitive or controversial  issues - that EPA and its contractor should
follow in identifying and addressing potential COI and biases, and  for  ensuring that panels are

                                          39

-------
balanced as a whole."  We do not plan to revise the Peer Review Handbook per se, because we
believe it stands, as written, as a good resource and is not intended to be a "cook book" of
specific types of procedures. Rather, ORD and the Science Advisor believe that, to clarify how
to manage peer reviews involving sensitive or controversial issues, it is more appropriate to use
supplemental guidance and instructions that articulate the procedure(s) for identifying and
addressing potential CO3 and biases as well as for ensuring that panels are balanced as a whole
(e.g., follow and adhere to the guidance and intent of the Handbook and Federal Acquisition
Regulations).

       ORD and the Science Advisor view the specific recommendations and detailed
suggestions (particularly Recommendations 3-1 to 3-4) as useful to consider in providing such
supplemental guidance as a means to help ensure "that the Handbook is followed when
conducting peer reviews of highly sensitive or controversial subject matters, and that issues of
potential bias and overall panel balance are sufficiently addressed" (see conclusions page 25 of
draft report). A list of example questions that peer review leaders and contractors might use can
be incorporated into the supplemental instructions. Example questions could be: "What funding
from any present or prior clients (e.g., industry, EPA, others) have you received?" and, "Have
you expressed viewpoints on the issues being reviewed; if so, when and in what forum?"  Also,
the instructions can detail specific actions peer review leaders and contractors can take to
address potential CO1, documentation of actions, and public involvement. We believe these
example questions and actions will incorporate the OIG's specific points (Recommendation 3-1,
a.-f.).

       Therefore, we request Recommendation 3-1 be revised to read:  "EPA should consider
providing further guidance on procedures."

Recommendation 3-2: Specific Recommendations to the Assistant Administrator for
Research and Development for the development of training materials

       ORD's Office of Science Policy has conducted peer review training for the Agency based
on the policies and procedures described in the Agency's Peer Review Handbook and guidance
from the Office of Management and Budget (OMB).  In addition, there  are regular meetings of
the Agency's peer review coordinators to discuss peer review issues. We are currently revising
the training materials and plan to conduct additional training in the future. We request the report
be changed to state that training materials will be "revised" v. "developed."

Recommendation 3-3: Specific Recommendations to the Assistant Administrator for
Research and Development concerning the development of a peer review website

       The Agency will develop a peer review website. We agree that the recommended
changes to the existing website information and other changes required to respond to the
proposed OMB guidelines on peer review should be available online.
                                          40

-------
Recommendation 3-4:  Consider adoption of forms used by the Science Advisory Board
(SAB) and the National Academy of Sciences (NAS)
      The suggestion to use information-collection forms to obtain as much information as
necessary is also very helpful. The Science Policy Council will be asked to adopt an Agency-
wide policy directing its contractors to use the forms developed by the SAB and NAS.


      We have also included the ORD corrective action plan to address recommendations made
in the draft report. Since ORD has already completed some of the required actions and identified
milestones for otheis, we believe the final report should state this and be closed upon issuance.

      We appreciate the OIG making a careful evaluation of the accusations that were made
about the peer review of the EPA Draft World Trade Center Report. Peer reviews are an
important component of the Agency's quality assurance program, and we take seriously fceir
conduct. Should your staff have questions or require further information, please have them
contact Cheryl Varkalis at 202-564-6688.

Attachments (2)

cc:  W. Farland
    L. Matthews
    K. Dearfield
    P. Preuss
    M. Lorber
    J. Morris
    C. Varkalis
                                         41

-------
                                                          Appendix D
                ORD Corrective Action Plan
OIG Special Report: Review of Conflict of Interest Allegations Pertaining to the Peer
Review of EPA's Draft Report, "Exposure and Human Health Evaluation of Airborne
              Pollution from the World Trade Center Disaster"
                       Assignment No. 2003-001585
                          September 15, 2004
Rec
#
2-1


















3-1







3-2








Recommendation

We recom mend that the Director,
National Center for Environmental
Assessment, implement procedures
to ensure that guidance in EPA's
Peer Review Handbook is followed
in carrying out NCEA peer reviews,
specifically that:
(a) NCEA's peer review contracts,
as well as any EPA peer review
Contractors' consultant agreements
and subcontracts, contain written
guidance concerning potential
conflicts of interest and biases, and
(b) NCEA 's potential panelists are
asked about the industry financing
they have received, and their
relationship with any clients whose
interests might be affected by the
subject being peer reviewed.
Revise the Peer Review Handbook
or issue supplemental guidance -
applicable to peer review of
sensitive or controversial issues -
that EPA and its contractor should
fallow in identifying and addressing
potential conflicts of interest and
biases, and for ensuring that panels
are balanced as a whole.
Develop training materials for peer
review leaders and others involved
in peer reviews for identifying and
addressing potential conflicts of
interest, biases, and ensuring panel
balance, and ensure that peer review
leaders, EPA project managers, and
EPA contractors receive this
training.
Action
Official







Director, NCEA






Director, NCEA




Assistant
Administrator
for Research and
Development





Assistant
Administrator
for Research and
Development





Corrective Action








Memorandum on peer review
conflicts of intere sts distributed to
NCEA staff




Memorandum on peer review
conflicts of interests distributed to
NCEA staff


Develop additional guidance to
supplement the existing Peer
Review Handbook






Revise existing training materials
and augment as necessary to
provide additional information
concerning conflicts of interests.





Due
Date







Completed
10/07/2004





Completed
10/07/2004



6/30/2005







6/30/2005








                                 42

-------
3-3
3-4
For highly sensitive or controversial
peer reviews, consider the
development of procedures for
using the Agency's website to
electronically post proposed panel
designs, and obtain public
comments on proposed panel
designs prior to panel formation, in
a manner similar to that presently
employed by the Science Advisory
Board.
in regard to the overriding need to
obtain as much information as
necessary to provide reasonable
assurance that potential panelists
will be impartial, consider the
information-collection forms that
are used by the Science Advisory
Board and the National Academy of
Sciences.
Assistant
Administrator
for Research and
Development
Assistant
Administrator
for Research and
Development
Develop a Peer Review Website
to post peer review guidance and
information on specific panels
Request Science Policy Council to
adopt an Agency-wide policy
directing its contractors to use the
forms developed by the SAB and
NAS.
6/30/2005
6/30/2005
43

-------
                                                                   Appendix E
                   NCEA Policy Announcement
                                  OCX -- 7 2004



Policy Announcement- NCEA-EM-05-O1

Subject:     Avoiding Peer Review Conflicts of Interest (COT)

From:       George W. Alapas
            Deputy Director for Management
            National Center For Environmental Assessment
                                                        OFFICE OF
                                                 RESEARCH AND DEVELOPMENT
To:
Purpose
Staff Directors
Division Directors
      This memorandum implements a new approach related to identification and resolution of
Conflict of Interest (COI) for external Peer Review Panelists. The affected extramural vehicles
are NCEA's peer review contracts Support for Peer Review and Risk Assessment Guidelines
Activities and NCEA's Interagency Agreement with the Department of Energy/ORISE IRIS
Technical Support for Human Health Risk Assessment, Peer Review and Related Activities.

Effective Date: Immediately

Background

        To assure that peer reviewers are providing impartial advice on the scientific merits of
   NCEA products, NCEA is incorporating specific COI provisions into Task Order (TO)
Statements of Work (SOW) for peer reviews. These provisions will serve as a "tool" to aid the
contractor in identifying an actual or potential COI that might impair the objectivity of peer
reviewers.
                                       44

-------
Policy Statement

       Panel peer reviews procured through NCEA's peer review contracts Support for Peer
Review and Risk Assessment Guidelines Activities and NCEA's IAG with Department of
Energy/ORlSE IRIS Technical Support for Human Health Risk Assessment, Peer Review and
Related Activities must include the attached COI requirements in the TO SOW, unless the
requirement is waived by the NCEA Director.
Guidance

       Each TO procuring peer reviewers shall include the Conflict of Interest Analysis and
Certification (Attachment 1) and Revised Conflict of Interest Section (Attachment 1 a).  For
independent Government estimating purposes, 40 labor hours at a Pt3 or comparable rate is
allocated to accomplish this task. This estimate may be adjusted once information on the actual
hours incurred is obtained from the contractor. In addition to the mandatory COI task, standard
language has been developed for purposes of providing Opportunity for Public Input to Panel
Selection (Attachment 2). This is an optional task to be included in the SOW at the discretion of
the TO Project Officer.


       Please address questions concerning implementation of this policy to Karen Summers,
NCEA's Extramural Management Specialist, at (202) 564-4453.
cc:     Associate Directors
       Deputy Division Directors
       Branch Chiefs
Attachments (3)
                                         45

-------
                                                                            Attachment 1
Task #. - Conflict of Interest Analysis and Certification (Mandatory)

a.      Prior to selecting expert panelists, the contractor shall perform an evaluation to determine
       the existence of an actual or potential COI for each proposed panel member.  The
       contractor shall incorporate the attached yes/no questions (a.- i.) and requests for
       supporting information (j.-r.) into its established process to evaluate and determine the
       presence of an actual or potential COI. All information provided by the proposed panel
       member shall be verified by the contractor.

b.     The contractor shall resolve issues of actual or potential conflicts of interest and panel
      composition before assembling the panel. As each situation must be evaluated on a case-
      by-case basis after consideration of specific circumstances, the contractor may consult
      with the Contract Officer (CO) in carrying out these responsibilities. Consultation
      between the contractor and the CO must be documented and provided to the CO to assure
      transparency in the process and fill disclosure, if questions arise concerning COI.

c.     The contractor shall provide written basis and signed certification that might be made
      public for concluding that there are no unresolved actual or potential conflicts of interest
      issues among the panel members,  and to ensure that the panel is suitably balanced with
      respect to any actual or apparent bias.

Deliverable: Written basis
             Signed Certification (See TO  "Conflict of Interest" Section)

d.      The contractor shall require each panel member to provide a signed declaration that the
       panel member is not arranging any new professional relationship with, or obtaining new
       financial holdings in, an entity which is not yet reported to the contractor or could be
       Viewed as related to the topic under discussion and its associated stakeholders.

Deliverable: Panel member signed declaration

e.      The contractor shall provide the Project Officer (PO) resumes for all panel members.
       These resumes might later be made public.
                                           46

-------
                                                                          Attachment 1
Deliverable:   Resumes

Task #. - Conflict of Interest Analysis and Certification
         Questions and Supporting Information
       a.
      b.
      c.
      d.
      e.
       f.


       g-


       h.
 To the best of your knowledge and belief, is there any connection between the
 subject chemical or topic and any of your and/or your spouse's compensated or
 uncompensated employment, including government service, during the past 24
 months? Yes	No	

 To the best of your knowledge and belief, is there any connection between the
 subject chemical or topic and any of your and/or your spouse's research support
 and project funding, including from any government, during the past 24 months?
Yes	No	

 To the best of your knowledge and belief, is there any connection between the
 subject chemical or topic and any consulting by you and/or your spouse, during
 the past 24 months? Yes	No	

 To the best of your knowledge and belief, is there any connection between the
 subject chemical or topic and any expert witness activity by you and/or your
 spouse, during the past 24 months? Yes	No	

 To the best of your knowledge and belief, have you, your spouse,  or dependent
 child, held in the past 24 months, any financial holdings (excluding well-
 diversified mutual funds and holdings, with a value less than $15,000) with any
 connection to the subject chemical or topic? Yes	No	

 Have you made any public statements or taken positions on or closely related to
 the subject chemical or topic under review? Yes	No	

 Have you had previous involvement with the development of the document (or
 review materials) you have been asked to review? Yes	No	

 To the best of your knowledge and belief, is there any other information that
 might reasonably raise a question about an actual or potential personal conflict of
 interest or bias?  Yes	  No  	

 To the best of your knowledge and belief, is there any financial benefit that might
 be gained by you or your spouse as a result of the outcome of this review?
 Yes        No  	
                                          47

-------
j-      Compensated and non-compensated employment (for panel member and spouse):
       list sources of compensated and uncompensated employment, including
       government service, for the preceding two years, including a brief description of
       Work.

k.     Research Funding (for panel member): list sources of research support and project
       funding, including from any government, for the preceding two years for which
       the panel member sewed as the Principal Investigator,  Significant Collaborator,
       Project Manager or Director. For panel member's spouse, provide a general
       description of research and project activities in the preceding two years.

1.      Consulting (for panel member): compensated consulting activities during the
       preceding two years, including names of clients if compensation provided 15% or
       more of annual compensation. For panel member's spouse, provide a general
       description of consulting activities for the preceding two years.

m.     Expert witness activities (for panel member): list sources of compensated expert
       witness activities and a brief description of each  issue  and testimony.  For panel
       member's spouse, provide a genera! description of expert testimony provided in
       the preceding 2 years.

n.     Assets: Stocks. Bonds. Real Estate. Business. Patents.  Trademaiks. and Royalties
       (for panel member, spouse and dependent children): specific financial holdings
       that collectively had a fair market value greater than $ 15,000 at any time during
       the preceding 24-month period (excluding well-diversified mutual funds, money
       market funds, treasury bonds and persona] residence).

o.     Liabilities (for panel member, spouse and dependent children): liabilities over
       $ 10,000 owed at any time in the preceding twelve months (excluding a mortgage
       on personal residence, home equity loans, automobile  and consumer loans).

p.     Public Statements: A brief description of public statement and/or positions on or
       closely related to the matter under review by the panel member.

q.     Involvement with document under review: A brief description of any previous
       involvement of the panel member in the development of the document (or review
       materials) the individual has been asked to review.

r.      Other potentially relevant information: A brief description of any other
       information that might reasonably raise a question about actual or potential
       personal conflict of interest or bias.
                                        48

-------
                                                                            Attachment 1 a
Insert into SOW Under Conflict Of Interest Section
(Mandatory)

1.  Certifying describing analysis and conclusion

          The contractor shall provide the CO and PO written certification, within the time
specified in the Task # of the SOW after an award, that:

       a.    The contractor has resolved all conflict of interest issues, either by
            eliminating a particular reviewer from the panel or by determining that the
            interest will not impair the individual's objectivity nor create an unfair
            competitive advantage for any person or organization.

       b.    The contractor recognizes its continuing obligation to identify and report any
            conflicts of interest arising during performance of the task order

       c.    All personnel who perform work under this task order or relating to this task
            order have been informed of their obligation to report any conflict of interest
            to the contractor who shall, in turn, report to the contracting officer

2.  Ongoing Compliance Review during contract performance

       a.    The contractor shall require advanced notification from panelist concerning changes
            to information disclosed under Task #.a.

       b.    The contractor shall inform the CO and PO of any change in financial or professional
            relationships that may create either an actual or potential conflict of interest or bias
            during the period of performance.

       c.    The contractor shall consult the CO and PO concerning available options in cases
            where actual or potential conflict of interest is determined.

3.  Disclosure of Information Used in Conflict of Interest Evaluation

       The financial and professional information obtained by the contractor as part of the
evaluation to determine existence of actual or potential conflict of interest is considered private
and non-disclosable to EPA or outside entities except as required by law or requested as part of a
formal investigation by the EPA Office of Inspector General, General Accountability Office,
Congressional Committee.
                                            49

-------
                                                                             Attachment 2
Task # - Opportunity for Public Input to Panel Selection
(Task required only when public comment on a panelist is required)
a.  Public opportunity to recommend panel members

       As the contractor assembles a proposed panel, Ihe contractor shall review and consider
public recommendations. Public recommendations for panel members will be provided as
identified in the EPA generated Federal Register Notice. This notice will identify how members
of the public can provide information needed to judge the expertise and appropriateness of the
recommended reviewer.

b.  Public comment on panel composition

       Subsequent to EPA's acceptance of the comprised expertise of the proposed panel
members, the contractor shall make publicly available a description of the proposed panel, the
contractors' statement as to why they judge the panel to be without conflict of interest, or
without significant bias that has been appropriately disclosed and balanced. The contractor shall
invite public comment on the composition of the proposed panel with respect to expertise,
conflict of interest and bias. The contractor shall assemble and consider the comments received
and make any further adjustments appropriate to the panel Copies of any comments received
shall be provided to the EPA project officer.
                                           50

-------
                                                                      Appendix F
                                Distribution
EPA Headquarters
Agency Followup Official (2710A)
Agency Followup Coordinator (2724A)
Audit Followup Coordinator, Office of Research and Development (8102A)
Audit Liaison, Office of Research and Development (8102R)
Associate Administrator for Congressional and Intergovernmental Relations (1301 A)
Associate Administrator for Public Affairs (1101 A)
Inspector General (2410)
                                        51

-------