United States Science Advisory EPA-SAB-EEC-COM-00-001
Environmental Board (1400A) October 1999
Protection Agency Washington DC ivww.epa.gov/sab
v>EPA AN SAB REPORT:
IMPROVING THE EFFICACY
OF SCIENCE ADVISORY
BOARD REPORTS - A
STUDY OF THE
ATTRIBUTES OF
SUCCESSFUL TECHNICAL
REVIEWS
REVIEW BY THE
RETROSPECTIVE
SUBCOMMITTEE OF THE
SCIENCE ADVISORY BOARD'S
ENVIRONMENTAL ENGINEERING
COMMITTEE
-------
October 28, 1999
EPA-SAB-EEC-COM-00-001
Honorable Carol M. Browner
Administrator
U.S. Environmental Protection Agency
Washington, DC 20460
Subject: Improving the Efficacy of Science Advisory Board Reviews: A
Study of the Attributes of Successful Technical Reviews by the
Environmental Engineering Committee
Dear Ms Browner:
One result of the Science Advisory Board's Executive Committee November
1997 retreat was a recommendation that the standing committees review their past
activities and products to develop recommendations for improving SAB reviews. The
Environmental Engineering Committee formed a Subcommittee in 1998 to undertake
this effort and its report on this retrospective study is attached for your consideration.
Moreover, the Subcommittee's findings and recommendations have been transmitted to
the chairs of all the SAB's standing committees.
The Subcommittee intended to develop persuasive, data-based arguments for
changes in the modus operand! within the EEC and, where appropriate, within other
parts of the SAB. These changes would help the SAB to make a more positive
difference in the value of and in the acceptance of SAB advice to the Agency. The
Subcommittee gathered information by examining the impacts of eight of its past
reports of various types to discern why some were more effective than others. The
Subcommittee also interviewed 13 people having extensive knowledge of the SAB
program and products. An additional 22 contacts were made concerning the merits and
influence of specific EEC reports.
The Subcommittee's findings and recommendations are summarized in Chapter
4 of the attached report. The issues addressed are:
Topic Selection and Charge
Going beyond a narrow charge
Developing a receptive clientele for self-initiated reviews
Screening proposed topics for importance
Working closely with Program Offices
-------
Review Process
Careful selection of chairs and subcommittee members
Communicating during a review with stakeholders
Including minority opinions
Report Completion
Planning to meet timeliness requirements
Framing recommendations to maximize value and impact
Prioritizing and categorizing recommendations
Communication and Follow-up
Briefing sponsoring offices
Following up on important reports for longer time periods
We hope the Agency, as well as the Board, the EEC and other committees, will
find these recommendations to be useful. We appreciate the opportunity to conduct
and report on the outcomes of this interesting retrospective study. We look forward to
your response.
Sincerely,
/signed/
Dr. Joan Daisey, Chair
Science Advisory Board
/signed/
Dr. Dr. Hilary Inyang, Chair
Environmental Engineering Committee
Science Advisory Board
/signed/
Dr. Ishwar P. Murarka, Chair
Retrospective Subcommittee
Environmental Engineering Committee
-------
NOTICE
This report has been written as a part of the activities of the Science Advisory
Board, a public advisory group providing extramural scientific information and advice to
the Administrator and other officials of the Environmental Protection Agency. The
Board is structured to provide a balanced, expert assessment of scientific matters
related to problems facing the Agency. This report has not been reviewed for approval
by the Agency; hence, the comments of this report do not necessarily represent the
views and policies of the Environmental Protection Agency or of other Federal
agencies. Any mention of trade names or commercial products does not constitute
endorsement or recommendation for use.
Distribution and Availability : This Science Advisory Board report is provided to the
EPA Administrator, senior Agency management, appropriate program staff, interested
members of the public, and is posted on the SAB website (www.epa.gov/sab).
Information on its availability is also provided in the SAB's monthly newsletter
(Happenings at the Science Advisory Board). Additional copies and further information
are available from the SAB Staff.
-------
ABSTRACT
"Improving the Efficacy of Science Advisory Board Reviews: A Study of the Attributes
of Successful Technical Reviews by the Environmental Engineering Committee" (EPA-
SAB-EEC-COM-00-001)
In response to the suggestion of the SAB Executive Committee, the
Environmental Engineering Committee study of representative products over the past
few years. The aim of the study was to develop persuasive, data-based arguments for
enhancing technical reviews within the EEC and within the SAB in general.
Information was gathered by examining the impacts of eight reports of various
types and discerning why some were effective and others were not. The findings were
strengthened by interviewing 13 people having extensive knowledge of the SAB
program and products; an additional 22 personal contacts were made concerning
specific reports.
Findings and recommendations were developed in four aspects of the review
process, including:
a) Topic Selection and Charge
b) Review Process
c) Report Preparation
d) Impact and Communication
Key Words: Retrospective Study, Improving Efficacy, SAB Reports, Communication ,
Impact, Feedback
-------
U.S. ENVIRONMENTAL PROTECTION AGENCY
Science Advisory Board
Environmental Engineering Committee (FY98-99)
Retrospective Study
CHAIR
Dr. Ishwar P. Murarka , ISH Inc., Cupertino, CA
MEMBERS
Dr. Nina B. French , Sky+, Oakland, CA
Dr. Hilary I. Inyang , University of Massachusetts, Lowell, MA
CONSULTANTS
Mr. Richard Conway , Union Carbide (retired), Charleston, WV
Dr. Raymond C. Loehr, The University of Texas, Austin, TX
Dr. Frederick G. Pohland , University of Pittsburgh, Pittsburgh, PA
Dr. Calvin H. Ward , Rice University, Houston, TX
SCIENCE ADVISORY BOARD STAFF
Mrs. Kathleen W. Conway , Designated Federal Officer
Science Advisory Board (1400A), U.S. EPA, 401 M Street, S.W.,
Washington, DC 20460
Mrs. Mary M. Winston , Management Assistant
Science Advisory Board (1400A), U.S. EPA, 401 M Street, S.W.,
Washington, DC 20460
in
-------
U.S. ENVIRONMENTAL PROTECTION AGENCY
Science Advisory Board
Environmental Engineering Committee (FY99)
CHAIR
Dr. Hilary I. Inyang , University of Massachusetts, Lowell, MA
MEMBERS
Dr. Edgar Berkey, Concurrent Technologies Corporation,
Pittsburgh, PA
Dr. Calvin C. Chien , E. I. DuPont Company, Wilmington, DE
Mr. Terry Foecke, Waste Reduction Institute, St. Paul, MN
Dr. Nina Bergan French , SKY+, Oakland, CA
Dr. Domenico Grasso, University of Connecticut, Storrs, CT
Dr. JoAnn Slama Lighty , University of Utah, Salt Lake City, UT
Dr. John P. Maney, Environmental Measurements Assessment, Hamilton, MA
Dr. Michael J. McFarland , Utah State University, River Heights, UT
Ms. Lynne M. Preslo , Earth Tech, Long Beach, CA
SCIENCE ADVISORY BOARD STAFF
Mrs. Kathleen W. Conway , Designated Federal Officer
Science Advisory Board (1400A), U.S. EPA, 401 M Street, S.W.,
Washington, DC 20460
Mrs. Mary M. Winston , Management Assistant
Science Advisory Board (1400A), U.S. EPA, 401 M Street, S.W.,
Washington, DC 20460
IV
-------
TABLE OF CONTENTS
1. EXECUTIVE SUMMARY 1
2. INTRODUCTION 3
3. DATA COLLECTED 5
3.1 Approach 5
3.2 Data 5
4. FINDINGS AND RECOMMENDATIONS FOR CHANGE 8
4.1 Topic Selection and Charge 8
4.3 Report Completion 9
4.4 Communication and Follow-up 10
TABLE 1 Outcomes and Recommendations for Improving the Efficacy of
SAB Reviews and Reports
TABLE 2 Subjects of EEC Reports Analyzed for Their Impact
TABLE 3 Questions and Respondents in Open-Ended Interviews
APPENDIX A: Synopses of Representative Reports
1. Resolution On the Use of Mathematical Models by EPA for Regulatory
Assessment and Decision-Making
2. Review of Draft Agency Guidance for Conducting External Peer Review of
Environmental Regulatory Modeling
3. Review of MMSOILS Component of the Proposed RIA for the RCRA Corrective
Action Rule.
4. EPA's Environmental Technology Innovation and Commercialization
Enhancement Program (EnTICE)
5. Review of the Waste Incineration Research Program
6. Review of the Global Climate Change Engineering Research and Development
Program
7. Review of a Methodology for Establishing Human Health an Ecologically Based
Exit Criteria for the Hazardous Waste Identification Rule (HWIR)
8. Leachability: Recommendations and Rationale for Analysis of Contaminant
Release
APPENDIX B: Summary of Data from the General Telephone Interviews
-------
1. EXECUTIVE SUMMARY
Recognizing that the approaches to environmental protection at EPA are
changing and that, to be most effective, the SAB needs to change with them, the
Executive Committee held a retreat in November 1997. The resulting Science Advisory
Board's 1997 Strategic Plan (EPA-SAB-98-010) was published in August 1998. The
Executive Committee recommended that the Board spend much more of its total
energies on providing strategic, forward looking advice, while maintaining and
improving the quality, utility, and timeliness of its activities focused on Agency-
requested peer review of EPA products.
Three individuals who have served as Environmental Engineering Committee
chairs participated in the retreat. In early 1998, to support the recommendations, the
Environmental Engineering Committee (EEC) formed a Retrospective Subcommittee to
review that Committee's past activities and products and to develop from the outcomes
an implementing strategy that might apply to the SAB as a whole. The Subcommittee
was formed using both current and former EEC chairs and members.
The Subcommittee approached the charge by developing a body of information
(data) from which findings and recommendations were derived. Thus, such findings
and recommendations would be well-founded and persuasive. In developing the data
base, the Subcommittee examined the impacts of eight reports of various types,
discerned why some were more effective than others, and interviewed 13 people
having extensive knowledge of the SAB programs and products. Subcommittee
members contacted an additional 22 people for opinions concerning the impacts of
specific reports.
Table 1 summarizes the outcomes and recommendations of the Subcommittee.
Chapter 4 of the report presents the recommendations together with the findings upon
which the recommendations were based. The Subcommittee submits the findings and
recommendations to promote continued improvement in the conduct of the review
activities of the SAB and its constituent committees.
-------
Table 1
Outcomes and Recommendations for Improving the Efficacy of SAB Reviews and Reports
A. Topic Selection and Charge
1. The SAB should in most reviews look for issues broader than the charge, and appropriately
address them in a constructive, supportive manner.
2. The SAB should, for most self-initiated studies, select them on the basis of their own
experience and then further cultivate an interested/receptive clientele within the Agency
before pursuing.
3. The SAB should screen proposed topics against criteria that include technical challenge and
high value to the Agency. In other words, choose the topics for which the SAB advice has the
high potential to make a difference.
4. The SAB staff should work with program personnel to develop more candidate reviews that are
important and challenging.
5. The SAB should inform itself about planned Agency efforts and strive to become involved,
when appropriate, at an early stage.
B. Review Process
1. The SAB should pay careful attention to the selection of chairs of Subcommittees. An ideal
chair is committed and capable of producing the advice in a timely and useful manner. The
SAB should also pay careful attention to selection of review committee members, using
consultants more frequently to ensure the breadth of experience and productivity required to
produce high-quality reports.
2. The SAB should ensure that scientifically diverging minority points of view are retained after ful
discussion and are not excluded from the reports.
C. Report Completion
1. Timeliness should be assured by developing a reasonable, agreed upon schedule between SAE
and sponsor, adequate writing time immediately following the public meeting, preparation of
drafts of the whole reports or sections by knowledgeable and committed Subcommittee
members, and frequent exchanges of drafts and communication between chair, members and
SAB staff.
2. The SAB should continue to communicate, as appropriate, with sponsors during all stages of a
review, including providing consensus drafts of the report to the sponsors and other interested
parties within the FACA process.
3. The likely effect of SAB recommendations on EPA programs should be postulated as a check
on the way they are framed. An explicit discussion of the potential impacts of the
recommendations should be encouraged.
4. When recommendations are numerous, varied, or complex, the SAB should simplify their
presentation; categorizing the recommendations and setting priorities are useful ways to do
this.
D. Communication and Follow-up
1. The SAB should first brief the sponsoring office between the last consensus draft and before
final publication, and again engage in dialog with the sponsor after EPA receives the report
and responds to it.
2. The SAB should follow-up important reports that are not acted upon to inquire "what is
happening" and to encourage/support program initiatives in implementing SAB advice.
-------
2. INTRODUCTION
At a retreat in November 1997, the SAB Executive Committee (EC) of the
Science Advisory Board (SAB) considered how the SAB might effectively enhance the
quality of science at EPA in the current era of environmental protection. Overall, the
Executive Committee found that the SAB needs to provide more strategic, forward-
thinking advice and more timely, high-quality peer review of selected topics.
To assist the EC in formulating strategies for improving SAB reviews, the
Environmental Engineering Committee (EEC) formed a Retrospective Subcommittee to
review the EEC's past activities and products, and to develop related recommendations
for improvement. The Subcommittee was charged with determining the attributes of
successful reviews (and problems with reports having lesser impacts) and
recommending steps the EEC could take to increase the likelihood of completing
reviews that have high value to the Agency. It was anticipated that many of these
recommendations may apply to other SAB activities as well.
Current EEC members, past EEC Chairs, and leaders of important EEC studies
served on the Subcommittee. The Subcommittee met twice in mid-1998, the first time
to gather information and the second for analysis and writing. The Subcommittee
recognized that the SAB had considered similar issues in the past. Some examples
are:
a) The 1989 report of the SAB Executive Committee's Mission and Functioning
Subcommittee, chaired by Dr. W. Lowrance.
b) EEC chair, Dr. Ishwar P. Murarka's July 12, 1994 letter to EC chair, Dr.
Genevieve M. Matanoski, "SAB Self-study/Analysis of the EEC".
c) The 1994 report of the Science Advisory Board's Reinvention Committee, The
Science Advisory Board: What's Next? (EPA-SAB-EC-95-008).
d) 1995 report of the Radiation Advisory Committee, A Retrospective Review of
SAB/RAC Activities , (EPA-SAB-RAC-95-009).
The EEC's Retrospective Subcommittee aimed to develop new and more
persuasive arguments for change within the EEC and other parts of the SAB. The
Subcommittee plan reached its goals through:
a) examination of the impacts of eight reports to develop data-based
recommendations rather than opinions;
-------
b) strengthening these recommendations with information gained in interviews of
program personnel and other knowledgeable people; and
c) presentation of a comprehensive set of recommendations for change that can
remind present and future SAB members of ways to improve impact.
-------
3. DATA COLLECTED
3.1 Approach
The Subcommittee used multiple approaches to develop recommendations to
improve the effectiveness of SAB reviews. One approach consisted of selecting and
critically evaluating eight prior EEC reports covering the full spectrum of perceived
impacts on Agency actions; in the process of evaluating these reports, Subcommittee
members spoke with 22 individuals knowledgeable about the individual reports. The
Subcommittee examined these reports for topic origin, charge, subcommittee process,
report content and communication, and impact on the Agency.
The Subcommittee also conducted telephone interviews and communications
with 13 persons familiar with the SAB and its products, although not necessarily the
eight reports evaluated in Appendix B. In addition, the Subcommittee and EEC
discussed these interview issues with Agency staff at the July 22-24, 1998 EEC
meeting.
3.2 Data
Table 2 lists the subjects of the eight EEC reports studied. Appendix A provides
the full title and a synopsis for each report. The synopses are organized by the four
evaluation elements (topic and charge, process, report and communication, impact).
The analyses of these reports, supplemented by the telephone interviews, served as
the basis for the findings and recommendations. The eight reports considered included
two proactive commentaries, two research-in-progress evaluations, one program
evaluation, and reports on three regulatory issues.
Table 3 provides the questions and respondents in the open-ended interviews.
The Subcommittee interviewed 13 people having knowledge of the SAB products.
Appendix B presents the interview questions and results.
The Subcommittee used the results of the report analyses, the interviews, and
the discussions with Agency staff at the EEC's July 1998 meeting to establish the
findings and recommendations presented in this report.
-------
Table 2
Subjects of EEC Reports Analyzed for Their Impact 1
Rl. Mathematical Models for Regulatory Assessment2
R2. Peer Review in Regulatory Modeling
R3. MMSOILS Model for RCRA Regulatory Impact Assessment
R4. Technology and Commercialization Enhancement Program (EnTICE)
R5. Incineration Research
R6. Global Climate Change Research
R7. Hazardous Waste Identification Rule
R8. Leachability as Affects Contaminant Release 2
1 Full report titles are given in Appendix A
2Self-initiated study
-------
Table 3
Questions and Respondents in Open-Ended Interviews
Questions:
What SAB reports were useful to you or had an impact on the Agency
actions?
What made these reports useful or made them have an impact?
Which SAB reports did you find not useful?
What made some SAB reports not useful?
Respondents:
Donald G. Barnes, Staff Director, Science Advisory Board
Dorothy A. Canter, Science Advisor, Office of Solid Waste and Emergency
Response
Donald Clay, formerly of the Office of Toxic Substances and the Office of Air and
Radiation
Jeffrey Denitt, formerly of the Office of Solid Waste
Linda Greer, Natural Resources Defense Council
Clinton Hall, Director, Subsurface Protection and Remediation Division, National
Risk Management Research Laboratory, Office of Research and Development
Penelope M. Hansen , Director, Technology Coordination Staff, National Risk
Management Research Laboratory, Office of Research and Development
Robert Huggett, formerly Assistant Administrator for Research and Development
and former SAB Executive Committee member
Walter W. Kovalick, Jr., Director, Technology Innovation Office, Office of Solid
Waste and Emergency Response
E. Timothy Oppelt, Director, National Risk Management Research Laboratory
Dorothy Patton, Director, Office of Science Policy, Office of Research and
Development and formerly of the Office of General Counsel
Frank Princiotta, Director, Director, Air Pollution Prevention and Control
Division, National Risk Management Research Laboratory, Office of Research
and Development
Michael Shapiro, Principal Deputy Assistant Administrator, Office of Solid Waste
and Emergency Response
NOTE: Appendix A provides the names of those persons interviewed concerning specific reports
-------
4. FINDINGS AND RECOMMENDATIONS FOR CHANGE
Based on the evaluation of the eight reports of various types (including 22
interviews), on the 13 telephone interviews, and discussion at a public meeting, the
Subcommittee formulated the following findings and recommendations for its changes
to improve the efficacy of EEC (and presumably SAB) reviews. The pertinent reports
(R1-8 in Appendix A) and/or general telephone comments (Appendix B) are referenced
after each finding. Recommendations follow each finding.
4.1 Topic Selection and Charge
4.1.1 The Subcommittee finds that, even for reviews responding to a narrow
Agency request, SAB reports can have wider more beneficial impact by proactively
addressing broader issues in a constructive, supportive manner (R2, interviews).
Therefore, in most reviews, the SAB should look for issues broader than the charge,
and then appropriately address them in a constructive and supportive manner.
4.1.2 The Subcommittee finds that self-initiated studies (commentaries and
resolutions) are most worthwhile when:
a) based on broad experience of the SAB;
b) an interested/receptive clientele can be identified or developed; and
c) they address situations that the Agency should change from a science
perspective, can change from a regulatory perspective, and will consider
changing from a policy perspective. (R1 did that; R8 evidently did not).
Therefore, the Subcommittee recommends that, when selecting self-initiated studies,
the SAB should
a) select issues that appear important based on the personal experience of
the Committee members; and
b) cultivate an interested/receptive clientele within the Agency before
pursuing the initiative.
4.I.3 The Subcommittee finds that the SAB contributes best when a topic is
significant in terms of:
a) impact on the Agency;
b) the degree of contentiousness;
c) extent of technical challenge; and
d) importance to top management (R5,6 and interviews).
Therefore, the SAB should screen proposed topics against criteria that include
technical challenge and high value to the Agency. In other words, the SAB should give
-------
priority to activities where its advice has a high probability of making a positive
difference. A transparent screening process should be employed using objective
criteria. Also, the SAB staff should work with program personnel to develop more
requests for reviews that are important, timely and challenging.
4.1.4 The Subcommittee finds that SAB reports have both more immediate and
less disruptive impact when the SAB is involved at an early stage of a regulatory effort
(R2), rather than after completion of an effort (R3,7). Therefore, the Subcommittee
recommends that the SAB committee chairs and/or DFO inform themselves about
Agency activities, both regulatory and non-regulatory, and strive to seek SAB
involvement, when appropriate, at an early stage.
4.2 Committee Review Process
4.2.1 The Subcommittee finds that a SAB Committee more frequently produces
a high-quality review, on time, when a knowledgeable, hard-working chair either drafts
the majority of the report or leads the committee to draft sections of a report (R1,2,5
and interviews). Therefore, the Subcommittee recommends that the SAB select
Subcommittee chairs who are committed to and capable of leading the production of a
report in a timely and useful manner. The SAB should also pay more attention to the
selection of reviewers, using consultants more frequently to ensure the breadth of
experience and expertise required to produce high-quality reports.
4.2.2 The Subcommittee finds that consensus reports are desirable and that
minority views should be included (interviews). Therefore, the SAB should ensure that
scientifically diverging opinions are retained after full discussion and are not excluded
from the reports if consensus is not reached.
4.3 Report Completion
4.3.1 The Subcommittee finds that timeliness and technical content determine
whether a SAB report is useful (R 2,4,5,6 and interviews). Therefore, the
Subcommittee recommends that timeliness should be assured by:
a) developing a reasonable, agreed upon schedule between the SAB and
sponsor;
b) providing adequate writing time immediately following the public meeting;
c) drafting the entire report or sections by one or more knowledgeable and
highly committed Subcommittee members; and
d) frequently exchanging drafts and communication between chair, members
and SAB staff.
-------
4.3.2 Within the structure developed by FACA, the Subcommittee finds that
communication with the sponsor at all stages of the review and report drafting results in
higher impact (R2,5,8 and interviews). Therefore, the Subcommittee recommends that
the SAB continue to communicate, as appropriate, with sponsors during all stages of a
review, including providing consensus drafts of the report to the sponsors and other
interested parties within the FACA process.
4.3.3 The Subcommittee finds that inappropriately critical SAB reports can set
EPA programs back for several years (R3,7); also, if the SAB's recommendations are
too broadly written, they can be impossible to implement (R8). Therefore, the
Subcommittee recommends that review panels postulate some likely effects of the draft
recommendations on EPA programs to determine whether they are written to achieve
the intended impact. This process will also serve as a check on the way the
recommendations are framed.
4.3.4 The Subcommittee finds that the impact of individual recommendations is
diminished when there are too many recommendations that are not organized. Priority,
topic, and timing are often useful ways of organizing the recommendations. Therefore,
the Subcommittee recommends that, when recommendations are numerous, varied, or
complex, the SAB should organize its recommendations in a manner useful to the
intended audience.
4.4 Communication and Follow-up
4.4.1 The Subcommittee finds that some reports do not receive the attention of
higher level program officials (interviews). Therefore, the SAB should engage in dialog
with the sponsoring office between the last consensus draft and final publication, and
then again brief the sponsor and AA (if needed) after EPA receives the report and
responds to it.
4.4.2 The Subcommittee finds that some important SAB reports may lie
dormant, even with Agency supporters, for five to ten years before having a major
impact (R2,6,8 and interviews). Therefore, the SAB should periodically follow-up on
important reports that are not acted upon in order to inquire "what is happening" and to
encourage/support program initiatives that implement SAB advice.
10
-------
APPENDIX A
Synopses of Representative Reports
1. Resolution On the Use of Mathematical Models by EPA for Regulatory
Assessment and Decision-Making (EPA-SAB-EEC-89-012)
Topic Choice and Charge
This report was a self-initiated resolution. This resolution on the use of
mathematical modeling for regulatory assessment and decision-making grew out of the
EEC's experience reviewing modeling studies by several EPA offices. The Committee
noted that certain problems were common to the various modeling efforts and believed
that these common problems would be best called to the Agency's attention through a
more general resolution on modeling.
Subcommittee Review Process
The Chairman of the Subcommittee took the lead in preparing the resolution.
Drafts of the resolution were presented and widely discussed at a series of Committee
and Executive Committee meetings during 1988.
Report and Communication
The seven-page resolution on the use of mathematical models offered eight
main points on seeking better balance between field and laboratory data; obtaining
confirmation of models; establishing an Agency-wide task group to assess and guide
model use; and conducting peer review at various levels. Since the submission of the
modeling resolution, the SAB has transmitted several additional SAB reviews and
commentaries on modeling to various offices of the Agency.
The SAB transmitted the modeling resolution to Administrator Lee M. Thomas
on January 13, 1989. No record of an official response to the SAB resolution is
available. The proponents of the resolution hoped that by drawing this persistent
problem to the Administrator's attention, specific work, such as development of
validated environmental assessment models with integral uncertainty analysis
capability would be undertaken and that these models would be well-documented,
peer-reviewed implementations with software, capable of producing uncertainty-
bounded best estimates for a range of increasingly detailed input data.
The concerns captured in the resolution also arose in other SAB committees
during reviews dating back to the early 1980s. The Radiation Advisory Committee
cited the June 1988 draft resolution in its review of the technical basis for the
A-1
-------
Radionuclides NESHAP (EPA-SAB-RAC-89-003). In 1992, the Radiation Advisory
Committee submitted a separate commentary on the status of EPA Radionuclide
Models (EPA-SAB-RAC-COM-92-001), reiterating many of the recommendations
contained in the 1989 modeling resolution. More recently, the Environmental
Economics Advisory Committee has cited the resolution in its Commentary on Peer
Review of Research Used in Support of Environmental Policy
(EPA-SAB-EEAC-COM-94-003).
Impact Assessment
Conversations with EPA personnel, including Dorothy Canter, Gary Foley,
Rosemary Russo, and Dave Brown, together with the review of published materials
form EPA, confirm that the modeling resolution had and continues to have major
positive impacts. One result was that in March 1992, then Deputy Administrator F.
Henry Habicht created an ad hoc Agency Task Force on Environmental Regulatory
Modeling (ATFERM).
The Task Force was given 12 months to recommend specific actions EPA
should take to improve model development and their use in policy, regulatory
assessment and decision-making. ATFERM's conclusions and recommendations for
improvement to ensure the scientific integrity of the use of models were published in a
guidance document in March 1994 (EPA-500-R-94-001).
AFTERM also produced a draft "Agency Guidance for Conducting Peer Review
of Environmental Regulatory Modeling" in 1993 which was then reviewed by the EEC
(EPA-SAB-EEC-LTR-93-008). The EEC review led to important additions and
refinements to guidance as indicated in the response by Administrator Carol Browner
on January 31, 1994.
In June 1994, the Administrator promulgated a peer review policy to reconfirm
and expand EPA's commitment to peer review scientific and technical work products
used in Agency decision-making. In January 1998, a Peer-Review Handbook (EPA-B-
98-001) was completed to complement 1994 Policy and to assist the Agency in meeting
its goal of enhancing the quality and credibility of Agency decisions.
In November 1997, EPA held a Models-2000 workshop to develop
recommendations for a Models Implementation and Improvement Plan for enhancing
modeling within EPA. In January 1998, the Agency established a Committee on
Regulatory Environmental Modeling (CREM) to promote consensus within the Agency
on mathematical modeling issues. Correspondingly, the Executive Committee (EC)
formed the Environmental Models Subcommittee to help the Agency in this endeavor.
Overall, the consensus response of the Agency personnel and the actions undertaken
within the Agency allude to a positive impact of the SAB modeling resolution.
A-2
-------
2. Review of Draft Agency Guidance for Conducting External Peer Review of
Environmental Regulatory Modeling (EPA-SAB-EEC-LTR-93-008)
Topic Choice and Charge
The ad hoc Agency Task Force on Environmental Regulatory
Modeling(AFTERM), created by the Risk Assessment Council, asked the EEC to review
its "peer review guidance", one of ATFERM's first products to support better
management of model development and application. At the same time, the EPA's
Council of Science Advisors (CSA) began preparation of an Agency Policy on Peer
Review. Recognizing the early stage of both programs, the EEC decided to address
the ATFERM's modeling peer review guidance and modeling issues in the Agency
Policy on Peer Review. The EEC also provided some general advice on the peer
review process. Incidentally the AFTERM report was developed in response to the
EEC's "Resolution on the Use of Mathematical Models by EPA for Regulatory
Assessment and Decision Making" (EPA-SAB-EEC-89-012).
Subcommittee Process
The ATFERM draft EPA Peer Review Guidance for Regulatory Modeling was
dated January 26, 1993. The eight-person Subcommittee assembled to review this
Guidance met on March 3-4, 1993 and issued its report on July 8, 1993. Consensus
drafts were provided earlier. Keeping the period between initial meeting and report
issuance short required close coordination and frequent communication between Chair,
committee, and staff.
Subcommittee Report
The concise eight-page letter report both supported the AFTERM program and
recommended important improvements. These changes were constructively phrased,
as were the cautions included in the letter report. Some issues addressed regarding
the peer review process were criteria, review elements, documentation, and closure.
The report was proactive, going well beyond model review to address AFTERM
continuance past its planned 12-month life, the peer review of model applications
(regulatory decision making) as well as model development, and the misuse of default
values. Many of the recommendations, e.g., documentation and closure, were thought
to be applicable to the fledgling Agency's Policy on Peer Review.
Impact Assessment
The January 31, 1994, response from the Administrator to this report included
both a point-by-point response to the Subcommittee's comments and a revised draft
Guidance incorporating many changes made in response to the SAB review. The
A-3
-------
Council of Science Advisors endorsed the response; this endorsement helped insure
the comments were considered in formulating the broad Agency Policy on Peer
Review. Finally, after the ad hoc AFTERM completed its assignment, the Administrator
decided EPA should continue coordination efforts on modeling and also required peer
review for certain models that would be used in regulatory decisions.
Office of Science Policy (OSP) staff have kept alive the "guidance for peer
review of models" since 1993. However, it has been dormant, until recently.
Reportedly, the guidance is being incorporated into the current Models 2000 program.
According to Dr. Dorothy Patton, it had been "redelivered" by program personnel
(albeit without further SAB input) to OSP many times through the interim 5 years.
Administrator Reilly issued the Agency Peer Review Policy by the end of his
term (19 January 1993). Administrator Browner re-issued the Policy in 1994. In
February 1998, the SPC issued a "Handbook on Peer Review"; the handbook includes
detailed requirements in documentation, closure, and other elements recognized in the
EEC's July 1993 report.
3. Review of MMSOILS Component of Proposed Regulatory Impact Analysis
(RIA) for the RCRA Corrective Action Rule (EPA-SAB-EEC-94-002)
Topic Choice and Charge
This report by the MMSOILS Model Review Subcommittee (MMRS) was
prepared as part of the SAB's review of the "Draft Regulatory Impact Analysis for the
Final Rulemaking on Corrective Action for Solid Waste Management Units: Proposed
Method for Analysis". This report was one of six reports reviewing different aspects of
OSWER's RCRA/RIA corrective action cost/benefit analysis and its application. The
focus of the review was the Agency's draft document entitled "MMSOILS: Multimedia
Contaminant Fate, Transport and Exposure Model Documentation and User's Manual"
dated September 1992.
The Agency, through the Office of Solid Waste and Emergency Response,
asked the SAB to review specific elements of the multi-media contaminant fate,
transport and exposure model MMSOILS, with regard to the methodology used to
predict contaminant concentrations in the environment and the resultant implications on
human health and ecological risk assessments. Specifically, the review dealt with:
a. the adequacy of methods for using a screening level model where there is
substantial subsurface heterogeneity and/or where nonaqueous phase
liquids (NAPLs) are present,;
A-4
-------
b the appropriateness of the Agency's approach for aggregating releases
from solid waste management units (SWMUs) in order to estimate
concentrations at exposure points as a function of time; and
c. the adequacy of the Agency's approach for developing long-term
effectiveness and failure scenarios for site remedies.
Committee Review Process
The MMRS held public reviews on June 22, 23, and 24, 1993. The MMSOILS
document that was the subject of this review was developed jointly by the Office of
Research and Development's (ORD), Office of Health and Environmental Assessment
(OHEA), Exposure Assessment Group (EAG), and the Office of Environmental
Processes and Effects Research (OEPER). A broad cross-section of EPA personnel
from these offices made presentations to the Committee. The SAB report was prepared
by the MMRS during subsequent writing periods.
Report and Communication
In brief, the EEC's MMSOILS report found that:
a. the interagency coordination represented by the whole RIA effort was a
"model approach" and should be adopted in other programs;
b. MMSOILS addressed all major contaminant pathways and was
computationally efficient; and
c. MMSOILS might be a valid screening tool, when applied to simplified
cases, for assessing the relative risks and costs associated with
alternative regulatory options.
However, the Committee identified two unquantifiable uncertainties that seriously
diminished the utility of MMSOILS relative to its use in the draft Corrective Action
Regulatory Impact Analysis (RIA), namely :
a. inadequate and inaccurate input parameters resulting from sparse or
inaccurate information, poor parameter estimation, especially relative to
source terms, and suspected over-reliance upon default parameters; and
b. application of the model to cases outside its range of validity, such as
sites with complex hydrogeological conditions or sites where NAPLs are
present.
A-5
-------
Considering the uncertainties, the Committee concluded that MMSOILS output
could be wrong by several orders of magnitude at complex sites. The Committee gave
numerous specific recommendations for augmentation of the RIA modeling and for
obtaining cost/benefit estimates derived from alternative approaches. Expert peer
review was found to be a general need for most aspects of MMSOILS use for RIA.
The MMSOILS report was submitted by the SAB to Administrator Carol Browner
on November 19, 1993. The SAB received a response to all six of the RIA reports from
the Administrator on March 21, 1994. The response was in the form of a table
summarizing the issues raised by the MMRS and its recommendations, and the status
of Agency activities to address the recommendations. Significant activity was underway
in response to the report, and all recommendations were being addressed in a positive
manner.
Impact Assessment
Based on interviews with Agency personnel, including Barnes Johnson, David
Brown, Larry Reed, and Elizabeth Cotsworth, the Agency response to the MMSOILS
report was immediate and positive. The Agency was working under a congressionally
mandated deadline to perform a regulatory impact analysis for RCRA. Hence, the
MMSOILS report was topically important and timely. However, EPA staff involved in
preparing the Agency's draft document on use of MMSOILS were naturally
disappointed by the lack of endorsement of their report by the SAB, but appreciated the
counsel. Many of the MMRS recommendations required additional effort with long lead
times which could not be accomplished within the deadline for Agency response. Some
EPA staff felt the MMRS did not recognize the effort involved in preparing the draft
report and that the Committee had been too critical of the Agency's efforts. Corrective
action on solid waste management units has not been submitted to Congress by the
Agency. One lesson seems to be earlier involvement by the SAB through either
consultation or a two-stage review.
4. EPA's Environmental Technology Innovation and Commercialization
Enhancement Program (EnTICE) (EPA-SAB-EEC-95-016)
Topic Choice and Charge
The Office of Environmental Engineering and Technology Demonstration
(OEETD) in the EPA Office of Research and Development (ORD) requested this
review, which was performed by the Innovative Technology Subcommittee of the
Science Advisory Board's Environmental Engineering Committee.
A-6
-------
The charge was to review a proposed strategy of establishing third-party
centers, temporarily supported by EPA, to generate credible data and information for
the market place and permitting arenas; the information was intended to help decision-
makers promote the use of innovative technologies in a more informed and confident
manner. Primary issues focused on costs and early self-sufficiency of verification
centers, EPA imprimatur and oversight, accountability and measures of success.
Secondary issues focused on the stakeholders, barriers and incentives, verification
protocols, pollution prevention, technology categories and priorities, and state
permitting and reciprocity.
Subcommittee Review Process
The six-person Subcommittee received and OEETD white paper describing
EnTICE and options for structuring, as well as the written charge, and met on May 2-3,
1995 with briefings by and to EPA personnel. The Subcommittee report was prepared
during the subsequent writing periods.
Report and Communication
The 18-page report on the EnTICE review specifically addressed each issue
accompanying the initial charge to the Subcommittee. Major findings and
recommendations were, in brief: recognition of an opportunity for EPA to provide
leadership in addressing technological development and U.S. competitiveness,
enhanced by a successful technology verification programs; the need for development
of third-party verification with protocols, EPA support and oversight, and parallel
programs addressing critical issues, impeding alternative technology infusion into the
marketplace; the need for re-assessment and use of pilot programs where success
made a difference in technology acceptance; and the need for vivid recognition of the
total stakeholder community; the need for facilitation of technology permitting and
reciprocal use and communication of data and information.
The EnTICE review report was submitted to Administrator Carol Browner on
August 25, 1995. Although there is no record of an official response to the report,
Administrator Browner's response to a companion report issued the same day indicates
the Agency's intent to move ahead "vigorously" on these issues, consistent with SAB
findings and recommendations. The companion report is the Hazardous Air Pollutant
(HAP) Monitoring Subcommittee's Review on Verification of innovative Continuous Air
Emission Monitors (EPA-SAB-EEC-95-018). The EEC's HAP Monitoring
Subcommittee report reviewed projects proposed under EnTICE and then commented
more broadly upon technology development and commercialization issues. The
EnTICE and HAP efforts resulted from the January, 1994, draft Technology Innovation
Strategy (EPA 543-K-93-002) prepared by EPA's Innovative Technology Council (ITC).
A-7
-------
The associated EEC report is, EPA's draft Technology Innovation Strategy (EPA-SAB-
EEC-95-013).
Impact Assessment
Interviews with EPA personnel indicate that the EnTICE review had immediate
and positive effects on the verification strategy then under development. This is most
evident in technology verification. Those interviewed included Penelope Hansen,
Director, Environmental Technology Verification Program; E. Timothy Oppelt, Director,
National Risk Management Research Laboratory; and Walter Kovalick, Director,
Technology Innovation Office.
Indeed, the growth of the EPA Environmental Technology Verification (ETV)
Program highlights without question and with specific reference to the EnTICE review
(pages 5-6) the effectiveness of the review process in stimulating action and providing
effective guidance. The Verification Strategy (EPA/600/K-96/003) articulates the goals,
operating principles and future plans for the ETV. This strategy has been endorsed by
EPA Administrator Browner by correspondence to Vice President Gore and to chairs
and ranking minority members of various Congressional committees.
As of April 1998, two ETV pilots had verified twelve technologies. The resulting
verification statements document and communicate the findings. EPA expects to verify
approximately 300 innovative technologies within the first decade of the ETV program.
A Quality Control and Management Plan for the Pilot Period (1995-2000) has been
prepared and published (EPA/600/R-98/064). Again the SAB input was acknowledged.
Overall the consensus response by EPA personnel to the impact assessment was
positive.
5. Review of the Waste Incineration Research Program
(EPA-SAB-EEC-96-004)
Topic Choice and Charge
ORD requested that the EEC review their ongoing waste incineration program
at the National Risk Management Research Laboratory, Air Pollution Prevention and
Control Division, Research Triangle Park, North Carolina. The charge included a
review of future issues identified, such as Products of Incomplete Combustion (PICs),
program integration between past and future work, and effectiveness of in-house
research to meet short-term and long-term agency needs.
Subcommittee Review Process
A-8
-------
The review took place in September 1995 at the RTP facility. Senior ORD
research staff briefed the Subcommittee on each of the main program areas. Copies of
all presentation material and other background materials were provided. The
Subcommittee was well-balanced with combustion and environmental experts, many
with years of experience in this area. The chair provided an oral summary of the
Subcommittee's findings to the EPA management, technical staff, and others present
during the last day of the review. Although a report was outlined at the meeting, writing
assignments were not completed during the review meeting. Difficulties with EPA's
FY96 budget, including repeated continuing resolutions and a month-long furlough of
Agency staff, interrupted the normal report preparation sequence. The report was
approved by the EEC in May 1996, by the EC in June 1996, and finally issued nearly
one year after the review, in August 1996.
Report and Communication
The review found that the program was focused on very important issues to the
Agency, and that more funding should be provided to the laboratory to allow it to build
core competency. Recommendations included collaboration with other research
institutions, and the need to keep combustion in the Agency's core research despite
public perception of difficulties with combustion-related technologies such as
incineration. The Subcommittee sensed that because of opposition to combustion-
related systems, EPA was neglecting this research area, even though combustion is an
important activity within our society. In her May 1998 response, the Administrator
stated that the Agency agreed with all findings and recommendations in the SAB report.
Impact Assessment
Based on telephone discussion with Frank Princiotta, the report had an impact in
one main area: the Combustion Branch developed a formal strategic plan. However,
the priorities of the entire organization have changed dramatically since the review was
conducted, based largely on new priorities from the Administrator (e.g., mercury).
6. Review of the Global Climate Change Engineering Research and
Development (R&D) Program (EPA-SAB-EEC-LTR-93-013)
Topic Choice and Charge
The EPA Office of Research and Development requested SAB review of its April
1993 draft report on Global Climate Change Engineering Research and Development
Program (GCERDP). This program is managed by the ORD's Air and Energy
Engineering Research Laboratory (AEERL), Research Triangle Park, North Carolina.
A-9
-------
The report contained background information on key topical areas such as greenhouse
gas emissions estimation/database management, methane migration research, and
biomass utilization research, as well as possible direction for future programs should
significantly greater resources become available. Specifically, the review dealt with
strategic and tactical issues concerning rational and scientific soundness of approach,
rigor and practicality, project selection, proposed expansion, significance of
concentration, and request for guidance.
Subcommittee Review Process
The 10-member Global Climate Change Engineering Research Subcommittee
(GCCERS) met publicly May 26-27, 1993 at the EPA AEERL. The Subcommittee
included members of the EEC, consultants previously involved with global climate
change research, and others who had contributed to the SAB's review of EPA's Report
to Congress on Potential Effects of Global Climate Change on the United States
(EPA-SAB-EC-89-016). Both draft and final reports were prepared during subsequent
writing periods.
Report and Communication
The Subcommittee report found that the existing GCCERDP was rational,
scientifically sound and rigorous, with a reasonable and practical selection of projects.
It endorsed the focus on identification and characterization of key anthropogenic
sources of methane and source control by engineering solutions. It also made
recommendation for further development of the Global Emissions Data Base (GloED),
with a focus on the needs of potential users and more attention to definition of data
quality and uncertainty. Similarly, the need to gather and disseminate information on
potential technologies for recovery and use of methane with interagency collaboration
was stressed, along with attention to waste management facilities, biomass conversion,
transportation fuels, and global emissions of tropospheric ozone depleting precursors.
The Subcommittee's review of GCERDP was submitted as a letter report to
Administrator Carol Browner on August 12, 1993. Administrator Browner responded on
December 10, 1993 with a point-by-point response prepared by the Office of Research
and Development, and an expression of appreciation which extended an invitation to
conduct future scientific reviews on selected programs. It also acknowledged that the
needs and opportunities for research and development in this and other areas would
far outstrip EPA financial resources for the foreseeable future, and that "difficult but
unavoidable choices" would need to be made.
Impact Assessment
A-10
-------
Based upon interviews with EPA personnel, including Frank Princiotta, Director,
AEERL at the time of the review, E. Timothy Oppelt, Director, National Risk
Management Research Laboratory, and Susan A. Thorneloe, Senior Project Officer,
Atmospheric Protection Branch, some progress has been made toward addressing the
continuance of the global climate research initiative, but largely in activities already in
effect. The growth and expansion into other areas appears to have been curtailed by
budgetary constraints and a change in priorities, hence the impacts of the SAB review
are not readily evident.
This outcome has been presented in a detailed follow-up on EPA's Global
Climate Change Program by Frank Princiotta, Director, Air Pollution Prevention and
Control Division at Research Triangle Park (1998) which systematically outlines the
Agency response to recommendations in the original SAB report. Both declining
budgets and change in program focus were cited as limitations to implementing SAB
recommendations. Pollution control infrastructure assessment, development of
in-house software for mitigation assessment, and alternatives to MFCs in supermarkets
are now emphasized.
However, The President has attempted to focus Congressional attention on an
enhanced program with greatly improved funding throughout the various federal
agencies, including EPA; this is the Global Climate Technology Initiative (President
Clinton, October 22, 1997). The November 1998 STAR Grants RFA lists Integrated
Assessments of the Consequences of Climate Change as a priority area. If resources
or priorities change, EPA could revitalize its global climate change initiative using the
SAB letter report guidance as a logical basis for action. Hence, although global climate
research in the Agency appears somewhat dormant at the moment, an executive
mandate with concomitant funding could provide EPA a window of opportunity for
resumption of its original agenda.
7. Review of a Methodology for Establishing Human Health and Ecologically
Based Exit Criteria for the Hazardous Waste Identification Rule (HWIR)
(EPA-SAB-EC-96-002)
Topic Choice and Charge
The Office of Solid Waste (OSW) requested a SAB review of the March 1995
draft, "Development of Human Health and Ecologically Based Exit Criteria for the
Hazardous Waste Identification Project". OSW prepared this document to support the
Hazardous Waste Identification Rule (HWIR). The intent of the HWIR is to establish
human health-based and ecologically-based waste constituent concentrations (exit
criteria) for constituents in wastes below which listed hazardous wastes would be
A-11
-------
reclassified and become delisted and non-hazardous wastes under the Resource
Conservation and Recovery Act (RCRA).
The draft HWIR document described a proposed methodology for calculating
exit concentrations of 192 chemicals for humans and approximately 50 chemicals for
ecological receptors, based on a consideration of five types of waste management
units; numerous release, transport and exposure pathways; and biological effect
information.
The charge to the Subcommittee contained 17 specific questions about the
proposed methodology. After discussing the charge, the Subcommittee agreed to
focus on the larger issues because a detailed peer review of each of the individual
equations, parameters or assumptions was beyond the resources and time available to
the Subcommittee. However, summary responses to the charge questions are included
in the Subcommittee Report.
Subcommittee Review Process
After receiving the draft methodology and the charge, the 11-person
Subcommittee met publicly in Washington, DC on April 26-27 and May 30 - June 1,
1995. The Subcommittee heard presentations from the Agency, its contractors, and
members of the public regarding the proposed methodology for calculating exit criteria.
In addition to reviewing the proposed methodology, the Subcommittee requested and
reviewed sample calculations for two chemicals in order to evaluate the implementation
of the total methodology. The Subcommittee report was prepared during the
subsequent writing periods.
Report and Communication
The 30-page report of the HWIR Methodology review addressed each charge
with focus primarily on the larger issues. The Subcommittee concluded that the
proposed methodology has a number of critical flaws that must be corrected in order to
develop scientifically defensible exit criteria. The Subcommittee recommended that the
proposed method of calculating exit criteria, which considers individually each
exposure pathway , be abandoned in favor of true multi-pathway calculations in which a
receptor concurrently receives contaminants from a source via all pathways. In
addition, the Subcommittee urged the Agency to: conduct substantial validation and
peer review of the overall methodology; provide a systematic examination of
parameters and uncertainties; calculate ecologically-based exit criteria for those
chemicals for which a minimum data set is available; and rewrite the documentation for
clarity and transparency. The Subcommittee estimated that a concerted effort to
correct the major flaws in the methodology could be completed within a relatively short
A-12
-------
time, perhaps a year or two, if a concerted effort is instituted with the appropriate
scientific involvement.
The HWIR Methodology review report was submitted to the Administrator Carol
Browner on May 23, 1996. The official response to the report from Administrator
Browner was sent on August 29, 1996. Two separate follow-on communications were
made to clarify the major aspects of the SAB Advice. In a telephone conversation
between Mr. Mike Shapiro (OSW Office Director), Dr. Mark Harwell (Subcommittee
cochair) and Dr. Donald Barnes (SAB Staff Director), the major conclusion that the
methodology lacked scientific defensibility for its intended regulatory use and that an
adequate scientific basis might be developed with a concerted effort was clearly
emphasized. In a face-to-face meeting between Mr. Mike Shapiro, Dr. Donald Barnes
and Dr. Ishwar Murarka (Subcommittee Cochair) further discussions of the SAB
comments occurred.
The written response from the Administrator acknowledged that SAB review had
raised significant issues and that the OSWER and ORD were working together to
develop a strategy to address the SAB comments and fulfill the Agency's statutory
obligations within a reasonable time frame. The response further stated that once such
a strategy is developed, the Agency would be interested in meeting with the SAB to
discuss the proposed approach.
A-13
-------
Impact Assessment
Based upon conversations with EPA personnel, including Barnes Johnson,
Rosemary Russo, Jim Brown and Mike Shapiro, it is clear that the SAB review had a
major impact on the Agency - the Agency took the SAB advice to obtain an extension
of approximately five years from the U.S. Circuit Court for promulgating the HWIR,
specifically citing the SAB report.
After obtaining the extension, the Agency assembled a team of experts from the
OSWER and the ORD to address the technical issues raised in the SAB review. A
detailed plan has been prepared by the Agency to describe how the new/revised HWIR
methodology is being developed, verified and will be applied to obtain the risk-based
exit criteria for the HWIR. The Agency has commissioned a blue ribbon panel to peer-
review the plans for the revised methodology. The SAB report has been extensively
used in formulating the plans for revising the HWIR methodology. Overall, the general
response from the Agency to the question of impact was that the SAB review did raise
significant issues which are being resolved in the revised methodology to the extent
possible within the deadline imposed by the court.
8. Leachability: Recommendations and Rationale for Analysis of
Contaminant Release (EPA-SAB-EEC-92-003)
Topic Choice and Charge
Over a decade (1981-91), the Environmental Engineering Committee (EEC) of
the Science Advisory Board (SAB) reviewed a number of EPA subjects and issues
involving Leachability phenomena, either as a major or minor factor in the review. In
these various reviews, the Committee noted a number of problems and issues relating
to Leachability phenomena that were common to a variety of programs, rules, and
Agency procedures. The Committee believed that these common problems and issues
would be best called to the Agency's attention through a general set of
recommendations on Leachability phenomena rather than in the specific individual
reviews.
Believing that the scientific principles of contaminant Leachability need broader
understanding and exposition, the EEC undertook the initiative, with the concurrence of
the SAB Executive Committee, to conduct a self-initiated review to:
a. Consider the fundamental scientific principles that can reliably describe
contaminant release/transport, and, in particular, to consider the
A-14
-------
controlling characteristics of the source, the leaching media and the
importance of dynamic considerations; and
b. Suggest how the scientific principles can be applied to determine how a
waste will leach when present in the environment, according to a
prescribed scenario.
Subcommittee Review Process
The Leachability Subcommittee (L.S.) was formed by the EEC and had 13
members. The group convened a project scooping and planning session in Houston,
Texas on December 15-16, 1989 immediately following a Workshop related to this
topic. The L.S. then followed this with a one-day session in Washington DC on
February 26, 1990, which was devoted to assessing the Agency's varied needs for
Leachability-related information. The day's activities and findings were then discussed
with the full EEC on February 27, 1990. This was followed by a Workshop on
Leachability on May 9, 1990 in Washington DC. The Workshop was conceived as a
vehicle for distinguished scientists, engineers, and practitioners in the field to focus on
the scientific principles and issues relating to Leachability phenomena. The Workshop
was video taped, so that those unable to attend from EPA, or any other interested
parties, could have the benefit of this exchange of information.
The Leachability Workshop assisted the L.S. of the SAB's EEC to better define
the fundamental scientific principles that control Leachability. Further, the workshop
assisted the SAB and the attendees in ascertaining how Leachability phenomena and
tests can be applied on an appropriate and consistent basis to determine how a waste
will leach when present under various scenarios in the environment.
This SAB self-initiated study to provide the Agency general and broad-based
guidance on release of contaminants from sources spanned a period of 18 months from
initiation to transmittal of the report to the Administrator.
Report and Communication
The Leachability report recommended an Agency-wide effort to:
a. study and better understand the mechanisms controlling Leachability;
b. develop better conceptual models for waste management scenarios;
c. evaluate stresses affecting long-term contaminant release potential;
A-15
-------
d. develop a variety of contaminant release tests and test conditions to
assess potential release of contaminants from sources of concern;
e. improve mathematical models to complement laboratory tests, and
f. field-validate leach tests before being broadly applied.
The Leachability Subcommittee sensed that improving the state-of-the-art of
quantitatively predicting the rate of contaminant release from sources should be a high
priority of the Agency. Hence, the Subcommittee recommended establishment of an
inter-office, inter-disciplinary task group, including ORD, to help implement their
recommendations and devise an Agency-wide protocol for evaluating release
scenarios, tests, procedures, and their applications. The Subcommittee also called for
increased core research on contaminant release and transport within the waste matrix.
The Leachability report was submitted by the SAB to Administrator William Reilly
on October 29, 1991. The SAB received a response to the report from the Administrator
on March 2, 1992. Each of the nine recommendations was favorably addressed, and
specific examples were given of ongoing Agency activities in response to the
recommendations. However, no commitment was made to establish an inter-office task
group to implement the recommendations. Rather, the issue was referred to the
Environmental Monitoring and Management Council, which is charged with, among
other things, improving the coordination of EPA's methods development program to
avoid duplication and to avoid inconsistencies in quality assurance and quality control
guidelines. No further response to the Leachability Report was received by the SAB.
Impact Assessment
Based upon interviews with the EPA personnel, including Gail Hansen and Greg
Helms both of OSWER, the impacts of the SAB recommendation on Leachability are
somewhat limited. The Agency staff agrees with most of the recommendations
contained in the SAB review. They acknowledge that this SAB report has been widely
distributed within the Agency and is often used as a scientific reference document. In
1998, EPA began to use the SAB report more as it became active in addressing
regulatory issues on leaching test. In 1998, the Agency completed a technical report to
update some of the contents of the SAB report, particularly the aspects relating to
leaching of contaminants from oily wastes. Overall, the Agency staff acknowledges
that the report is valuable to them, but the SAB recommendations have only been
utilized in a limited fashion.
The topic remains important. In 1999, the EEC submitted a commentary to the
Agency entitled, "Waste Leachability: The Need for Review of Current Agency
Procedures" (EPA-SAB-EEC-COM-99-002). OSWER convened a public meeting July
A-16
-------
23-24, 1999, to define directions for modifying and/or developing leaching test methods
to meet the multiple needs in the Agency's programs.
A-17
-------
APPENDIX B
Summary of Data from the General Telephone Interviews
The interviewees deemed the following attributes as those that make an SAB
review successful.
Topic Choice and Charge
The SAB contributes best when the topic is significant, contentious, technically
challenging, and important to top management.
The SAB contributes best when the topic is primarily scientific, not political or primarily
policy.
The SAB should match the type of review to the life cycle of the topic.
The SAB works best when the scope of the topic is well defined and can be grasped by
the subcommittee, or is tiered from simple to complex, if necessary.
The SAB should go beyond the charge, when appropriate, to get to the heart of the
issue.
Committee Review Process
The SAB works best when the review group is composed of members who are
prepared to provide appropriate scientific opinions.
The SAB contributes most when it achieves consensus; however, the SAB contributes
even when it does not achieve consensus if the differing views are clearly
presented in the report.
The SAB works best when both the review group (when in session) and Designated
Federal Official (DFO) maintain clear communications with the Agency sponsor.
The process works best when the sponsor's requirements, including timeliness, are
clearly defined at the beginning of the review.
Report Preparation Process
The SAB should provide verbal debriefings to the sponsor at the end of public
meetings.
The SAB should share drafts of report with sponsors (and other stakeholders) as early
as possible while still meeting all FACA requirements.
B-1
-------
To improve the timeliness of the final report, the SAB should write draft sections or, at a
minimum, detailed outlines before leaving the review; if necessary, a writing day
should be scheduled.
The SAB reports should be forthright about what is agreed, and what is known.
The SAB reports should treat the Agency professionally and consider in context what
the Agency can and cannot do.
The SAB reports that contain constructive criticism can shorten response time by
following up with sponsor.
It would be helpful if the SAB prioritizes recommendations to help EPA decide what to
focus on first.
The SAB should try to reach consensus; if consensus cannot be reached, then the final
report should clearly say so.
Final Report and Communication
The SAB should brief the EPA sponsoring office on findings after the report is approved
and before it is published.
The SAB report and findings should be available to sponsoring organizations within
expected time lines.
The SAB chair should follow up with sponsor, especially when the Agency takes no
action on important topics, even when no consensus advice was conveyed.
After the EPA responds to the report, the chair should brief EPA sponsors and the AA
(if needed) to engender agreement on actions consonant with the
recommendations.
B-2
------- |