IPA-SAB-EC-LTR-                 SABECLTR940
4-002
      wEPA
          THE SCIENCE ADVISORY BOARD

     Overview    of   SAB    Comments    and
     Recommendations  on the  Proposed  RIA
     for  the  RCRA  Corrective  Action  Rule
                EPA-SAB-EEAC-94-001  "Review of the  Contingent
                Valuation Method for the Proposed RIA for  RCRA
                Corrective Action Rule" by the Environmental Economics
                Advisory Committee

                EPA-SAB-EEAC-LTR-94-001  "Review of Economic
                Aspects of the Proposed RIA for the RCRA Corrective
                Action Rule" by the Environmental.Economics Advisory
                Committee

                EPA-SAB-EEC-94-002 "Review of MMSOILS component of
                the Proposed RIA for the RCRA Corrective Action Rule"
                by the Environmental Engineering Committee

                EPA-SAB-EPEC-COM-94-001  "Commentary  on the
                Ecological Risk Assessment for the Proposed RIA for the
                RCRA Corrective Action Rule" by the Ecological
                Processes and Effects Committee

                EPA-SAB-EHC-LTR-94-003 "Review of the Health Benefits
                for the Proposed RIA for the RCRA Corrective Action
                Rule" by the Environmental Health Committee

                EPA-SAB-EC-LTR-94-002 "Overview of SAB Comments on
                the Proposed RIA for RCRA Corrective Action Rule" by
                the RCRA/RIA Steering Committee
                                             November 1993
                                                 Recycled/Recyclable
                                                 Printed with Soy/Canda Ink on paper that
                                                 contains at least 50% recycled fiber

-------
EPA-SAB-EC-LTR-94-002   'Overview  of  SAB
Comments  on the  Proposed  RIA  for  RCRA
Corrective Action Rule" by the RCRA/RIA Steering
Committee

-------
               UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
                            WASHINGTON, D.C. 20460
                                                            OFFICE OF THE ADMINISTRATOR
 November 19, 1993                                           SCENCE ADVISORY BOARD

 EPA-SAB-EC-LTR-94-002

 Honorable Carol M. Browner
 Administrator
 U.S. Environmental Protection Agency
 401 M Street, S.W.
 Washington, D.C. 20460

      Subject: Overview of SAB Comments and Recommendations on the Proposed
      RIA for the RCRA Corrective Action Rule.

 Dear Ms. Browner

      This report is one of a series of six reports (listed in Appendix A) generated by
 the SAB in response to the subject request from the USEPA's Office of Solid Waste
 and Emergency Response (OSWER). It contains a brief summary and overview of
 the salient conclusions of the other five reports as well as some observations,
 comments and recommendations of the RCRA/RIA Steering Committee (RRSC). A
 roster of RRSC is in Appendix B.

      At the October, 1992 meeting of the Executive Committee (EC), the Science
 Advisory Board (SAB) was asked by OSWER to review its then-nearly-complete RIA
 methodology which was being applied to the cost/benefit analysis required prior to
 promulgation of the Agency's final RCRA Corrective Action Rule.  The EC, recognizing
 the importance, complexity, creativity, and novelty of OSWER's work and its
 multi-disciplinary  character, established the RRSC to assure that certain significant
 aspects of the RIA-both methodology and application-received appropriate attention
from the relevant SAB standing  committees.

      At a public meeting on January 29,1993, the RRSC concluded, on the basis of
 presentations by  and discussions with OSWER personnel, that four individual SAB
 standing committees should undertake reviews of the major segments of the
 RCRA-RIA with appropriate inter-committee liaison. The Environmental Economics
Advisory Committee (EEAC) reviewed the contingent valuation (CV) methodology

-------
(hereafter called CV-1) and the application of CV in the specific case of the RCRA-RIA
(hereafter called CV-2). The Environmental Engineering Committee (EEC) reviewed
the principal fate and transport model. MMSOILS. used in the RCRA-RIA. The
Ecological Process and Effects Committee (EPEC) reviewed the ecological risk
assessment portion of the RCRA-RIA and the Environmental Health Committee (EHC)
reviewed the human health risk assessment portion of the RCRA-RIA.  The RRSC
provided coordination and its own insights with respect to the RCRA-RIA methodology
and its application.

Interpretation of the Charge

      The charge for these reviews is contained in two separate requests for reviews
of RCRA-RIA components: the Groundwater Contingent Valuation (CV) Study
(October 21,1992) and the MMSOILS model for fate and transport (March 26, 1993).
The latter request  also asked the SAB to comment on  "the implications that the model
has on the human and ecological risk assessment" and to "consider during their
review several practical factors including the baseline risk assessment  and the fact
that the RIA is a predictive analysis".  Thus the RRSC  concluded that how these risks
were determined needed to be reviewed as well.  Each of the standing committees of
the SAB has addressed its portion of the charge in their separate reports. The RRSC
has taken as its charge the task assigned it by the EC of ensuring that the significant
aspects of the RIA - both methodology and application - received appropriate
attention from the  relevant SAB standing committees, of ensuring that coordination
exists where needed and, ultimately, providing its own  comments and overview.  As
noted by the EHC, this draft methodology  is actually a  screening analysis which
provides preliminary estimates rather than definitive analysis reflecting  site specific
details.
    ^**^ -—
      The SAB was not asked to review the costs of corrective action  or the
procedures for estimating them, nor was it supplied with the detailed background
information needed to da so. The SAB has thus reviewed only the estimation  of the
benefits of the corrective action and the methodologies used in deriving the
expressions of the benefits. A review of the costs and their estimation methods might
produce comments. Thus, the absence of comments in this area does not constitute
any SAB position as to the costs of remediation or their methods of estimation.

-------
 General Comments

      This RIA is one of the most complete and complex that OSWER has
 undertaken and is the first to undergo detailed review by the SAB.  In spite of the
 scientific criticisms which follow, we commend the Agency for this major effort, for its
 openness of discussion with the SAB, and for its innovative attempts to apply a wide
 variety of types of information and procedures to assess the risks associated with solid
 waste management units and the benefits of abating those risks.

      The OSW/ORD working group is also to be commended for a well-coordinated
 and focused effort to  develop an RIA that will help the Agency and the Nation better
 understand the risks associated with RCRA sites and the costs and benefits of
 remediation - and  the size, complexity and difficulty of the analysis. The intra-agency
 coordination represented by this RIA is itself a model  approach that the Agency should
 apply to other programs to promote effective and efficient interactions of Program
 Office and ORD to ensure that Program Office activities represent state-of-the-art
 science and technology.

      Finally, despite the large amount of good work that has  gone into this RIA, it
 only  accounts for part of the benefits that may accrue from reducing health and
 ecological risks from RCRA sites.  The comments which follow include
 recommendations for both short term changes to improve this RIA and long-term
 investments in research and analysis to improve RIAs within the Agency.

 Overview of the Major Comments and Recommendations of the SAB Standing
 Committees

      To place into context the specific comments and recommendations of the
 RRSC, some of the major findings of the  SAB's standing committees are summarized
 here.  The reader is referred to the individual reports for full and detailed descriptions
 of these and other findings.  In each case, the standing committees offer
 recommendations for  both short term and long term improvements.

      The Agency's CV-1  document represents a substantive contribution, extending
 understanding  of the issues associated with contingent valuation as a method of
 estimation of non-market values.  Even so, concerns were raised about the method
which need answering: whether the pretesting and design are such as to truly assure-

-------
that a well-defined groundwater commodity was understood properly by the
respondents; whether use of the Box-Cox econometric estimates alone is acceptable
and defensible; whether embedding was adequately treated; whether the "per
household" non-use values from CV-1 can be regarded as either upper or lower bound
values; and others.  Among the concerns raised about CV-2, given also the concerns
about CV-1, itself, are: whether the application of values obtained in  CV-1 to the
different set of circumstances of CV-2 is possible; that EEAC could not endorse the
McClelland et al study as a basis for EPA to determine the non-use values of
groundwater; and that the hedonic analysis was not actually used in  the RIA.

      CV methods in general are still controversial and the EEAC concluded that
more information was needed to apply CV in this RIA. Nonetheless, the EEAC
showed sufficient confidence in the approach to recommend that further research be
undertaken,  particularly to resolve whether the CV approach can produce information
useful in RIAs.

      Fate and transport information is fundamental in assessing exposures and,
therefore,  human health and ecological risks.  While noting that the methods and
formulations used in the MMSOILS model are well known, documented and accepted
and that underlying assumptions are dearly stated in the RIA, the EEC concluded that
here, too,  there are difficulties (though if MMSOILS is applied to simplified case
studies it might be a valid screening tool for assessing the relative risks and costs
associated with alternative regulatory options).  The primary difficulties are: sparse or
inaccurate information, poor parameter estimation especially relative  to source terms,
suspected over-reliance upon default parameters, and that the Model is applied  to
cases outside the range of its validity. Given these shortcomings, many of which were
already realized by the Agency, the most basic and pressing concern is whether the
use of a generic model  such as MMSOILS is appropriate as a basis for the
assessment of regulatory costs and benefits at the national level since the fate and
transport estimates that comprise the model output may be wrong by orders of
magnitude for many complex sites. In its report the EEC recommends ways in which
the Agency can augment exposure and cost/benefit estimates using alternative
approaches.

      The human health risk assessment methods used in the RIA are well known
and often  used in the regulatory arena. There is a question as to how well they can
be used to estimate risks - or exposure levels of concern for risk - in this case of

-------
 multiple exposures, for analytical rather than regulatory purposes.  Also there are
 fundamental differences between the usual methods used for carcinogens and non-
 carcinogens and the results that can be obtained in each case. Much of the criticism
 of the human health risk analysis in CV-2 has to do with a confusing use of risk and
 exposure terminology and is easily rectified. We urge that it be rectified.  The EHC
 also urges that quantities be calculated to describe the population exposed at levels of
 concern for cancer and non-cancer adverse effects, and the attendant risks so they
 are comparable and offers suggestions for presenting non-monetized benefits.  Some
 of the proposed calculations can be carried out in the short term (calculating the
 number of people exposed to carcinogens at levels of concern for cancer risk, as is
 already done for noncarcinogens for non-cancer adverse effects risk) whereas_the
 method proposed for calculating population risks for noncancer adverse effects will
 take more time and development.  Of all of the sections of RIA, this section is the
 most easily improved through the application of existing scientific knowledge.

      The EPEC recognizes the formidable task undertaken by the Agency in the
 ecological risk assessment but raises several questions about it. Among the concerns
 raised are these: the major pathway considered is not necessarily the most likely to
 cause adverse ecological effects, the range of ecological endpoints considered is
 limited, there is no consideration of the ecological risks and benefits of site
 remediation, there is insufficient discussion of data sources and assumptions, and
 there are a number of application and interpretive errors. The  EPEC also suggests
that the ecological risk assessment be recast in a form consistent with the Agency's
"Framework for Ecological Risk Assessment" (EPA/630/R-92/001, February,  1992).

Specific Comments and Recommendations of the RRSC on RCRA-RIAs

     _ a).  The RRSC noted with pleasure that each major chapter in the RIA
           contained a final section on limitations which served to enhance the
           understanding of the reader/user of what meaning can be ascribed to the
           contents of the chapters.  It is recommended that the sections be
           enlarged where necessary to address the criticisms from the SAB
           relevant to the final text of the  RIA.

      b)    It is important to improve the assessment of fate and transport, and
           therefore of estimated exposures,  as much as possible since the large
           error band seen by the EEC can only seriously compound and make

-------
      more uncertain the assessment of both human health and ecological
      risks and the benefits to be derived by their abatement. Greater reliance
      should be placed on measurements of exposure to supplement and
      validate model predictions, as called for in the Exposure Assessment
      Guidelines.

c)    The risks created by the remediation process should be addressed.
      EPEC identified additional ecological risks in their report such as
      impacts on terrestrial wildlife, habitat, and biodiversity which were not
      adequately addressed.  Additional risks include: loss of contaminants to
      other media during pumping, treatment, excavation, and hauling, e.g.,
      transferring groundwater eventually to surface water, stripping of volatile
      compounds, air entrapment of soil particles; accidents to workers during
      remediation and transportation; and puncturing a confining structure and
      contaminating a deeper aquifer during installation of wells.  These risks
      are relevant because several Superfund Records of Decisions have been
      amended/overturned due in part to risks to workers and off-site
      communities during remediation.

d)    While the cost estimates in the RIA for corrective action were not
      evaluated in a consensus manner by the SAB, several issues of possible
      concern were identified by the EEC: Should additional sites be included.
      e.g., pre-HSWA land treatment units, very large DoD sites,  more spill
      sites? Was the cost of a given cleanup underestimated, e.g., quantity of
      soil to be remediated, labor costs under hazardous conditions, insurance,
      inflation? What is the comparison of the cost in the draft RIA report with
      costs developed in  1992 by the University of Tennessee's Waste
      Management Research and Education institute? Should other cost
      categories be included, e.g., transaction costs and government
      administrative costs?

            We recommend that these questions be addressed by  EPA during
      the public comment period.

e)    The RRSC has considered sampling strategy since, as in all
      assessments, results and interpretations depend on the samples used  in
      the RCRA-RIA. The current RIA gives much consideration  to this subject-

-------
       and the sample used is large relative to that used in other cases.  RIAs
       operate on a national scale and have multiple components (economics,
       engineering, human health, and ecological effects).  The choice of
       sample sites has to be representative of the population of sites in  the
       country and it also has to consider the attributes of these multiple
       components.  The RCRA RIA sampling design was based primarily on
       the size and type of waste sites but did not consider the nature of the
       ecological risks (based on exposure or effects) or, necessarily, the
       different nature of health risks (and exposures), in choosing the sites.
       Thus it is not evident that either type of risk assessment is representative
       because ecological and health criteria were not part of the stratification
       process.

             We therefore recommend that in future  RIAs the sample designs
       incorporate criteria appropriate for all aspects of the RIA.  The categories
       of samples should include estimates of central tendency and dispersion
       and a discussion of the sources of variability within each category.
       Estimates of uncertainty should be included in the National Assessment.
       In some cases, it may be necessary to use different designs to address
       particular types of risks. For the long-term the Agency should consider
      whether calculation of a National Assessment is a useful decision-making
      tool for rulemaking, particularly where site-specific conditions are highly
      variable.

f)     The RRSC supports the recommendations for further research into and
      development of the methodologies needed to perform RIAs as set forth
      in the other five reports. Some further questions which research may be
      able to answer are: (a)  Can the CV methods be sensitive  enough to
      distinguish between the values of clean, cleaner and cleanest ground
      water? This question is of importance in examining different corrective
      action options which may yield significantly different costs and different
      levels of dean-up.  If this distinction cannot be made, the  distribution of
      CV values would stand  as an invariant and might have little to say about
      which option to choose, if any. And (b) what are the CV values of sound
      ecosystems or of good health? These questions, if they can be answered
      without overlap with other CV values, could greatly assist in valuing
      benefits.

-------
       g)     Chapter 13 of the RIA should be renamed to indicate that costs and
             benefits are characterized, since the non-monetized and the monetized
             benefits cannot be compared to costs in the same manner.  Monetization
             is not always possible so other types of characterizations should be
             presented in ways that make comparisons possible and that facilitate
             judgments about costs versus benefits; also terms should be dearly
             defined and used consistently.  Indeed, even if complete monetization is
             achieved, presenting additional characterizations of costs and benefits is
             highly desirable. All characterizations should include a description of
             their uncertainties. Specific suggestions for improvements are provided
             in Appendix C to this report  Similar clarifications of meaning and,
             definition should be made consistently throughout the entire report.

       In closing, we  commend the Office of Solid Waste for its pioneering efforts in
the development of this RCRA RIA. Regulatory impact assessments by their very
nature are not site specific and operate at the national scale or even international
and/or global scale.   Based on our review of this RIA, we recommend that the Agency
build on the experience gained here to develop a technical support document (TSD)
providing guidance on the development of an RIA. The TSD should include a variety
of approaches for assessing the economic, human health, and ecological benefits and
costs associated with proposed regulation.  We suggest that the TSD incorporate as
building blocks the Human Health Risk Assessment Guidelines and the Framework for
Ecological Risk Assessment

       Finally, we make one additional recommendation. In the first stages of
approaching and defining a major project such as the RIA, the Agency might consider
availing itself of the consultation role of the SAB.  In this role the individual members
and consultants of SAB committees offer advice and comments as individuals in
public meetings on points of interest raised by the staff members about their nascent
project Although the occurrence of such a consultation is recorded and reported to
the Administrator, the details of the advice are not. Such advice  at an early stage,
can serve to raise questions that are better addressed early rather than dose to the
end of a project The SAB may later conduct a peer review of the final agency
document
                                      8

-------
      The SAB is pleased to have had the opportunity to review this important
project, and we look forward to your response to these comments, as well as
reviewing other RIAs in the future.

                               Sincerely yours,
Dr. Raymond C. Loehr, Chair                    Dr. Paul Deisler, Chair
Executive Committee                            RCRA-RIA Steering Committee
Science Advisory Board                         Science Advisory Board
                                     9

-------
APPENDIX A. LIST OF SAB REPORTS REVIEWING DIFFERENT ASPECTS OF
OSWER'S RCRA/RIA CORRECTIVE ACTION COST/BENEFIT ANALYSIS
METHODOLOGY AND ITS APPLICATION.

The reports are, in brief:

1.    EPA-SAB-EEAC-94-001 "Review of the Contingent Valuation Method for the
      proposed RIA for RCRA Corrective Action Rule" by the Environmental
      Economics Advisory Committee  (Also referred to as CV-1)

2.    EPA-SAB-EEAC-LTR-94-001 "Review of Economic Aspects of the p'roposed
      RIA for the RCRA Corrective Action Rule" by the Environmental Economics
      Advisory Committee (Also referred to as CV-2)

3.    EPA-SAB-EEC-94-002 "Review of MMSOILS component of the Proposed RIA
      for the RCRA Corrective Action Rule" by the Environmental Engineering
      Committee

4.    EPA-SAB-EPEC-COM-94-001 "Commentary on the Ecological Risk
      Assessment for the proposed RIA for the RCRA Corrective Action Rule" by the
      Ecological Processes and Effects Committee

5.    EPA-SAB-EHC-LTR-94-003 "Review of the Health Benefits for the proposed
      RIA for the RCRA Corrective Action Rule" by the Environmental Health
      Committee

6.  _ .EPA-SAB-EC-LTR-94-002 "Overview of SAB Comments on  the proposed RIA
      for RCRA Corrective Action Rule" by the RCRA/RIA Steering Committee
                                 A-1

-------
APPENDIX B.
                US ENVIRONMENTAL PROTECTION AGENCY
                        SCIENCE ADVISORY BOARD
                     RCRA/RIA STEERING COMMITTEE

                                ROSTER

CHAIRMAN

    -  Dr. Paul F. Delsler, Jr., 11215 Wilding Lane, Houston, Texas 77024

MEMBERS

      Dr. Richard A. Conway, Union Carbide Corporation, South Charleston, West
      Virginia 25303-0361

      Dr. Kenneth L. Dickson, University of North Texas, Denton, Texas 76203-3078

      Dr. Allen V. Kneese, Resources for the Future, Washington, D.C. 20036

      Dr. Verne A. Ray, Pfizer Inc., Groton, Connecticut 06340

      Dr. Arthur C. Upton, New York Medical Center (Retired), Sante Fe, New Mexico
      87501

SCIENCE ADVISORY BOARD STAFF

      Dr. Edward S. Bender, Designated Federal Officer, US EPA/Science Advisory
    "Board,  401 M Street, S.W. (A-101F) Washington, D.C. 20460

      Mrs. Marcia K. Jolly (Marcy)
      Secretary to the Designated Federal Official
                                 B-1

-------
 APPENDIX C.  Recommendations from the Steering Committee for Clarifying the Cost
 Benefit Presentation

       The characterizations of costs and benefits in Chapter 13 can be improved,
 particularly in the way the information is presented in Section 13.3 and in Exhibits 13-1
 and 13-2.  These exhibits, in particular, are very important since they summarize the
 output of the entire cost/benefit study; great care should therefore be taken to be sure
 they are not easily misunderstood or misused.

       In Exhibit 13-1, for example, the values given might be designated as
 "Preferred Value"  instead of "Effect of Corrective Action" and a second line'added
 entitled "Range of Estimates" with the corresponding figures to give the decision
 maker some immediate understanding of the uncertainties. In the draft as written, the
 non-use value of ground water would then be shown as the preferred value of $2.3
 billion (as now, unless changed in response to the SAB review) and the ranges of the
 estimate would be given as $0.17-18.0 billion (using different numbers if the numbers
 should change as a result of the SAB review). Exhibit 13-2 also offers  opportunities
for improvement ranging from changing captions to more suitable ones (including
 correcting risk terminology)  to adding further information to make useful comparisons
 possible. Here are some examples of desirable changes: the figure of  $18.7 billion
 should be shown as the preferred value with its ranges of estimate as suggested for
 Exhibit 13-1; the caption "Avoided Non-Cancer Effects" should be changed to
 "Avoided Non-cancer Exposures of Concern", which is what they are; the 100 to 12
 million "cases" under the Non-Cancer column should become "exposures of concern"
since these are not cases of actual effect as in the case of cancer, and at least one
additional column needs to  be added to the exhibit.  It would shed further light on the
 benefits to  be obtained is one giving the number of "Avoided Cancer Exposures of
 Concern"; it would be based on the numbers of people, exposed at levels of exposure
yielding a risk of cancer of 10 or higher using much the same information already
used to estimate cancer risks (as suggested in the EHC report). As the exhibit now
stands, the cancer and non-cancer effects - the totality of health effects - cannot be
compared or placed in context with each other.  Although the definitions of what is "of
concern" differ for cancer and non-cancer effects, they do nonetheless exist and are
accepted as meaningful and so this additional column, compared to the one retitled
"Avoided Noncancer Exposures of Concern", would provide useful additional
information to the decision maker. If, in the future, columns for non-cancer effects

                                     C-1

-------
comparable to the column entitled "Averted Population Cancer Cases" can be
provided, so much the better.  Such information might assist in providing a basis for
monetizing non-cancer effects avoidance as do the figures for cancer cases now in
hand in appropriate cases (medical costs and productivity losses avoided and the
like). Finally, expanding the footnote to offer some idea of the meaning of current
ecological risk and whether corrective action will have any effect on it will add further
to the value of this Exhibit in shedding light on benefits to be expected from  the
corrective action.

      The two exhibits should also  be changed to reflect the fact that there are two
baseline cases: one with the entire population either treating or substituting its_water
supply (which now forms the basis of the small monetary benefit ascribed to averting
treatment and substitution in Exhibit 13-1) and a second in  which no one in the
population treats or substitutes their water supply (the one for which the relatively
large decreases in cancer incidents shown in Exhibit 13-2 are estimated). As they
stand now, Exhibits 13-1 and 13-2 appear to be anomalous in this regard and the full
information that could be displayed in the characterization is not displayed.

      These few examples illustrate ways in which the characterization of the costs
and benefits can be greatly enriched; they and others like them apply not only to the
two exhibits cited but to Chapter 13 as a whole.
                                     C-2

-------
APPENDIX D.    List of Acronyms

CV        Contingent Valuation Methodology
EEAC      Environmental Economics Advisory Committee (SAB/EEAC)
EEC       Environmental Engineering Committee (SAB/EEC, also referred to as
           "The Committee")
EHC       Environmental Health Committee (SAB/EHC)
EPEC      Ecological Processes and Effects Committee (SAB/EPEC)
MMSOILS   A Mathematical Model for Soils (Includes other media transfer from
           soils.)
OSW       Office of Solid Waste (U.S. EPA)
OSWER    Office of Solid Waste and Emergency Response (U.S. €PA) '
RCRA      Resource Conservation and Recovery Act
RRSC      RCRA RIA Steering Committee
RIA        Regulatory Impact Analysis
SAB       Science Advisory Board (U.S. EPA)
SWMUs    Solid Waste Management Units
                                 D-1

-------
EPA-SAB-EEAC-94-001 'Review of the Contingent
Valuation Method for the Proposed RIA for RCRA
Corrective  Action  Rule"  by  the  Environmental
Economics Advisory Committee

-------
      United States      Sdtnct Advteoiy '        EPA^AB-EEAC^44)01
      EnvfconiMntel      Board (1400F)         Novwnbtr 1993
      Protection Agency     W««hhgtoo DC

©EPA AN SAB REPORT:

     CONTINGENT VALUATION

     METHODOLOGY  (CV 1)
    REVIEW OF THE CONTINGENT
    VALUATION METHOD FOR THE
    PROPOSED RIA FOR RCRA
    CORRECTIVE ACTION RULE BY THE
    ENVIRONMENTAL ECONOMICS
    ADVISORY COMMITTEE

-------
                UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
                             WASHINGTON, D.C. 20460
                             November 19, 1993
                                                              SCENCE ADVISORY BOARD
 EPA-SAB-EEAC-94-001
 Honorable Carol M. Browner
 Administrator
 U.S. Environmental Protection Agency
 401 M Street, S.W.
 Washington, DC 20460

      Subject:     Science Advisory  Board's Review of the Contingent Valuation
                  Method for the proposed RIA for the RCRA Corrective Rule

 Dear Ms. Browner.

      At the October, 1992 meeting of the SAB's Executive Committee (EC) the
 Board was asked by the Office of Solid Waste and Emergency Response (OSWER) to
 review the methodology for the draft Regulatory Impact Analysis (RIA).  This
 cost/benefit analysis is required prior to promulgation of the Agency's final Resource
 Conservation and Recovery Act Corrective Action (RCRA) Rule. The EC, recognizing
 the importance, complexity, and novelty of OSWER's work and its multi-disciplinary
 character, established an ad hoc RCRA-RIA Steering Committee (RRSC) to assure
 that certain aspects of the RIA  - in both methodology and application - received
 appropriate attention from the relevant SAB committees.

      At a public .meeting on January 29, 1993, the RRSC concluded, on the basis of
 presentations by, and discussions with, OSWER personnel, that four SAB individual
 committees should review the major segments of the RCRA-RIA.  Specifically, the
 RRSC agreed to review: a) the contingent valuation (CV) study used in the RCRA RIA
 analysis (CV-1, by the Environmental Economics Advisory Committee (EEAC)); b) the
 application of CV in the RCRA-RIA (CV-2, by the EEAC); c) the principal fate and
 transport model (MMSOILS). used in the RCRA-RIA (by the Environmental Engineer-
 ing Committee  (EEC)); d) the ecological risk assessment portion of the RCRA-RIA (by
the Ecological Processes  and Effects Committee (EPEC)); and f) the human health
 benefits assessment portion of  the RCRA-RIA (by the Environmental Health Commit-
tee (EHC)).
                                                                   Rtcyctodfltocyc
                                                                   Pi««d M piper Hut ooMInt

-------
      Addressing other aspects of the Charge:

      a)    The Committee does not believe that the pre-testing and survey design
            techniques offer convincing evidence that a well-defined groundwater
            commodity was understood properly by all the respondents.

      b)    The Committee does not believe that any of the three possible methods
            for separating the non-use or passive use values from total values can
            be established as preferred at this time.  Each should be regarded as
            exploratory rather than a tried-and-true method for estimating non-use
            values from survey responses.

      c)    The Committee deems it impossible to judge whether the Box-Cox
            econometric estimates alone provide an acceptable and defensible
            method for dealing with the scenarios and the large bids associated with
            them.

      d)    The Committee does not believe that the approaches for treating embed-
            ding, scenario rejection,  and the potential effects of non-bidding respons-
            es can be assessed for their reliability on the basis of the information
            provided in the report.

      In many  respects, the Committee feels that the problems in  using the study
results to meet  the needs of the RIA effort arise from requirements imposed on the
research. These include the need for both a separate estimate of  nonuse value and
for a method that abstracted from the specific features of the local  conditions associat-
ed with each specific case of groundwater contamination. After reviewing the full
record, the Committee feels that this strategic decision on EPA's part is the cause of
many of the problems with the McClelland study as well as the difficulties of imple-
menting the research findings in the RIA context.  The approaches taken to deal with
these requirements have no basis in the theory of non-market valuation, nor precedent
in practice, and were never subjected to peer review.

      The Committee's report offers specific suggestions for further research to help
resolve the questions raised by this study. Five general areas seem especially
relevant for the  corrective action RIA:

      a)    For the most  part, economic research has focused on developing esti-
            mates of the values of the typical or representative household

-------
      We appreciate the opportunity to review these issues, and look forward to
 receiving your response to our comments.

                                 Sincerely,
                       Dr. Raymond C. Loehr, Chair
                       Science Advisory Board
      Dr. Allen Kneese, Co-Chair           Dr. V.TCerry Smith, Co-Chair
      Environmental Economics            Environmental Economics
      Advisory Committee                 Advisory Committee
ENCLOSURE

-------
                                 ABSTRACT
       The EEAC addressed the design, conduct, and results of the contingent
. valuation study (undertaken for the EPA Office of Solid Waste by Drs. McClelland,
 Schulze, et a/.), focusing on a Charge organized around five major questions: a) the
 survey respondents' understanding of groundwater resources; b) selection of the best
 method for estimating non-use values from the survey responses; c) use of the Box-
 Cox econometric procedure to address large bids; d) the problems of embedding, non-
 bids, and scenario rejection; and e) the applicability of the valuations obtained in this
 study as a basis for EPA to determine the non-use values of groundwater. The
 Committee commends EPA staff for supporting exploratory research of this nature.
 There is little doubt that this study represents a substantive contribution, extending our
 understanding of the issues associated with contingent valuation estimation of non-
 market values. Addressing the last, but most encompassing element of the Charge
 first, the Committee can not endorse the McClelland et a/, study as a means'of
 generating valid and reliable estimates of the nonuse values associated with cleaning
 up contaminated groundwater Specifically, the Committee has no confidence that the
 respondents were dear  about what it is they were being asked to value.  Although the
 study was innovative in a number of respects, this most basic failing gives the
 Committee no choice but to question the validity of the findings. Addressing other
 aspects of the Charge: a) The Committee does not believe that the pre-testing and
 survey design techniques offer convincing evidence that a well-defined groundwater
 commodity was understood properly by all the respondents; b) The Committee does
 not believe that any of the three possible methods for separating the non-use or
 passive use values from total values can be established as preferred  at this time; c)
 the Committee deems it impossible to judge whether the  Box-Cox econometric esti-
 mates alone provide an  acceptable and  defensible method for dealing with the
 scenarios and the large  bids associated with them; and d) the Committee does not
 believe that the approaches for treating embedding, scenario  rejection, and the
 potential effects of non-bidding responses can be assessed for their reliability on the
 basis of the information provided in the report.  The EEAC feels that the problems in
 using the study results to meet the needs of the RIA effort arise from requirements
 imposed on the research by the EPA, including the need for separate estimate of
 nonuse value and for a method that abstracted from the specific features of the local
 conditions associated with  each specific case of groundwater  contamination.  The
 approaches taken to deal with these requirements have no basis in the theory of non-
 market valuation, nor precedent in practice, and were never subjected to peer review.
 The Committee's report offers specific suggestions for further  research to help resolve
 the questions raised by this study, including the criteria for deciding which households
 would be among the groups demanding increases in the  amount available of specific
 commodities or values and study of the sensitivity of CV outcomes to the survey
 methods used

 KEYWORDS: contingent valuation; groundwater; hedonic models; nonuse values;
 RCRA RIA

                                      ii

-------
            UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
                          SCIENCE ADVISORY BOARD
             ENVIRONMENTAL ECONOMICS ADVISORY COMMITTEE
                               RCRA RIA REVIEW

 CO-CHAIRS
 Dr. Allen V. Kneese, Resources for the Future, Washington DC

 Dr. V. Kerry Smith, Department of Economics, North Carolina State University, Raleigh NC

 MEMBERS
 Dr. Nancy E. Bockstael, Department of Agricultural and Resource Economics, University of
 Maryland, College Park, MD

 Dr. A. Myrick Freeman, Department of Economics, Bowdoin College, Brunswick, ME *

 Dr. Charles D. Kolstad, Department of Economics, University of Illinois, Urbana, IL

 Dr. William Nordhaus, Department, of Economics, Yale University, New Haven CT

 Dr. Bryan Norton, School of Public Policy, Georgia Institute of Technology, Atlanta GA

 Dr. Wallace E. Dates, Department of Economics, University of Maryland,College Park, MD

 Dr. Paul R. Portney, Resources for the Future, Washington DC

 Dr. Robert Repetto, World Resources Institute, Washington, DC

 Dr. Richard Schmalensee, Massachusetts Institute of Technology, Cambridge MA

 Dr. Robert N. Stavins, Kennedy School of Government, Harvard University, Cambridge, MA

 Dr. Thomas H. Tietenberg, Department of Economics, Colby College, Waterville, ME

 Dr. W. Kip Viscusi, Department of Economics, Duke University, Durham, NC

 SAB COMMITTEE LIAISONS
 Dr. William Cooper (EPEC), University of Michigan
Mr. Richard Conway (EEC), Union Carbide Corporation
Dr. Morton Lippmann (IAQC), Nelson Environmental Institute, New York University
Dr. Roger McClellan (CASAC), Chemical Industry Institute of Toxicology

DESIGNATED FEDERAL OFFICER
Mr. Samuel Rondberg, Environmental Health Committee, Science Advisory Board (1400F),
U.S. Environmental Protection Agency, Washington, DC 20460
                                       iii

-------
                         TABLE OF CONTENTS
 1  EXECUTIVE SUMMARY	   1

 2  INTRODUCTION	   6
      2.1  Background	   6
      2.2  Charge 	   6

 3  DETAILED DISCUSSION	   9
      3.1  Organization of the Report 	   9
     • 3.2  Outline of Committee's Review Procedures for the Preparation of
           this Report	.^.   9
      3.3  Committee Evaluation and Response to the EPA Charge  	".'... 11
           3.3.1  Comparison with the NOAA Panel Guidelines	  11
           3.3.2 Specific Issues of Concern	  14
                 3.3.2.1  Definition of the Commodity  ..:	  14
                 3.3.2.2  Embedding	  17
                 3.3.2.3  Partitioning of Total Values	  17
                 3.3.2.4  Consistency with Mitchell-Carson Existence Value
                       Study  	  19
                 3.3.2.5  Box-Cox Estimation and Treatment of Zero Bids  ...  20
                 3.3.2.6  Statistical Analysis of Pre-test and Final Samples ...  21
           3.3.3  Responses to Specific Charges	  23

4  CONCLUSIONS AND RECOMMENDATIONS   	  26
      4.1  Summary	  26
      4.2  Recommended Research  	  27

5  REFERENCES	R-1

-------
                        1  EXECUTIVE SUMMARY

    .  The EPA Office of Solid Waste (OSW) has been developing methodologies to
assess the benefits of cleaning up (corrective action) contamination at active hazard-
ous waste facilities operating under the authority of the Resource Conservation and
Recovery Act. OSWs definition of potential benefits includes non-use values of dean
groundwater. The OSW (along with the Office of Policy, Planning, and Evaluation)
has supported a Cooperative Agreement with the University of Colorado to conduct a
contingent valuation study (McClelland et a/. ,1992) to estimate these non-use values.

      The OSW requested that the SAB Environmental Economics Advisory Commit-
tee (EEAC) address the design, conduct and results of the contingent valuation study
with particular attention to methodological limitations that have been identified in the
literature on  contingent valuation.  The Committee was asked to review the study in
the context of the  state of the art for contingent valuation methodology, but to be
aware of the practical constraints under which the effort had been carried out (e.g.,
budget limitations). The Committee devoted three meetings (in whole or in part) to
this review - December 18,  1992, April 30, 1993, and July 13, 1993, completing a
detailed review of the research. The Charge for the EEAC review was organized
around five major  questions  posed by the OSW, and summarized below:

      a)    Did survey respondents understand groundwater sufficiently to give a
            meaningful value to the commodity in question?

      b)    Which of three  similar methods should be used for estimating non-use
            values from the survey responses?

      c)    Is the Box-Cox econometric procedure an acceptable and defensible
            method for addressing large bids?

      d)     Embedding has been identified as a problem in contingent valuation
            studies.  Were the methods used to address the issue of non-bids,
            scenario rejection and embedding  reasonable?

      e)    Are the valuations obtained in this study sufficiently accurate and repro-
            ducible  so as to be used, in part, as a basis for EPA to determine the
            non-use values of groundwater? Can the SAB  advise the Agency on

-------
            analyses of the survey results to improve the estimation of non-use
            values for groundwater?

      Addressing the last, but most encompassing, element of the charge first, the
Committee can not endorse the McClelland et a/, study as a means of generating valid
and reliable estimates of the non-use values associated with cleaning up contaminated
groundwater.  Moreover, the Committee questions the study's applicability within the
context of the forthcoming RCRA Corrective Action Rule Regulatory Impact Analysis
(RIA).  This CV study design flows from the premise that the values people place on
cleaning up groundwater that they are told will have no effect on their own water
supplies can be measured independently of the specific circumstances associated with
each contaminated groundwater resource. The benefits (or value) provided to people
from cleaning up contaminated groundwater are measured by what people'would be
willing to give up to obtain that improvement. With most commodities, these "sacrific-
es" can be measured from market transactions.  For many environmental commodities
this is not possible.  Indeed, in situations where people experience satisfaction from
the knowledge that a resource is restored or preserved irrespective of any direct use
they might make of it, there is a particular need for the type of innovative research
strategies attempted in this McClelland et a/, study. Their approach used survey
methods to ask a representative sample of adult decision makers what they would be
willing to pay to dean up a contaminated groundwater resource. While the survey
approach is not accepted by all economists, it has been accepted  in principle for
natural resource damage assessment by the recent (Federal Register, January 15,
1993) report of the National Oceanographic and Atmospheric Administration (NOAA)
panel of distinguished social scientists (including two Nobel Laureates) composed  of
five economists and a survey researcher.
      The  Committee's concerns about the McClelland et a/, findings relate primarily
to specific details of this study's application  of the CV technique. It was apparent to
Committee members that people answering the study's survey instrument could have
interpreted the services provided by cleaning up contaminated groundwater in a
numberof different, conflicting ways. There is no way to know which of these multiple
meanings these respondents adopted in answering the valuation questions.

      Addressing other aspects of the Charge:

      a)     The Committee does not believe that the pre-testing and survey design
            techniques offer sufficient evidence to ensure that a well-defined ground-
            water commodity was understood property by all the respondents. The
            sample sizes for the pre-tests were not large enough to permit

-------
             multivariate analysis that would allow evaluation of some of the judg-
             ments made by the principal investigators.

       b)    The Committee does not believe that any of the three possible methods
             for separating the non-use or passive use values from total values can
             be established as preferred at this time. Each should be regarded as an
             innovative research technique and not one of proven reliability for esti-
             mating non-use values from survey responses.

       c)    It is not possible to judge whether the Box-Cox econometric estimates
             alone provide an acceptable and defensible method for dealing with the
             scenarios and the large bids associated with them. Comparative evalua-
             tions of these results, and the results of other approaches for dealing
             with the skewness in the valuation responses  across a number of sur-
             veys, should be undertaken before accepting the Box-Cox results.
             Because there are several different hypotheses about the source of large
             bids, the literature does not provide one set of criteria for discriminating
             among these approaches.  Comparative evaluations provide one means
             of evaluating whether the specific approaches used to address skewed
             bid distribution influences the summary statistics used for policy analysis
             or the conclusions drawn or based on survey findings.

       d)    We do not believe that the approaches  for treating embedding, dealing
             with scenario rejection, and reflecting the potential effects of non-biding
             responses can be assessed for their reliability on the basis of the infor-
             mation provided in the report.

       The above problems notwithstanding, the Committee wishes to commend EPA
staff for supporting exploratory research of this nature.  There is little doubt that this
report represents a substantive contribution, extending our understanding of the issues
associated with contingent valuation estimation of non-market values. It  provides new
research insights to the evolving literature on contingent valuation. However, such
innovation is not the criterion that the Committee was asked to use in evaluating the
report, nor the foundation for use of the study by EPA.  In many respects, the Commit-
tee feels that the problems in using the  findings of the study to meet the  needs of the
RIA effort arise from requirements imposed on the research - the need for a separate
estimate of non-use value and for a method that abstracted from the specific features
of the local conditions associated with each  specific case of groundwater contamina-
tion. After reviewing the full  record, the Committee feels that this strategic decision on

-------
 EPA's part is the cause of many of the study's problems as well as the difficulties of
 implementing the research findings in the RIA context.  The approaches taken to deal
 with these requirements have no basis in the theory of non-market valuation, nor
 precedent in practice, and were never subjected to peer review.

       The Committee's report offers specific suggestions for further research to help
 resolve the questions raised by this study.  Five general areas seem especially
 relevant for EPA's activities associated with the corrective action RIA:

       a)    For the most part, economic research has focused on developing esti-
            mates of the values of the typical or representative household demand-
            ing increases in the amount available of specific commodities (or im-
            provements in them) and not the criteria for deciding which households
            would  be among the groups having such demands or values. Address-
            ing this question is fundamental to the task of measuring aggregate
            values for the cleanup of groundwater resources and may well  have a
            much greater quantitative impact on these aggregate estimates than do
            refinements in estimates of  the representative household's values for
            changes in some environmental commodities.1

       b)    Detailed study of the sensitivity of CV outcomes to the survey methods
            used - whether telephone, in-person, or mail surveys is needed. The
            evidence provided to the Committee suggests that there remains some
            questions in the profession  about the results provided by telephone or
            mail surveys in comparison  with in-person interviews. More research on
            this topic is dearly warranted.

       c)    Research addressing issues associated with defining changes in envi-
            ronmental commodities is central to understanding whether the contin-
            gent valuation responses are internally consistent.  The recommended
          '  research would involve developing methods and practical protocols that
            could be used to understand how environmental commodities are best
            measured and how those measures relate to the descriptions offered in
            the framing of contingent valuation questions.
    'Given the extremely fmKed nature of the available reeeareh on the value of groundwater cleanup, the Committee feete that
In thfe caee both typee of research are exceptional/ important

-------
       d)     Research on the need for, and the ability of, respondents separating their
             total value of increases in environmental commodities (such as the
             cleanup of contaminated  groundwater) into use and non-use compo-
             nents.

       e)     Research on the development of methods for gauging the potential
             implications of using a hypothetical setting to elicit valuations in relation-
             ship to real choices is important to the ongoing development of the
             contingent valuation method.  Research evaluating whether or not
             contingent valuation results can be transferred from one setting to others
             is also critical. The Committee believes this should be a central compo-
             nent of the future research that EPA undertakes in support of its pojicy
             evaluation.

      The Committee assumes  that there will be a continuing need to evaluate
analyses intended to appraise household's willingness to pay for improvements in
environmental commodities. There is simply not sufficient information on the proverbial
"research shelf' to address in a  meaningful way many of the questions that have been
posed to the  EEAC about specific decisions in the design of contingent valuation
surveys and in the transfer of results from such surveys.  EPA staff have attempted to
do their best to meet policy needs in the  presence of this limited information.  Their
Charge to the EEAC  reflected a  desire to have the Committee's judgement, based on
its collective research experience, substitute for a documented research record on
these issues. Our conclusions reflect the fact that the problem is not simply that EPA
has been unable to sustain a research program to address these questions.  Overall,
there does not exist a set of research available for the Committee to use in forming its
own judgements,  EPA should begin to sponsor the research required to evaluate the
methods used in its policy evaluations and to develop more experience in valuing
changes to important environmental resources.

-------
                             2  INTRODUCTION

• 2.1  Background

       The EPA Office of Solid Waste (OSW) is working to complete a final rule
 requiring the dean-up of contamination of active hazardous waste facilities operating
 under the authority of the Resource Conservation and Recovery Act (RCRA).  A
 significant area of benefits from the rule is expected to be the dean-up of groundwater
 contamination, as well as other environmental media. To provide dedsion makers
 with a complete assessment of the benefits of corrective action, OSW has adopted a
 broatJ definition of potential benefits that indudes non-use values of dean groundwa-
 ter.

       The OSW has provided support to a Cooperative Agreement with the University
 of Colorado to conduct a contingent valuation study to estimate these non-use values:
 The contingent valuation study is moving toward completion, under the direction of
 Drs. McClelland, Schulze, Lazo, Waldman, Doyle, Elliot, and Irwin.

       The OSW requested that the SAB address the design, conduct and results of
 the contingent valuation study. The researchers were aware of the need to cope with
 four potential sources of error that have been identified in the literature on contingent
 valuation. These sources indude: (1) large bids not indicative of willingness-to-pay;
 (2) scenario rejection or refusals to bid; (3) embedding; and (4) effects of context on
 bids.

       Although the study was conducted to place emphasis on pretesting, instrument
 design, sampling, and econometric analysis of CV survey data, there were also a
 number of practical constraints.  The survey instrument length was limited to insure a
 reasonable response rate from a mail survey. A mail survey was used because
 estimates on a national scale were needed; the sample size had to be large enough to
 allow comparison among survey variants and tests of several key questions. Lastly,
the budget was limited.  The Committee was asked to review the study in  the context
of the state-of-the-art for contingent valuation methodology, but to be aware of the
constraints noted above.

2.2  Charge

      The specific charge was organized around five questions. They are summa-
rized below:
                                      6

-------
a)    The survey instrument design was developed through focus groups using
      a cognitive survey design approach and different survey variants. Two
      levels of context were included among the survey variants to test the
      effect of context.  This was particularly important because groundwater is
      an exotic commodity. There was concern during pretesting that ground-
      water would not be understood well enough by respondents to properly
      give a value. Were the pretesting and subsequent survey design tech-
      niques reasonable methods to use?  Does OSWER have sufficient
      evidence and support, after using these  methods, to insure that ground-
      water was understood enough to property give a value?

b)    The survey was designed to allow estimation of non-use values for,
      groundwater by three possible methods. The three methods produced
      similar results.  Does the SAB have a preference regarding which
      method should be used for estimating non-use values from the survey
      responses?

c)    The study used the econometric technique of Box-Cox transformations to
      address large bids. OSWER believes that the information content of
      large bids needs to be considered in the results. Is the Box-Cox econo-
      metric procedure an acceptable and defensible method for addressing
      large bids?

d)    The survey had to address the  issue of non-bids or scenario rejection by
      using careful survey design techniques.  Scenarios that people would
      accept were pretested extensively. The  cognitive survey design ap-
      proach was instrumental in facilitating the pretesting. Embedding has
      also been identified as a problem in contingent valuation studies.
      Embedding was addressed by posing explicit questions that ask the
      respondent to rethink a previous answer and reduce their bid based on
      how much the values previously given for groundwater cleanup were
      actually for other environmental problems. Were the methods used to
      address the  issue of non-bids, scenario rejection and embedding reason-
      able?

e)    In developing regulations under its several legislative mandates, the
      Agency is required to produce analyses that determine the benefits of
      regulations intended to protect groundwater.  Are the valuations obtained
      in this study sufficiently accurate and reproducible so as to be used, in

-------
part, as a basis for EPA to determine the non-use values of groundwa-
ter? Can the SAB advise the Agency on analyses of the survey results
to improve the  estimation of non-use values for groundwater?
                         8

-------
                         3 DETAILED DISCUSSION

 3.1  Organization of the Report

       The Committee's report on the review of this contingent valuation study is
 organized into three sections.  In the first section we describe our evaluation process.
 We include this section because the steps we took in our review are quite relevant to
 our judgment on the report

       In the second section, we discuss the implications of the NOAA Contingent
 Valuation Panel's Report for the evaluation of McClelland, et a/., as well as several
 issues that were not addressed in the charge. We summarize our conclusions-with
 respect to each of the issues identified in the charge. And we present our overall
 reactions to this report.

       In the third section, we identify three major areas of substantive research
 concerning the use of contingent valuation methods for the valuation of nonmarket
 resources such as groundwater. We recommend that EPA initiate a program of
 research in each of these areas.

 3.2  Outline of Committee's  Review Procedures for the Preparation of this Report

       The McClelland et at. contingent valuation study of groundwater reviewed was
 prepared as part of the research necessary for EPA's Regulatory Analysis for its
 Corrective Action Rule for RCRA sites. Our review was undertaken in several stages.
 A detailed review was required  because the Office of Solid Waste did not provide any
 information indicating that the McClelland et a/, contingent valuation report had been
 subjected to a peer review process prior to submitting it to the Committee.2  Indeed, it
 is our understanding that the research design for the benefit transfers planned as part
 of the Regulatory impact Analysis was not subjected to a comprehensive external
 review. We believe this was a mistake.   It had a significant effect on the design of the
 McClelland et a/, research, and therefore we will return to it below.  In the absence of
 a documented peer review of the draft final report, the Committee believed it was
 essential to  conduct a detailed peer review of the research as well as to evaluate the
 issues associated with the Charge given to the Committee.
  ^Subsequent to the preparation of this report, peer review material* were provided to one of the Committee Chair*. These
material* addre»*ed an interim report, not the draft final report. Moreover, there. wa« no indication of how the research design
was modified to respond to the comments.

                                       9

-------
      The first stage of our review consisted of a briefing on December 18,1992 by
EPA staff on the goals specified for the contingent valuation study. In this presenta-
tion EPA staff outlined its needs in relationship to the Regulatory Impact Analysis.
The staff identified a pre-defined set of research goals that were imposed on the
principal investigators and that conditioned their decisions concerning research
strategies. One of the goals was to obtain an estimate of the non-use values (or
"passive use" value, using the terminology of the 1989 Court of Appeals decision and
the NOAA Contingent Valuation  Panel Report) for the cleanup of groundwater
resources that could be applied as a separate component to any use values estimated
separately on a site specific basis. It is our understanding, based on that briefing, that
thi*goal was specified by EPA staff.  It was not based on a recommendation of the
principal investigators as a component of their research design. Nor was it a goal that
was developed as a result of existing research findings on the measurement of use
and non-use values (i.e., the earlier Mitchell-Carson (1989) study using focus groups
to evaluate how people conceived of groundwater resources and whether they might
have non-use values for them).

      Following the overview of EPA goals, the Committee Co-Chair (Smith) present-
ed an overview of contingent valuation methods and an initial summary of the key
elements in the McClelland et a/, contingent valuation survey. Because the Commit-
tee was aware of the ongoing NOAA Contingent Valuation Panel's assessment of the
CV method, the Committee decided to wait until the report of that panel was available
before undertaking a detailed review of the McClelland et a/, study (The report was
subsequently published in the  Federal Register for January 15,1993).

      The second stage of our review process involved assembling a detailed set of
questions  for the principal investigators of the CV study.  These questions summarized
the issues that arose in the Committee's initial review of the McClelland et a/, study.3
The questions posed to the authors also outlined the concerns that the Committee felt
needed  to be addressed by the principal investigators in a subsequent meeting.

      As part of the preparation of the equivalent of a peer review, the Committee
requested three sets of consultative activities.   First, the Committee asked Dr. Robert
C. Mitchell to evaluate the background information forming the basis for the design of
the contingent valuation survey.  Mitchell, along with Dr. Richard T. Carson, had
conducted for EPA qualitative  research on the issues associated with attempting to
   3Th« qu»*ttont were trantmtted to the author* In a memorandum from Kerry Smith and Aten Km«M to WBfam Schutzt
January 11,1993.

                                      10

-------
 measure non-use or existence values for groundwater prior to the McClelland et al.
 report cited above).  The McClelland et a/, report indicates that it intended to build
 upon the findings of the Mitchell-Carson study. The charge given to Mitchell (Smith
 and Kneese memorandum, March 23,1993) proposed that his review focus on the
 interpretation given to this earlier Mitchell-Carson work by McClelland et al.

       The Committee asked Drs. Richard Bishop and Gregory Poe to evaluate the
 implications of their recent study of the influence of information on individual's valua-
 tion of groundwater cleanup. These researchers were asked to consider their findings
 in relation to the information that was provided to respondents in the McClelland et al.
 contingent valuation study.  Finally, Dr. Kevin Boyle was asked to prepare a detailed
 review of the literature including the McClelland et al. study. This review was to-
 include a comparative evaluation  of all aspects of the earlier literature on the contin-
 gent valuation method to value cleanup of groundwater resources (K. J. Boyle,  1993).

       The third stage in the Committee's review process involved a second meeting
 of the Committee (a) to: discuss the report of the NOAA Panel on contingent valuation
 methods; (b) to allow the principal investigators to describe their research design in
 more detail and to provide detailed responses to the January comments and ques-
 tions; and (c) to hear the reports of the three groups of commissioned consultants to
 the Committee. Following this meeting, a draft report of the Committee was prepared
 and circulated to the Committee members.  The Committee's third and final meeting
 evaluated and revised this report  and finalized the Committee's recommendations
 about the study. During the final  meeting, Dr. William Schulze presented some
 clarifying information to the Committee. A summary of the authors' written responses
 to Committee questions (McClelland, Schulze, and Lazo,  June 23, 1993) was circulat-
 ed shortly before the final meeting.

 3.3  Committee Evaluation and Response to the EPA Charge

  3.3.1 Comparison with the NOAA Panel Guidelines.

      The  Committee decided to  delay its review of the McClelland et al. report until
after the NOAA Panel's report on  the contingent valuation method was available.  This
was done because the Committee did not have the resources or time to undertake its
own review of the CV method.  The NOAA Panel concluded that "contingent valuation
studies can produce estimates reliable enough to be the starting point for a judicial or
administrative determination of natural resource damages - including lost passive use
value."  After reviewing the objectives of the NOAA Panel and its recommendations,
                                      11

-------
 the EEAC acknowledges that the mandate requested of the NOAA panel differs from
 that associated with using the results of a CV analysis for general policy purposes.
 Moreover, the Committee does not intend to endorse or agree with all aspects of the
 NOAA report  Rather, we have simply used the NOAA Panel's guidelines as one of
 several starting points for an evaluation of the McClelland et al study.

       The NOAA Panel's guidelines for the design and implementation  of CV survey
 methods included4: use of in-person interviews; use of a referendum (or discrete
 choice) format in asking contingent valuation questions; obtaining a high response rate
 to the questions; investigating the responsiveness of the "willingness to  pay respons-
 es" to changes in the amount of the environmental commodity offered to respondents;5
 attempting to determine whether respondents understood the tasks, considered their
 budgets in the process of answering contingent valuation questions, and believed the
 scenario presented to them.  With discrete choice questions, the NOAA Panel
 recommended a follow-up to any yes/no questions with specific attempts to identify
 responses that indicated concern over available resources to make the stated pay-
 ments.  Scenarios should  demonstrate that respondents considered seriously the
 private and public substitutes for the commodities offered as well as these budget
 constraints.

       Because the McClelland et al. study was conducted long before the NOAA
 report was available, it is unrealistic to expect a direct correspondence between the
 procedures used in the McClelland et al. report and the NOAA Panel's recommenda-
 tion. It also should be noted that not all of the members of this Committee fully
    These comments paraphrase the specific requirements identified In the NOAA Pane's summary of conditions required to
satisfy their burden-of-proof requirement They also include the recommendations made about specific implementation proce-
dures. The specific cftabon tor text of the report is: federa/ ffegister. January 15,1983. VoL 58, f 10, pp. 4601-4614.

    5 The NOAA Panel report uses the term 'scope* to refer to changes In the environmental commodities that are intended to
represent the- injuries to one or more specific environmental resources. K tt identified ae 'scope* In their burden of proof
requirements and that specific patterns of responsiveness In wBngness to pay acknowledges that the predictions torn economic
theory about the properties of the wfflngness to pay are lotted. This is especialy true when there may be more than one
environmental resource affected and the reductions hi Injuries (Le., Increases In the environmental commodities Involved) that are
being valued may be different across resources. The specific text of (he  NOAA Paners report that seems to be identified as
being associated win the scope requirement describe* the condUon as blows: fiaffonalry in fe weakest form requires certain
Und* of consistency umong cnofce* made by MMduafe. for instance, t tn MMdutl cnooss* some purcnases af a gtvon aef of
price* and Income, tout I some prices M and (her* are no other changes, th» good* 1h»t th* MMdiMl woutt now buy m»k»
Mm or ner Better off. Slmfany, we would expect an MMdutf* preference* over pubic good* fie. btfdgo*. nfenwsys, a* ouaify)
to reflect me same Und e/consistency.

       Common notion* ofnOoniKy Impose other rvowre/nenb which tn relevant In dMarsnf contexts, t/susfy, ttough not
a/ways, ft is /MsenaMe to suppose fief more of something i^fMt* good *b»lt»r to king** in MMdu»l I* not a»1*ML
Thh *i gene/i/^/issued1 silos wmngnou to pey somewhat mo/e fermcye of a good t* Judgod by fte MMdutL Also, JT
me/gine/orJncramento/wff*vnesstopeytoredkflfone/a/noestfeclrw wMdemountalrMoVaivisMe. l(Isusuajy/K>(
rseaoneMe to assume thef* decfnes very ateuptV (emphasis added) pp 4604.

                                           12

-------
 endorse all of the NOAA Panel's guidelines. Nonetheless, the EEAC started its
 evaluation by considering the McClelland et al. study's design in relationship to the
• NOAA Panel recommendations.  Several notable differences were identified.
 McClelland et al. used mail surveys and not in-person interviews, a fact that was a
 concern of some members of the Committee. The interview format did not allow
 evaluation of whether the questionnaires were understandable to respondents of the
final survey and did not incorporate a test for the effects of changes in the scope of
 the commodity offered respondents, as recommended by the NOAA panel. While
 considerable attention was paid to a specific set of substitutes, the set of substitutes
for contaminated groundwater was  narrowly defined and the set of substitutes
 available with scenario rejection was not made dear. The NOAA Panel did not
 discuss the procedures employed in the McClelland et al. report for dealing wjtb-
 disembedding and for partitioning the  total willingness to pay into use and passive use
 components. Additionally the study did not utilize the close-ended or referendum
format for the contingent valuation questions, using instead a payment card approach:
 While the discussion of the pre-test results suggests that a closed-ended contingent
 valuation was evaluated, the questions used were not in the same format as recom-
 mended by the NOAA Panel (based on information given in the report, the principal
 investigator's comments, and the Boyle (1993) review).  The authors' pre-test did not
evaluate the discrete choice approach recommended by the NOAA Panel.  Thus, the
questioning format did not correspond to that recommended by the NOAA  Panel.

       Overall, the McClelland et al. study does not satisfy the burden-of-proof
standards described by the NOAA Panel for reliable CV assessments of the passive
use values associated with natural resource damages.  It is important to acknowledge,
however, that the NOAA Panel report  did not describe these features of CV studies as
absolute criteria for reliability. The report acknowledges that reliable estimates could
be realized without full adherence to all the recommendations. As the NOAA Panel
report dearly indicates, it is possible for CV studies to achieve comparable levels of
reliability without adhering to their recommendations, but the burden of proof must be
satisfied" by other means. Accordingly, given the Committee's position that it does not
necessarily endorse the NOAA Panel's recommendations, our evaluation of reliability
and validity was based on the general definitions of these concepts as they have been
used in the CV literature (R.  C. Mitchell and R. T. Carson, 1989). As such, this
evaluation is a professional judgment that took the comparison with the  NOAA criteria
as an approximate template in the Committee's first stage review of the  McClelland et
al. study.
                                      13

-------
       The McClelland et al. report exhibited several commendable features. Although
 it used mail questionnaires, it did achieve a relatively high response rate (63.4 percent
.of total questionnaires were returned; 43.7 percent of the total were available for
 regression analysis).  The report included an extensive amount of qualitative analysis
 associated with the design of their survey instrument and several pre-tests to evaluate
 its performance. At the early design stage of questionnaire development,  there was
 extensive attention to verbal protocol analysis, focus groups, and other qualitative
 approaches to evaluate the extent to which individuals were comprehending the
 commodity that was being offered to them.  Equally important, a series of  pre-tests
 were used to evaluate the sensitivity of responses to the structuring of the valuation
 questions. While concerns were raised about the relative sizes of the samples
 underlying these pre-tests, the Boyle (1993) appraisal and other reviewers suggested
 that this amount of work was among the  most extensive ever conducted in a contin-
 gent valuation study for groundwater cleanup.  Unfortunately, there does not appear to
 have been evaluation of the final survey instrument to determine whether the judge-
 ments from the pretests were consistent with an independent set of respondent's
 understanding of the CV questions.

  3.3.2 Specific Issues of Concern

      The Committee found difficulties with several elements of the design and
 analysis presented in the McClelland et al. report.  Some of these are highlighted in
 brief terms in our summary judgment.  We discuss  a few of them in more detail below.
   3.3.2.1  Definition of the Commodity

      The first of these concerns the definition of the commodity to be valued by
respondents. The Committee found that there were multiple interpretations of what
was offered to respondents based on the information conveyed by the questions
comprising the valuation task. The most direct explanation of the Committee's
reasons for concern stems from the reactions of Committee members to the question-
naires. Careful readings of the survey instrument by several members of the Commit-
tee resulted in different interpretations of the commodity that was being valued.  These
differences (along with similar questions about the survey instruments associated with
the experimental variations included as different versions) raised questions about the
ability of lay persons to understand the nature of the commodity they were being
asked to value.
                                      14

-------
       The Committee found that the sample sizes and evaluation of the information
 materials and pre-testing and design techniques were not sufficient to statistically test
 whether respondents had enough information and understood the commodities offered
 to them.  Equally important, it was not possible to discriminate between the various
 interpretations of the commodity offered to respondents. In order to present a
 comprehensible choice to respondents, the CV survey instrument must dearly specify
 the nature of the environmental commodity they will receive if they choose not to
 purchase what is offered and the modifications to that commodity, the access condi-
 tions and the nature of payment(s) if they choose to purchase what is offered. The
 no-purchase condition will be referred to as the default condition and all aspects of the
 change in the commodity as the commodity change.

       There is a basic problem in interpreting  the valuation question (Q11), p'rimarily
 associated with an ambiguity in the problems that can arise from  the contaminated
 water. In the base scenario, forty percent of the water used by the  community was
 described as coming from groundwater contaminated as a  result of  a leaking public
 landfill, in the discussion of the context for the commodity, contamination increased
 the risk of dying from cancer by about ten additional deaths per million per year
 among people who drink the water.  An event with comparable risk was described for
 respondents.6 However, it is subsequently stated that the water must not be used for
 drinking and cooking. Thus,  individuals may not perceive health risk to be a feature of
 the default conditions of water supply and risk  perception may not enter into the
 individual's decision process.  In this case the only implication of contamination is the
 possibility of water shortage.

       In fact, it is not dear what respondents view as the default scenario for their
 valuation. That is, in order to place a value on  the primary scenario  of complete
 deanup, respondents must dearly understand  the reference point for their valuation,
 that is, the condition of their water supply if complete deanup did not occur. This is
 important because acceptance/rejection of a proposed dean-up plan relative to a
 default option characterizes the commodity purchased.  The scenario could be
 interpreted as describing an increased risk and the prospect that a substantial fraction
 of their water supply would be contaminated. But since the scenario also specified
that it could  not be used for drinking  or cooking, it is not dear that those answering
this question would perceive any risk as being  present Furthermore no information
was provided on alternative supplies or response patterns.  Other options are  de-
scribed before the contingent valuation scenario.  But there is no statement about
   HTie term risk to uMd here to mean the Metime probability of death.

                                      15

-------
whether these alternatives would be able to be used for the particular groundwater
contamination being valued.  Under the most optimistic interpretation of questions Q6
through Q10, individuals might have selected one of four alternative base cases to
define the default scenario in considering their valuation of complete cleanup. The
default scenario constitutes the reference point for their valuation estimate.  The
difficulty is that we do not know which of these alternatives (if any) would function as
the method of choice for any particular respondent would choose in the event that
they did not select complete cleanup.

       In the base survey instrument, the respondent must somehow judge whether
the uncontaminated 60% of the water supply will be adequate.  There seems to be an
implicit assumption that the community must find an additional water source. .This
implication appears to arise from Q10 and is not stated elsewhere.  If the respondent
perceives a water shortage, all that is being asked is the demand for water.  Of
course, this interpretation makes the further assumption that respondents have
accepted the conclusion offered in the instrument that risk has been precluded by the
water policy in effect.

       Respondents' satisfaction ratings for each of the alternative approaches (i.e.,
containment,  public treatment, home treatment, or water rationing)  do  not help resolve
the ambiguities because they do  not provide a complete description of the adjustments
to their values associated with each of these reference points.  Versions C and D of
the questionnaire allow comparison of public treatment with  complete treatment, and
Version D elicits valuation adjustments in response to varying percentages of domestic
water supply contamination.  Nonetheless, this would not allow us to completely
decompose (by scenario comparisons) the values that were offered for anyone who
did not receive one of these questionnaires.

      The method of eliciting valuation estimates from some versions of the question-
naire as a percentage of the complete cleanup valuation caused some concern.
When individuals are offered changes in the scenario, they are offered adjustments to
the base case (i.e. a complete  cleanup scenario) with specific percentages of value
offered as the responses.  Where we have  a priori expectations that the values given
should be larger, the percentages are scaled at a larger rate; where there are a priori
expectations that the values should be smaller, the percentages are scaled over a
smaller range. This formatting  of the questions preconditions responses that individu-
als can give.  The literature provides no basis for judging the reliability of these
percentage adjustments.  Moreover, there does not appear to have been a test of
                                      16

-------
 using alternatives to determine whether the formatting influenced the plausible
 response.

   3.3.2.2  Embedding

      Another source of concern is the question associated with disem bed ding.
 Embedding is a term that has had a variety of interpretations in the economics
 literature, with widespread use of the concept generally associated with research
 reported by Kahneman and Knetsch (1992). In the context of this report, we have
 interpreted embedding as a situation where survey respondents report values for
 commodities that are more inclusive than what is desired by those analysts or policy
 makers intending to use their responses. This can arise because respondents...
 interpret the commodity differently from the analyst or because they assume that other
 commodities would be provided along with the one that has been offered (even though
 the survey instrument does not imply this to be the case).  The McClelland et al.
 disembedding question (Q12) asks individuals to reconsider the dollar amount they
 stated (in response to the preceding question,  Q 11) they would be willing to pay for
 complete groundwater cleanup and asks them  to characterize this amount as: (a) "Just
 for the stated groundwater program;" (b) "somewhat for the groundwater program  and
 somewhat a general contribution to all environmental causes;" (c) "basically a contri-
 bution to all environmental or other worthwhile  causes;" or  (d) "other." In question 13,
 a percent of a dollar amount is requested.  If individuals are asked in question 11 to
 describe the most they would be willing to pay  each  month on top of their current
 water bill for each of the next ten years for complete groundwater cleanup program,
 the Committee wondered why they would immediately thereafter be willing to admit
 that this response was actually for other things. It would be difficult to interpret how
 any household would respond to this question if they did not know it in advance, i.e. if
 the questions were conducted at an in-person interview or a telephone interview where
 respondents do not know what is coming next in the question sequence.  When these
 questions are presented in a mailed questionnaire (where it is assumed that respon-
 dents can read ahead), the response to this type of question is even more difficult to
 interpret and  there is no basis in the literature for assessing this  issue.  Thus the
 methods used in this study to adjust for embedding have not been evaluated in the
 literature.

   3.3.2.3 Partitioning of Total Values

      One of the objectives of the study was to estimate the non-use values of
cleaning up groundwater contamination. These estimates were to be used in the
                                      17

-------
benefit assessment required for the Regulatory Impact Analysis of RCRA corrective
action rule.  The Committee concludes that the report does not provide evidence that
respondents adequately distinguished the use and non-use values for cleaning up
contaminated groundwater. Three methods were used to estimate non-use values:
direct questions requiring allocation of total bids over composite categories; compari-
son of "willingness to pay" (WTP) responses for different commodity scenarios; and
extrapolation by fitting an assumed functional form to WTP bids for different levels of
water shortage.

      The first method was implemented through Question 14 which asked respon-
dents to decompose their complete groundwater cleanup bid into motivational catego-
ries.  A limited number of investigators have used this approach to isolate use .and
non-use values, but there is no evidence in the literature about the method's perfor-
mance.7  The maintained hypothesis that respondents to CV questions can decom-
pose their bids into constituent parts is difficult to accept and there is little evidence to
support its use. For example, it seems reasonable to suggest that people would have
difficulty decomposing their willingness to pay for something as familiar as an ice
cream sundae into percentages they would associate with texture, temperature, looks,
taste, etc. At the  minimum we would expect that this type of detailed information
would need  to be elicited very carefully - in effect "coaxed" from them in a very
detailed but logical way. The reliability of the second method is questionable due to
confusion over the definition of the commodity and default scenarios, as discussed
above. With regard to the third  method, we have no past experience with the relation-
ship between this method and the tasks requested of respondents, nor any reason to
believe that  the intercept of a fitted function can be interpreted as a non-use value.
Finally, there is little basis provided in the study for gauging the transferability of such
results from  other situations requiring the estimation of non-use values for cleaning up
contaminated groundwater resources.

      By  criticizing the techniques used in this study to separate use from non-use
values7~the Committee does not intend to address the broader issue of the validity of
that distinction. Some members of the Committee questioned whether it is appropriate
to ask respondents to partition their willingness to pay into discrete elements that were
associated with components of those respondents' total values. This should  be  a
matter for further research.
   7S«« Boyto (1893] for dtecuMfcn and tvakjatton.
                                      18

-------
   3.3.2.4  Consistency with Mitchell-Carson Existence Value Study

      The Committee found some inconsistency between the lessons learned from
the earlier Mitchell-Carson evaluation of existence values as reported by Mitchell and
the McClelland et a/, interpretation of these results. For example, McClelland et al.
summarize the objectives of their study:

            The aim of our current study is to estimate non-use values for groundwa-
            ter cleanup. This commodity, of great interest to the U.S. EPA, also
            appeared to be ideal fora methodology study since in early development
            work undertaken for U.S. EPA by Mitchell and Carson, it was apparent
            that (1) people were generally poorly informed about groundwater. con-
            tamination, and (2) people resisted non-use scenarios used for evalua-
            tion in which groundwater was preserved but never used.  In other
            words, the scenario was rejected by respondents.  Delighted with our
            exotic commodity, groundwater cleanup, our strategy was to apply two
            new methods in designing this survey instrument (page 22).

      By contrast, the Mitchell-Carson report concluded somewhat differently from
      what the authors indicated.  It notes:

            We believe it is possible to use the contingent valuation method to obtain
            a credible dollar measure of the existence values of groundwater by
            using a scenario of the type described in this chapter. The device of
            having people evaluate hypothetical GNN (groundwater not needed for
            human use) groundwater (aquifer)  that can be plausibly isolated from
            other aquifers and which lies at a great distance from those who are
            being interviewed should effectively eliminate any use values.  The
            concrete barrier plan  promises to provide a credible protective option for
            the GNN aquifers so that those who choose it will be stating how much
            they would be  willing  to pay to preserve the aquifer defined as GNN from
            contamination. These containment  features should minimize any influ-
            ence on people's willingness to pay that might result if they continue to
            believe that contamination in groundwater travels at a much higher
            velocity than is actually the case.... Given the difficulty of convincing
            people that the aquifer will never be needed for human use, a portion of
            the values will include the utility people get from vicarious protection.
            The vicarious protection values will be minimized by the scenario fea-
            tures that are intended to protect others from inadvertently  using ground-
                                      19

-------
             water in the basin.... There is no valid way to obtain separate mea-
             sures of the several types of existence value to this study.  First, it is too
             difficult to overcome people's belief about future use by others to design
             a scenario that would only capture stewardship values. Likewise, we see
             no way to design a scenario that would only capture bequest or inherent
             values for groundwater. Second, it is cognitively unrealistic to ask
             respondents to state what proportion of the total value they ascribe to
             each of these three types of existence values.  However, it will be
             possible to assess in a qualitative fashion the degree to which respon-
             dents are influenced by these several types of value by using the follow-
             up motivational questions (pp 83-85, emphasis  added).

      Both studies identify  difficulties in separating use and non-use values.  In part
because of the Mitchell-Carson work, and in part because of their own pre-test and
verbal protocol analysis, it was dear to the principal investigators that it would be
extraordinarily difficult to explain the nature of groundwater resources that would not
have any foreseeable uses to them or other people, to respondents. Mitchell and
Carson  recommended a separation by distance as well as an identification of other
aquifers to try to deal with this problem.  McClelland et a/, adopted a different frame-
work. Concern about substitutes in the form of alternative aquifers and about the
importance of describing plans for protecting groundwater that would be believable to
respondents did not appear to influence the way in which McClelland et a/, structured
their scenarios. The allocation format used in the McClelland et al. report does not
conform to the follow-up motivational questions as described by Mitchell and Carson.

   3.3.2.5  Box-Cox Estimation and Treatment of Zero Bids

      The Box-Cox regression methods analyzed what is referred to as the "reduced
willingness to pay," the willingness to pay bid for complete groundwater cleanup
adjusted by the reported disembedding percentage. The results reported by
McClefland-SchuIze in the second of the Committee meetings suggest that the Box-
Cox modelling approach used to deal with a skewed bid distribution was quite
sensitive to the authors' treatment of zero bids.  The magnitude of the  Box-Cox
parameter exhibited substantial variation with the substitutions imposed for the zero
values of the willingness to  pay bids. Despite the sensitivity to zero values, there was
no discussion of the treatment of zero bids or scenario rejection in the  present report.
Moreover, the Box-Cox transformation is only one of a number of ways of dealing with
skewed  error distributions and no comparative evaluation of the methods was under-
taken. Thus, the Committee concludes (in response to the third question of the
                                      20

-------
 Charge) that the report does not provide clear-cut evidence.that the Box-Cox proce-
 dure adequately addresses the issues posed by large bids, zero bids, and scenario
• rejection.  In addition, an evaluation of the zero responses, non-responses, large bids,
 and non-responses to the valuation scenarios in total are essential for judging the
 reliability of the analysis. The  existing contingent valuation literature offers an array of

 possibilities for dealing with each of these issues, and no single method has  been
 identified as preferred. In view of this work, it seems reasonable to suggest that
 comparative evaluations of the different methods for treating outlying observations,
zero responses, and non-responses would be important for gauging the comparative
 performance of the contingent  valuation scenarios offered in this study.

       The Committee recognizes that there are  a variety of statistical methods that
can be used to address large bids, zero bids, and other anomalies. For example, one
might employ some robust regression approach.  Using a statistical procedure to
address extreme observations  does not, however, solve the more fundamental
problem of why such extreme responses arose.  Do these responses reflect legitimate
heterogeneity in valuations, or  do they reflect a failure by respondents to understand
the survey and give meaningful responses? Choice of the appropriate statistical
solution depends largely on the factors generating these extreme responses. As a
result, the  Committee views these extreme bids as a matter of continuing  concern that
would warrant further research in future studies.

   3.3.2.6  Statistical Analysis of Pre-test and Final Samples

      The design of the McClelland et al. survey instrument was intended to include a
set of activities that would evaluate the amount of information that respondents
needed to value groundwater cleanup. These activities included the verbal protocol
analyses and pretests conducted in the initial stage  of the research to permit judg-
ments about the final success of the form of the  survey instruments in communicating
the information people required to decide about a groundwater cleanup policy offered
to them. Because the pretests played a key role in the survey instrument's design,
the Committee considered the nature of the empirical analyses undertaken to develop
the conclusions on the instrument used in the final survey.

       The empirical analysis included summary statistics, primarily means and
standard deviations, and some frequency plots of responses to the willingness to pay
questions against a few of the attitudinal variables for the pre-test and full  samples
together. There were also comparable statistics  for  a variety of subsets of the full
                                      21

-------
 sample. There was little multivariate analysis of the other willingness to pay ques-
 tions. While there is a much smaller sample associated with each of these variations
. in the questions, a comparative evaluation of multivariate functions estimated for each
 of the willingness to pay responses using the final sample would contribute information
 to help assess the reliability of the valuation estimates,  it would provide indirect
 evidence on the effects of changes in the scope of the commodity; for example, larger
 percentage shortfalls would be expected to be associated with greater use-related
 motives, where scenario changes from complete cleanup to containment or public
 treatment would be regarded as smaller amounts of the "cleanup" commodity.

     * Table 7.5 in the McClelland et a/, report provides suggestive evidence, both in
 the  full sample and the regression sample, of a response to variations in the com,-
 modity design . However, it is not dear whether these differences are associated with
 the  actual characteristics of the commodities or with particular features of the sub-
 sample (i.e. lower income level, socio-economic status, or other factors), so that these
 factors would need to be taken into account in appraising whether these were
 consistent differences in the valuation responses across scenarios.  The McClelland et
 a/, evaluation of the effects of changes in the commodity (across the versions of the
 survey questionnaire) did not independently evaluate the change in respondents'
 willingness to pay to the features of the commodities. It was conditioned on accepting
 the  Box-Cox model (and its predictions) as a maintained hypothesis. These predic-
 tions were then used, together with the percentage responses to commodity changes
 elicited from each respondent based on which  version of the questionnaire they
 received. No attempt was made to model the adjustments and  to incorporate this in a
 broader description of the adjusted valuations.  The Committee  believes that such
 models, both for the base case valuation and for other valuation responses that relate
 them to the characteristics of the respondent, would provide indirect evidence of the
 reliability of the survey methods.

       Concerns were also raised with respect  to the responsibility variable (Q 15).
 While trie" briefing from the principal investigators suggested that the  results are not
 markedly changed by removing this variable from the willingness to pay  models, the
 performance of the variable as a gauge of acceptance of the scenario and its relation-
 ship to other characteristics of respondents could be quite important

       More complete treatment of the sub-samples that include a) individuals living in
 areas with sites on the National Priority List; b) individuals who  do not receive water
 bills  - i.e. individuals with private wells; and c) individuals with varying experience with
 groundwater contamination, would offer opportunities for additional insights into the
                                       22

-------
 reliability of the responses. Because this work was not undertaken as part of the
 analysis of the survey and remains to be done, it is difficult to reach an overall
•conclusion on the reliability of the willingness to pay responses interpreted as total
 values for groundwater cleanup. Preliminary analysis undertaken by Committee
 members suggests that there may be further insights from this type of empirical
 evaluation.

  3.3.3 Responses to Specific Charges

      The Committee's response to each of the specific questions associated with the
 charge is as follows:

      a)    We do not believe that the pre-testing and survey design techniques
            used in this study offer sufficient evidence to ensure that a well-defined
            groundwater commodity was understood properly by all the respondents.
            The results of these analyses support the conclusion that groundwater
            issues were important to respondents. However, the sample sizes for
            the pre-tests were not large enough to permit multivariate analysis that
            would allow evaluation of some of the judgments made by the principal
            investigators.  In particular, the process of reducing the information set
            that was provided to individuals, the decisions on eliminating the risk
            information,  the selection of a valuation payment card over alternative
            methods, and the definition of the commodity itself were not sufficiently
            tested in the pre-test and verbal protocol analysis in a way that would
            enable us to judge whether these were the most understandable ap-
            proaches for dealing with these issues.

      b)    The Committee does not believe that any of the three possible methods
            for estimating the total willingness to pay for non-use or passive use
            values can be established as most preferred on the basis of the evi-
            dence that is currently available in the.literature.  This is not to suggest
            that the methods are incorrect or that they might not ultimately provide a
            basis for estimating non-use values in some cases.  The evidence to
            date however provides no basis for judging them.  They should be
            regarded as innovative research techniques and not verified methods for
            estimating non-use values from survey responses.

            The Committee remains skeptical of the method of asking respondents to
            decompose reported total values into use versus non-use or various
                                      23

-------
       motivational categories. These procedures have not been verified in the
       literature. There fore we recommend the principal focus of the analysis
       be on the total willingness to pay estimates for the complete containment
       scenario only.

c)     The Box-Cox econometric transformation as applied in this study is one
       of several methods for dealing with large bids and outliers. The esti-
       mates appear sensitive to the treatment of zero bids.  Comparative
       evaluations of other approaches for dealing with the skewness in the
       valuation responses should be undertaken before accepting the Box-Cox
       results.  In the absence of such comparative evaluations and detailed
       analysis of all the possible outcomes of the valuation questions, it is.not
       possible to judge whether the Box-Cox econometric estimates alone
       provide an acceptable and defensible method for dealing with the sce-
       narios and the large bids associated with them.

d)     We do not believe that the approaches for treating embedding, dealing
       with scenario rejection, and reflecting  the potential effects of non-bids
       can be assessed for their reliability on the basis of the information
       provided in the report  The procedures described in the McClelland et al.
       report are research innovations that have not been evaluated in the
       literature and were not evaluated as part of their study. Further,
       multivariate modelling of determinants of the disembedding responses
       and of the factors influencing whether individuals provided  no bids,  zero
       bids, or large bids would be essential  to answer the fourth  question in
       the EPA charge. Because this information was not provided as part of
       the report, it is impossible for the Committee to make the judgment
       required to address question four of the charge.

e)     The valuation responses in the survey for non-use value have not been
     '  demonstrated to be accurate and reliable enough  for the EPA to estimate
       the non-use values for groundwater contamination.  It is not dear that
       any of the three decomposition methods  provides a basis for estimating
       non-use values accurately and in a reproducible manner.  The literature
       does not provide specific documentation  indicating that these methods
       are capable of separating the total values, especially for a  situation
       where the commodity definition involves water rationing, risk, and
       groundwater protection as part of the  complete containment scenario.
       Disaggregating the components of the total value  on the basis of motives
                                24

-------
alone would not necessarily assure that the values are exclusively
associated with non-use values or that they could be transferred to
another hypothetical situation with distinctly different features.

Improving the estimates of non-use values will require a composite
scenario that includes the recommendations earlier provided by Mitchell-
Carson along with the insights provided in this study. That is, it will
require an evaluation that attempts to control the respondents' percep-
tions of uses for the groundwater by identifying other sources for their
water and localizing the effects of contamination to deposits that were
not now (or not thought to be) likely resources for future use.  Because
these alternatives were  part of what was recommended from the Mitch-
ell-Carson report and were identified as possible alternatives in the Boyle
(1993) summary of existing literature, it seems they should have been
part of the experimental design.
                          25

-------
               4  CONCLUSIONS AND RECOMMENDATIONS

• 4.1 Summary

       it is possible to evaluate the report from a number of different perspectives.
 From the perspective of contributions to methodology and the practice of contingent
 valuation analysis, there is little doubt that this report represents a substantive
 contribution, extending our understanding of the issues associated with contingent
 valuation estimation of non-market values. Its progressive development of information
 treatments, innovative experimental design, and attention to comparative analysis of
 the values provided by alternative methods of separating use from non-use or passive
 use values should be treated as new research insights to evolving literature on	
 contingent valuation. However, this is not the criterion that the Committee was asked
 to use in evaluating the report In many respects, the Committee feels that the
 problems in using the study results to meet the needs of the RIA charge arise from
 requirements imposed on the research - the need for a separate estimate of non-use
 value and for a method that abstracted from the specific features of local conditions.
 After reviewing the full record, the Committee feels that this strategic decision on
 EPA's part is the cause of many of the study's problems as well as the difficulties of
 implementing the research findings in the  RIA context. The approaches taken to deal
 with these requirements have no basis in  the theory of non-market valuation, nor
 precedent in practice.

       Finally, because of our criticisms of the definition of the commodity being
 valued and our concerns about whether respondents fully understood what was  being
 valued, we are unable to make a judgement as to whether the estimates of total value
 for groundwater cleanup are too high or too low. Thus, the  Committee concludes the
 study does not offer evidence that the survey has in fact developed estimates of non-
 use (or passive use) values.   Moreover, there is no basis for believing that estimates
 of the representative household's non-use values (or the total values) could be
 developed for groundwater cleanup independent of the particular circumstances
 associated with the groundwater that has been contaminated.

       As noted below, this conclusion does not imply that further research with the
 survey would not yield valuable insights into how households evaluate groundwater
 cleanup.
                                      26

-------
4.2  Recommended Research

      The Committee would like to commend EPA for supporting innovative policy
research of this type.  Environmental commodities have distinctive characteristics in
that they are often not the object of explicit or implicit market transactions.  In this
instance, there are no readily available economic data to enable economists to
precisely assess the value of removing groundwater contamination to those who will
not be using the potentially contaminated water.  To obtain such values, original
economic research is needed.  Research frequently will not completely resolve all of
the uncertainties with respect to benefit assessment in these controversial areas.
However, beginning the process of exploring the appropriate value to attach to these
problematic benefit components should be a high priority for future EPA funding..

      As the above comments indicated, the Committee believes that there are a
number of ways in which this specific research effort could be enhanced. For
example, providing respondents with a better understanding of the commodity and
ensuring that the responses isolate the non-use value of environmental damage are
clearly of paramount concern.  Indeed, had such a  program existed, prior to the
McClelland et a/, report, it would have been possible to provide more specific answers
to the methodological questions posed in the Charge to this Committee.

      The research issues associated with developing aggregate estimates of the
total value households place on groundwater cleanup are much too detailed to discuss
completely in this report  Five general areas that are identified from the McClelland et
al. report and seem especially relevant for EPA's activities associated with the
corrective action RIA will be discussed.  It is, of course, also true that further
multivariate analysis with the survey data from the McClelland et a/, survey along with
research investigating the survey mode, role of substitute groundwater resources, and
other features of the commodity described in their survey  instrument would be very
desirable and seems likely to enhance our ability to interpret the findings from the
study.

      The first of these has received very limited research attention.  It concerns the
extent of the market for environmental commodities like groundwater.  For the most
part, economic research has focused on developing estimates of the  typical or
representative household values for specific commodities  and not on  the criteria for
determining how many  households can be expected to hold these values. The
McClelland, Schulze, Lazo report (1993) includes some very preliminary and therefore
limited discussion of these issues.  Addressing this question is fundamental to the task
                                      27

-------
 of measuring aggregate values for the cleanup of groundwater resources and may well
 have a much greater quantitative impact on these estimates than would refinements in
 estimates of the representative household's values for changes in some environmental
 commodities.8

       The second area of research would involve a detailed study of the sensitivity of
 CV outcomes to the survey methods used • whether telephone, in-person, or mail
 surveys. The evidence provided to the Committee by Professor Dillman  (letter, March
 31,1993) when compared with the recommendations of the NOAA Panel suggested
 that there is not a consensus in the profession about the superiority of in-person
 interviews over mail surveys. More research on this topic is dearly warranted. Such
 research would consider not only the implications of the method used to  implement
 the survey but also the elicitation procedure and the role of information in the design
 of the survey questions.  It would require parallel efforts conducting the same valua-
 tion  exercise with at least two, and preferably  all three, of the methods available.  It
 would be important not to  compromise the evaluation by requiring that it provide
 "definitive" answers for an ongoing policy issue.  Rather it would be important to have
 the commodity offered be  a "real one" that is of some significance to the households
 that are interviewed, while the goal of the evaluation be dearly identified  as generic
 research, not estimates to be used in a specific subsequent RIA.

       A third area of research would address issues assodated with defining changes
 in environmental commodities.  This research  is central to understanding whether the
 contingent valuation responses are internally consistent.  The recommended research
 would involve developing methods and protocols for practice that could be used to
 understand  how environmental commodities are best measured and how those
 measures relate to the descriptions offered in  the framing of contingent valuation
 questions.  Consideration of the differences that might arise in circumstances when
 total values are dominated by use-related components in comparison with those when
 non-use related services dominate should be an important component of this design.
 Equally'important, to the extent that consideration of risk is an important element in
 the ongoing activity of EPA policy evaluations, some component of perceived risk
 should be incorporated as an additional factor in understanding  commodity definition
 for CV work.
    *Given the extremely Imtted nature of the avatabte research on the value of groundwater cleanup, the Committee feeh that
in thte ca»e both type* of research are excepttonaty important

                                      28

-------
      The fourth area of research arises from an implicit strategy for benefit transfer
that influenced the design of the CV study.  Based on the introduction to the study (as
well as earlier interim study reports and EPA staff briefings), it is dear that the study
was intended to be used as a basis for estimating national non-use values that would
be combined with separately computed estimates of components of the use values
resulting from groundwater cleanup. This strategy implicitly accepts the validity of
separating the use from non-use value. Before this approach can be adopted, there is
a dear need for research on whether respondents can understand this task in a
meaningful way.

   •   'The final high  priority area for research concerns the desirability of evaluating
the potential of these methods developed  in marketing research for calibration At CV
estimates and the development of methods for gauging the potential implications of
using a hypothetical setting to elidt valuations in relationship to real choices.  Is
calibration a meaningful concept in the relationship to  contingent valuation responses?
What would be the mechanism for developing calibration  adjustments?  These are
issues that are important to the  ongoing development  of the contingent valuation
method.  Research evaluating whether or  not contingent valuation results can be
transferred from one setting to others is also critical. The Committee believes this
should be a central component of the future research  that EPA undertakes in support
of its policy evaluation.

      This research  should not be  regarded as peripheral to policy evaluations, but
instead central to the ability of the EEAC to respond to the spedfic types of questions
posed by EPA staff about this study and the assodated RIA analysis. The Committee
assumes that there will be a continuing need to evaluate  analyses intended to
appraise household's willingness to pay for improvements in environmental commodi-
ties. There is simply not sufficient information on the research shelf to address in a
meaningful way many of the questions that have been posed to the EEAC about
contingent valuation and its use. EPA should begin to sponsor research required to
evaluate" "the methods used in its policy evaluations and to develop a greater set of
experience in valuing changes to important economic  resources.
                                      29

-------
                                Distribution List *

Administrator
Deputy Administrator
Assistant Administrators
Deputy Assistant Administrator for Research and Development
Deputy Assistant Administrator for Water
EPA Regional Administrators
EPA Laboratory Directors
EPA Headquarters Library
EPA Regional Libraries
EPA Laboratory Libraries

-------
                             5  REFERENCES

Boyle, K. J.  A Review of Contingent Valuation Studies of the Benefits of Groundwater
      Protection (report to U.S. Environmental Protection Agency), Research Triangle
      Institute, April 1993).

Dillman. D.A. Letter to Alan Cariin, EPA, March 31,1993

Kahneman, D. and Knetsch, J. K. Valuing Public Goods:  The Purchase of Moral
      Satisfaction," Journal of Environmental Economics and Management, Vol 22,
      January 1992, 57-70.

Kneese, A. and Smith, V.K. Memorandum to R.C. Mitchell, March 23,1993.  _ —

McClelland, G.H., Schulze, W.D. Lazo, J.K., Waldman, D.M., Doyle, J.K., Elliott, S.R.
      and Irwin, J.R. Methods for Measuring Non-Use Values: A Contingent Valua-
      tion Study of Groundwater Cleanup (Draft report to the U.S. EPA), University of
      Colorado, Center for Economic Analysis, October,  1992.

McClelland, G.H., Schulze, W.D. and Lazo, J.K. Additional Explication of Methods for
      Measuring Non-Use Values:  A Contingent Valuation Study of Groundwater
      Cleanup, Memorandum to the SAB/Environmental  Economics Advisory Commit-
      tee. Center for Economic Analysis, University of Colorado, June 29,1993.

Mitchell, R.C. and Carson, R.T. Existence Values for Groundwater Protection.  Draft
      Final Report to U.S. Environmental Protection Agency (Washington, D.C.:
      Resources for the Future, May  1989).

National Oceanographic and Atmospheric Administration (NOAA), Report of the NOAA
      Panel on Contingent Valuation, Federal Register, January 15,1993, Vol. 58,
      #10, pp. 4601-4614.-
                                    R-1

-------
EPA-SAB-EPEC-COM-94-001 'Commentary on the
Ecological Risk Assessment for the Proposed RIA for
the RCRA Corrective Action Rule" by the Ecological
Processes and Effects Committee

-------
                UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
                             WASHINGTON, D.C. 20460
                                                                 OFFICE OF THE ADMINISTRATOR
                                                                  SCIENCE ADVISORY BOARD
EPA-SAB-EPEC-COM-94-001

November 19, 1993

Honorable Carol M. Browner
Administrator
U.S. Environmental Protection Agency
401 M Street, S.W.
Washington, DC 20460

       Subject:      Commentary on the Ecological Risk Assessment for the
                   Proposed RIA for the RCRA Corrective Action Rule

Dear Ms. Browner:

       In response  to a request from the Office of Solid Waste and Emergency
Response (OSWER), the Science Advisory Board (SAB) has reviewed several aspects
of the draft Regulatory Impact Analysis (RIA) prepared in support of the RCRA
Corrective Action Rule. At the October 1992 meeting of the SAB Executive
Committee, the Board was asked to review several components of the draft RIA. The
Executive Committee, recognizing the importance, complexity and creativity of
OSWER's work and its multi-disciplinary nature, established an ad hoc Steering
Committee to assure that certain significant aspects of the RIA-both methodology and
application-received appropriate attention from the relevant SAB  Committees.

       At a public meeting on January 29, 1993, the Steering Committee concluded,
based on presentations by and discussions with OSWER personnel, that four SAB
committees, with appropriate inter-committee liaison participation, should review
major segments of the RCRA Corrective Action RIA as follows:  the Environmental
Economics Advisory Committee (EEAQ would review the contingent valuation
methodology and its application in the RIA; the Environmental Engineering
Committee (EEC) would review the MMSOILS fate and transport model;  the
Ecological Processes and Effects Committee (EPEQ would review the ecological
risk analysis; and the Environmental Health Committee (EHC) would review the
human  health risk assessment. In addition, the Steering Committee agreed to
                                                                       Mffed on p«ttr«MI evUM
                                                                       * toot 7S% ncyctod »•

-------
 prepare an overview report, if one is deemed necessary, to accompany the
 individual committee reports.

       This report is the result of EPEC's review of the ecological risk analysis in
 the draft RIA. We hope our comments on the draft RIA  will assist the Agency in
 further developing and integrating contemporary concepts of ecological risk
 assessment into the proposed rule on corrective action for releases of hazardous
 substances from solid waste management units.

 1. Statement of Charge and Review Process

       In the charge to the SAB, dated March 26, 1993, OSWER requested that
 the Board assess "the implications of the fate and transport modelling .assumptions
 on the ecological and human health risk assessment."  To accomplish this, EPEC
 designated two consultants to participate in  the EEC's  review of the MMSOILS
 model, and comments on this topic are largely included in the report of that
 committee.  However, we felt that it was important to evaluate the adequacy of
 the ecological risk chapter itself with the hope of improving this and future
 Agency risk assessments and cost/benefit analyses.  In consultation with  OSWER,
 we developed the following additional questions:

       a)    Given the constraints on available data and modeling assumptions, is
            the ecological risk assessment contained in the RCRA RIA consistent
            with the Ecorisk Framework developed by the EPA Risk Assessment
            Forum?

       b)    Are the ecorisk methodologies used in the  RIA appropriate for
            assessment of risk at this  broad scale  (i.e., national vs. site-specific
            assessment)? What additional analyses could be  added to strengthen
            the assessment?

       Our comments on the draft RIA are the result of a conference call on June
 7, 1993, by the Ecorisk Subcommittee of EPEC, and subsequent discussion of the
 document at  a public meeting of EPEC on June 21, 1993.

2. Summary of Findings

      We are pleased that the Agency is beginning to incorporate  ecological risk
assessment into the day-to-day policy and  regulatory decision  making process.  We
fully recognize the immensity of the task faced by the Agency in developing an
evaluation program sufficiently general to  have application to  the estimated 5,800
facilities and 100,000 solid waste management units (SWMUs) across the country

-------
 which will be subject to the RCRA Corrective Action Rule.  Any such exercise
 necessarily involves numerous trade-offs of site-specificity and accuracy vs. general
 applicability to many diverse facilities and locations.  Nevertheless, we feel that
 the present document is incomplete in its consideration of ecological risks and
 benefits of site remediation, and is not fully consistent with the Agency's Ecorisk
 Framework document.  Substantial additional effort will be  required to complete a
 defensible ecological risk analysis on a national scale, even one which covers only
 baseline conditions.

       There are at least three approaches open to the Agency:  a) the final RIA
 could present the full range of environmental endpoints that should be considered,
 then state clearly that only a subset have been evaluated in the document; b) the
 current ecological risk analysis could be refined and used to answer a-less"
 sweeping set of questions (e.g., how widespread is the risk of exceeding water
 quality standards and other ecological benchmarks in nearby surface waters); or, c)
 the Agency could choose to allocate additional resources to complete the full
 ecological risk assessment on a national scale. If resources  are not available for a
 full risk assessment, the Agency should, at  a minimum, adopt the first alternative.

 3.  Prioritization of Ecological Risks

       The assessment of benefits in the document is incomplete in that no
 attempt is made to assess the ecological benefits (or risks) which would result
 from site remediation.  Although the table of contents describes Chapter 8 of the
 draft RIA as addressing "ecological benefits," the chapter itself is titled "ecological
 threats."  The overall impression left by the document, and  exemplified by Exhibit
 13-2, is that ecological considerations, even the extent of ecological risk from the
 "no action" scenario, are not being factored  into the proposed cost/benefit analysis.
 The authors appear to have relegated both  human and ecological risks to
 secondary issues in deference to  non-use benefits calculated  from the developing
 contingent valuation method.

  —  As noted previously, the Environmental Economics Advisory Committee of
 the SAB is conducting a detailed assessment of the  contingent valuation (CV)
 methodology.  However, we feel strongly that the CV approach cannot yield a
 realistic valuation of a resource unless the respondents have a firm knowledge of
 and appreciation for the human health and ecological risks involved.

      A much better definition of the problem is needed at the very outset of the
 document in order to avoid misleading or false expectations  as to what can really
be done to quantify ecological risks in the absence of corrective action, as well as
benefits (or risks) of site remediation. A clear discussion of the realistic state-of-

-------
 the-science limitations, broad-based assumptions and generalized nature of results
 is needed to properly qualify and caveat any conclusions.

 4.  Risk Assessment Framework

       The Framework for Ecological Risk Assessment (Ecorisk Framework,
 EPA/630/R-92/001) developed by the Agency's Risk Assessment Forum provides a
 conceptual outline for the conduct of ecological risk assessments. The authors of
 the draft RIA do refer to the framework in the case study discussion; however,
 Chapter 8  of the draft RIA could be substantially improved by reorganizing it to
 more closely follow the ecological risk paradigm.  For example, the proximity
 analysis is essentially equivalent to the conceptual step referred to in the Ecorisk
 Framework as problem formulation.  Discussing the elements of the proximity
 analysis in terms of the problem formulation phase will ensure that the first step
 of the ecorisk assessment is properly structured, that key questions will be
 addressed and that meaningful results will be produced. Similarly restructuring
 the concentration-based screening analysis and case studies to follow the
 framework document steps  of stressor-response assessment and risk
 characterization phases will help identify weaknesses in the current document and
 provide guidance to ensure  that ecologically sound results are provided.

      The  problem formulation step appeared to be driven more by conveniently
 accessible data than by development of a conceptual model, probably due to time
 and resource constraints.  As a result, the major exposure pathway which was
 considered (chronically contaminated  surface water) is  not necessarily the most
 likely to cause adverse effects from RCRA sites, as was shown in the case studies
 in Appendix F of the RIA.  The results of the case studies  should now be used to
 reformulate the  conceptual model. The proximity analysis  should also include a
 component which identifies particularly valuable habitats, such as those containing
rare and endangered species. One source of information on the  location of rare
and endangered species is The  Nature Conservancy's Heritage System  database.

   — Clearly, the Ecorisk Framework approach has not been used to its full
benefit.  It is the process of this approach that is useful. By taking a systems
perspective on risks, the choice of endpoints, models, and stressors all make sense
(e.g., see pp.  13-14 of the Ecorisk Framework  document). This approach is
missing in  the draft RIA,  Rigorous application of the risk  assessment  approach
would have clarified numerous  questions left unresolved in the current draft,
including:

      a)    Why was  1 mile selected as the distance to be used in the proximity
            analysis?  The distance should  be selected based on ecological or

-------
             physical considerations.  Also, relative rather than absolute distances
             should be considered.  For example, sites down-slope or along steeper
             topographic gradients may be more susceptible than sites up-slope or
             on less steep areas.  The choice of area should also depend on the
             stress being considered (e.g., some chemicals are more mobile than
             others).

       b)    What was the ecological basis for the choice of the time scales (e.g.,
             why were exposures evaluated over a 128-year period)?

       c)    Why were particular receptors selected (e.g., shrew, hawk, owl)?  An
             ecosystem endpoint (e.g., increasing biodiversity, improving habitat
             quality in terms of food availability and cover) would have-been
             useful. We recommend that the authors of the RIA  consult with the
             Agency's Risk Assessment Forum for assistance in identifying
             appropriate ecological endpoints.

       Another important component of the Ecorisk Framework approach is the
analysis of uncertainty in the overall risk assessment.  The draft  RIA lacks
adequate treatment of uncertainties in the risk estimates. For example, it would
be useful to know the variability/uncertainty surrounding outputs from the
MMSOILS model, expected uncertainties of risk, and how valuation methods deal
with uncertainties.

5. Stratification of the Sample Frame

       Appendix A to the draft RIA provides details of how the universe of sites
was  stratified and how the subsample of facility sites was selected.  Facilities
potentially subject to the Corrective Action Rule were stratified according to the
magnitude of potential costs of corrective action (i.e., very large, large,  and other)
and  the  availability of data for  an assessment.  This stratification does not
consider the  distribution of ecologically-relevant site characteristics, and there are
no data'to indicate that the subsample selected is indeed representative of the
range  of ecological risks at all sites.  We were unable to assess  what other
extremes may be represented by those sites excluded.  For example, does the
subsample accurately represent  the myriad of soil types, groundwater and surface
water  flow regimes that could be encountered?  Thus, it is unclear how the Agency
can relate results  obtained using the subsample of sites to the entire population of
sites.

-------
 6.  Emphasis on Modeling - MMSOILS

      The results of the ecological risk assessment are highly dependent on the
 results of the MMSOILS model. Since MMSOILS has been the subject of a
 separate review by the Environmental Engineering Committee of the SAB, we
 include only overview comments here.

      We question the use of this simplistic model for the RCRA evaluation.  The
 model should be subject to formal, comprehensive sensitivity and uncertainty
 analyses.  Results of these analyses can be used to: a) determine the range of
 model outputs in relation to uncertainty on input data; b) determine what the
 critical data are for improving model  predictions;  and c) simplify the model
 structure without sacrificing accuracy or precision of model results.  At the very
 least, sensitivity analysis should be done to compare outputs from MMSOILS  with
 other chemical  fate models such as EXAMS.  It may be that using a variety of
 different existing fate and transport models, rather than any single model, is the
 most appropriate way to evaluate risks.

      The MMSOILS model is  emphasized as a screening tool.  However, it is
 clear that the model is used beyond screening in estimating quantitatively the fate
 and transport of contaminants.  Care must be taken in implementing and
 evaluating the model for these two different purposes.

      One major problem that must be confronted in  the development of
 multimedia models, such as MMSOILS, is the forcing  of differently scaled
 environmental transport processes into a single model construct. Forcing
 disparate temporal and spatial scales  into  a single model can produce inaccuracies
 in model results.  These scale considerations should be used to examine the basic
 model constructs of MMSOILS.

 7. Adverse Impacts  of Site Remediation

   — As mentioned  earlier, the draft RIA does not address the potentially adverse
 ecological impacts  of corrective  actions.  These actions could include mobilization
and release of large quantities of contaminants (e.g., during dredging of
contaminated sediment or due to accidental ruptures of tanks, dams, etc.);
 destruction of terrestrial habitat due to road-building, soil removal, etc.; and
greatly increased loadings of silt in streams due to erosion of exposed soil
Methodologies for  addressing these effects should  be developed and the impacts
should be considered in the overall analysis.

-------
8.  Use of Case Studies

      In the draft RIA, the three ecorisk case studies are used simply to document
the fact that ecologically significant contamination can be observed at some sites,
and that pathways for potential exposure do exist.  A much more valuable use of
the case studies would be to evaluate the conceptual model used for the ecological
risk assessment and the fate and transport predictions of the MMSOILS model.
Actual contaminant concentrations and hazard indices could be compared to
predicted values; land-use designations obtained from on-site surveys could be used
to verify land use designations obtained from maps, etc.

9. Assumptions and Incomplete Information

      The information provided in Appendix F of the RIA on the concentration-
based screening methodology is quite sketchy, and lacks sufficient discussion of
data sources and assumptions.  However, from the information given, it seems
possible that some inappropriate assumptions have been made.  Areas of
incomplete information include:

      a)    The time frame used in the model for predicted concentrations of
            contaminants in surface waters was an annual tune step, while the
            time frame used for deriving chronic ambient water quality criteria is
            4 days.  Mixing these different time scales is likely a problem since
            the annual average value could mask short-term extreme events.  The
            document should clearly state that this is a problem.

      b)    Many of the extrapolation factors described in Exhibit F-l are
            consistent with general usage by regulatory agencies, although better
            methods are now available in the scientific literature.  They are,
            however, derived principally from studies involving fish.   It appears
            from the documentation that data from invertebrate toxicity tests may
            also have been used in the assessment.  The values in Exhibit F-l
            should not be applied to toxicity test data for organisms other than
            fish.

      c)     The Hazard Index results (Exhibit F-3) are poorly  documented.
            Contaminants responsible for the values are  identified, but the
            pathways and receptors  are not. In addition, some of the index
            values are extremely high (1(T or more).  If environmental
            contaminants were really present at the levels indicated and the
            exposed species were as  sensitive as assumed in the analysis, overt
            ecological effects should  already have occurred and should be easily

-------
             detectable. The concern here is that most regulatory applications of
             the hazard index approach are deliberately conservative and designed
             to  overstate the magnitudes of ecological risks.  In this case,
             conservatism  can result in poor decision making because contaminant-
             related risks may be overstated relative to corrective action-related
             risks, which are not even being considered.

       d)     The document describes methods for estimating exposures and effects
             on benthic invertebrates and terrestrial wildlife, but is vague about
             data sources and assumptions.

       e)     In  the equation used to calculate aqueous benchmarks for piscivorous
             •wildlife (p. F-7), what is the source of the values for "species-
             sensitivity factor "  (SSF)? This may be the most important value in
             the equation.

       In addition to the many limitations of the methodology acknowledged in the
draft RIA, there are a number of specific application and interpretive errors in
Appendix F, including:

       a) Proximity Analysis

             (1)   p. F-l, paragraph 2: Land use categories were ranked in terms
                  of habitat value (i.e., surface water and terrestrial ranked
                  higher than residential, agricultural and industrial land uses).
                  However, assessing land uses in terms of vulnerability
                  ("ecosystems at risk") would result in the  opposite ranking.

             (2)   p. F-l, paragraph 3: "Smaller areas" of surface water are not
                  necessarily at lower risk than "larger areas":  for example,
                  greater volume and surface area tend to reduce risk in larger
                  bodies of water due to dilution and air stripping.

      b) Ecological Benchmark levels

             (1)   p. F-2, paragraph 4: There is no scientific rationale provided
                  for the  extrapolation factors used to account for variation in
                  species  sensitivity, extrapolation from acute to chronic value,
                  and high bioaccumulation potential.

             (2)   p. F-4, paragraph 2: One  cannot estimate BCF values when
                  log KQW is greater than 4 or 5, since the correlation is not
                                       8

-------
                   linear in that range:  many of the chemicals of concern have
                   log K^ values greater than 4.

             (3)    p. F-5, paragraph 2: "No observed effect levels" (NOELs),
                   which are protective of individual organisms, are not the
                   appropriate population threshold for wildlife.

             (4)    p. F-8, paragraph 1: The document states that, "BCF values in
                   the Superfund Chemical Data Matrix are the highest measured
                   for any aquatic species."  What is the impact of all these worst-
                   case estimates on the final  values for the screening-level surface
                   water criteria?
       c) Case Studies

             (1)    p. F-19, paragraph 2:  If, as stated, "the data do not allow for
                   separation of the effects of SWMU releases from permitted
                   releases under the facility's NPDES outfalls," the methodology
                   will not he useful for diagnostic purposes. Is this a generic
                   situation?

             (2)    p. F-21, paragraph 2:  The finding that parasites and fin rot
                   were most prevalent among fish collected closest to the facility
                   may not be significant, since most fish are not territorial in
                   lakes and move around a lot.

10.  Conclusion

      In conclusion, we support the inclusion of an ecological risk assessment in
the RCRA Corrective Action RIA. However, we are concerned that the current
draft does not incorporate the approach contained in the Ecorisk Framework, and
the assessment of ecological risks is incomplete. We  recognize that resource
limitations may preclude a complete ecological risk assessment in this document,
yet the approach taken may serve as a model for other risk assessments and RIAs.
Consequently, we recommend that the RIA be modified to:  a) more explicitly
follow the Ecorisk Framework; b) discuss which ecorisk factors were considered in
the RIA and why, c) discuss which ecorisk factors were not considered and why;
and, d)  discuss uncertainties associated with insufficient knowledge,  inadequate
data, natural variability, etc.

-------
      We sincerely appreciate the opportunity to review the progress of this
important activity. We hope these comments will be helpful to the Agency in
revising the RIA, and we look forward to your response on the issues we have
raised.

                                    Sincerely,
                              Dr. Raymond C. Loehr, Chair
                              Science Advisory Board
Dr. Kenneth L. Dickson, Chair
Ecological Processes and
  Effects Committee
                                          Dr. Alan W. Maki, Chair
                                          Ecorisk Subcommittee
                                     10

-------
               US. ENVIRONMENTAL PROTECTION AGENCY
                        SCIENCE ADVISORY BOARD
           ECOLOGICAL PROCESSES AND EFFECTS COMMITTEE

                                 ROSTER

                             June 21-23, 1993

 CHAIR

 Dr. Kenneth L. Dickson, Institute of Applied Sciences,    University of North
 Texas, Denton, Texas

 MEMBERS/CONSULTANTS

 Dr. Steven M. Bartell, SENES Oak Ridge, Inc., Center for Risk Analysis, Oak
 Ridge, Tennessee

 Dr. Edwin L. Cooper, Department of Anatomy and Cell Biology, School of
 Medicine,  UCLA, Los Angeles, California

 Dr. William E. Cooper, Zoology Department, Michigan State University, East
 Lansing, Michigan

 Dr. Virginia Dale, Environmental Sciences Division, Oak Ridge National
 Laboratory, Oak Ridge, Tennessee

 Dr. Mark A. Harwell, Rosenstiel School of Marine and Atmospheric Science,
 University of Miami, Miami, Florida

 Dr. Robert J. Huggett, Virginia Institute of Marine Science, College of William and
 Mary, Gloucester Point, Virginia

 Dr.-Alan W. Maki, Exxon Company, USA, Houston, Texas

 Dr. Frederic K Pfaender, Institute for Environmental Studies, University of North
 Carolina, Chapel Hill, North Carolina

 Dr. Anne McElroy, SUNY at Stoney Brook, Stoney Brook, New York

Dr. William H. Smith, Professor of Forest Biology, School of Forestry and
Environmental Studies, Yale University, New Haven, Connecticut

Dr. Terry F. Young, Environmental Defense Fund, Oakland, California

-------
SCIENCE ADVISORY BOARD STAFF

Ms. Stephanie Sanzone, Designated Federal Officer, Science Advisory Board
(1400F), U.S. EPA, 401 M Street, S.W., Washington, DC 20460

Mrs. Marcia K Jolly, Staff Secretary, Science Advisory Board (1400F), U.S. EPA,
401 M Street, S.W., Washington, DC  20460

-------
                                   NOTICE

      This report has been written as part of the activities of the Science
Advisory Board, a public advisory group providing extramural scientific
information and advice to the Administrator and other officials of the
Environmental Protection Agency.  The Board is structured to provide balanced,
expert assessment of scientific matters related to problems facing the Agency.
This report has not been reviewed for approval by the Agency and, hence, the
contents of this report do not necessarily represent the views and policies of the
Environmental Protection Agency,  nor of other  agencies in the Executive Branch
of the Federal government, nor does mention of trade names or commercial
products constitute a recommendation for use.

-------
                           DISTRIBUTION LIST

Administrator
Deputy Administrator
Assistant Administrators
Deputy Assistant Administrator for Research and Development
Deputy Assistant Administrator for Solid Waste and Emergency Response
EPA Regional Administrators
EPA Laboratory Directors
EPA Regional Libraries
EPA Laboratory Libraries

-------


-------
                UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
                            WASHINGTON, D.C. 20460
                             November 19, 1993               C*FK*C*THEADMINISTRATOR
                                                              SCIENCE ADVISORY BOARD

 EPA-SAB-EHC-LTR-94-003

 Honorable Carol M. Browner
 Administrator
 U.S. Environmental Protection Agency
 401 M Street, S.W.
 Washington, DC 20460

      Subject:     Review of the Health Benefits for the proposed RIA for the RCRA
                  Corrective Action Plan Rule by the Environmental Health Commit-
                  tee

 Dear Ms. Browner:

      At the October,  1992 meeting of the SAB's Executive Committee (EC) the
 Board was asked by the Office of Solid Waste and Emergency Response (OSWER) to
 review the methodology for the draft Regulatory Impact Analysis (RIA).  This
 cost/benefit analysis is required prior to promulgation of the Agency's final Resource
 Conservation and Recovery Act Corrective Action Rule. The EC, recognizing the
 importance, complexity, and novelty of OSWER's work and its multi-disciplinary
 character, established an ad hoc RCRA-RIA Steering Committee (RRSC) to assure
 that certain aspects of the RIA - in both methodology and application -received
 appropriate attention from the relevant SAB committees.

      At--a-public meeting on January 29,1993, the RRSC concluded, on the basis of
 presentations by and discussions with OSWER personnel, that four SAB individual
 committees should review the major segments of the RCRA-RIA.  Specifically, the
 RRSC agreed to review: a) the contingent valuation (C\fl methodology used in the
 RCRA RIA analysis (CV-1, by the Environmental Economics Advisory Committee
 (EEAC)); b) the application of CV in the RCRA-RIA (CV-2, by the EEAC); c) the princi-
 pal fate and transport model (MMSOILS). used in the RCRA-RIA (by the Environmen-
tal Engineering Committee (EEC)); d) the ecological risk assessment portion of the
 RCRA-RIA (by  the Ecological Processes and Effects Committee (EPEC)); and 0 the
human health risk assessment portion of the RCRA-RIA (by the Environmental Health
Committee (EHC)).

-------
       This letter was prepared by the SAB's Environmental Health Committee
 following the circulation (by mail) of initial comments prepared by a Committee
-Member, and a public teleconference held on September 24, 1993.  The report
 focuses on the risk assessment methodology used to generate the estimated impacts
 on human health resulting from proposed corrective action at RCRA facilities.  The
 March 23, 1993 memorandum from Richard Guimond to Dr. Donald Barnes described
 the Charge for the SAB's review expresses an interest (page 4) in the implications of
 the fate and transport model assumptions on the ecological and human health risk
 assessments. Before such an interest can be addressed, the risk assessments must
 be reviewed to determine whether they are sound, and to suggest improvements
 where they are not; otherwise comments on the implications of fate and transport
 assumptions may not be meaningful.

       Our report is organized into two sections - some overall strategic comments
 which follow in the body of this  letter, and detailed technical points, keyed to specific .
 sections of the draft RIA, which are incorporated in an Appendix enclosed with this
 letter.

       Based on our understanding of OSWER's goals,  the Committee views the draft
 methodology as a screening analysis, as opposed to a more detailed and definitive
 analysis. We have therefore reviewed the material with the following question in mind
 as our Charge: "Is this the best that can be done to provide a method for conducting a
 screening analysis?"

      The screening analysis methodology produced by the Agency was a very
 ambitious undertaking.  It is also of great potential importance since implementing the
 proposed RCRA Corrective Action regulation could cost many billions of dollars and, in
 the future, these techniques for estimating and comparing costs and benefits may well
 find application to other important cases as stated in the document.

      Irf general, the methods used are well known and correspond to "much-used"
 guidelines, methods and practices (GMP). These GMP have, for the most part, been
 developed for use in setting prudently protective standards of exposure to individual
 substances in specific regulatory situations. In the proposed RCRA/RIA methodology
 the GMP are being applied to a large set of complex cases involving multiple expo-
 sures to make calculations of total impact not contemplated when the GMP were first
 derived.  Determining whether the GMP  can be so applied, whether they are ade-
 quately applied, whether their application to determine overall health impacts is
 reasonable, whether the important assumptions and limitations have been clearly

-------
 identified, and what reasonable and feasible improvements can be suggested has
 been an objective of our review.
\
       The methods developed in Chapter 7 and Appendix E are obviously the result
 of much careful and thoughtful work. Care  has been taken to lay out and define the
 risk assessment process used and its components. The basic assumptions used in
 each of the steps are clearly stated, (including the way in which numerical values were
 chosen to bring in as  much realism as possible at each step through the use of site
 specificity where possible). Alternatives considered and different ways used to
 calculate the  end results are set forth so as to give an idea of sensitivities or possible
 alternative results (e.g., the decision to  analyze a "less than 100 percent effective-
 ness" corrective action scenario).  Lastly,  possible biases and uncertainties inherent in
 the whole procedure are identified.

       The net effect of the effort was to produce  a construct formed of many carefully
 selected parts with which a formal estimate  of cancer population risk can be made,
 both before and after  corrective action,  and  with which a formal estimate can be made
 of the numbers of persons exposed to contaminant levels which have some probability
 (extent unknown) of producing adverse, non-carcinogenic effects , also before and
 after corrective action. Thus, at least formally, the effect of the regulation can be
 measured in two different ways, one for cancer and one for non-cancer effects.

       Although those responsible for developing this construct can be congratulated
 on their achievement,  another group of workers might have made different specific
 choices at various junctures (for example, some other length of time than the 9 year
 exposure period, or the 128 year timeframe). However, it is doubtful if, given the state
 of science, a more "rigorous" method with greater certainty of giving "right" answers is
 possible at the present time.

       As a construct,  given both the state of science and the impossibility of validat-
 ing the=£alculations through any realistically  achievable, actual measurements, the
 results reported must be regarded as coming from an enormously complex, logically
 consistent, but mainly  hypothetical calculation - as must the results of most low
exposure-level risk assessments. We do not use  the term "hypothetical" in a negative
sense; rather, as carefully  devised as it  is, the construct cannot give, at best,  more
than a rough estimate of the actual situation. Using these  results in any kind of
cost-benefit balancing  must be done with this ambiguity firmly in mind.  We suggest
that in the final RIA, this point be made  abundantly dear to avoid even the possibility
that the results wfll be treated as definitive by anyone.

-------
      There are some areas where we suggest possible improvements, i It would be
 very useful to include a way to estimate, even crudely, the fraction of the population
 presumably exposed to significant levels of contaminants (HI > 1) who actually
 manifest adverse, non-cancer effects.  Without some attempt at providing such
 estimates for the most important cases, the cost/benefit calculation remains seriously
 incomplete.  By including such estimates, the monetization of both cancer and non-
 cancer effects avoidance benefits can be done in a formal sense, and that portion of
 the cost/benefit calculation would at least be present in the overall screening analysis.

       The Committee suggests that there is a methodology available to estimate the
 size of the population affected at a specified level of effects. We  suggest (and this is
 probably the most significant revision we propose) a change in the basic approach
 used for relating toxicant exposure and effects. The proposed methodology can be
 expanded to include use  of the benchmark dose concept for determining adverse
 effects, including reproductive and developmental toxicity, as described by  Crump
 (1984) and Kimmel  and Gaylor (1988), and offered as a suggestion in the EHC's
 report (EPA, 1990) reviewing proposed revisions to the EPA Guidelines for Develop-
 mental Toxicity. From this benchmark dose, a straight line can then be drawn to the
 dose level that represents a 100-fold margin-of-safety, plus any relevant uncertainty
 factors, i.e.,  the reference dose for developmental toxicity. The reference dose is
 considered to be an exposure level unlikely to cause human developmental effects.
 At the beginning of the low-dose extrapolation below the benchmark dose,  the actual
 probabilities of adverse effects may be close approximations of the linear extrapola-
 tion. As the extrapolation progresses toward the reference dose, the calculated
 probability of adverse effects may exceed reality to some unknown extent due to the
 presumed threshold phenomena operative for most developmental toxicants.

      Use of this model would allow both pre- and post-remediation quantification of
 reduction of risk in a manner markedly consistent with that for low-dose extrapolations
for carcinqgenests.  This calculation could be used to monetize remediation efficacy.
The example given here has been for developmental toxicity, but it is suggested that
the report also attempt similar type calculations for reproductive toxicity and other
non-carcinogenic toxicologic endpoint assays. (It should be noted that this method
does not lend itself to quantification of effect severity, e.g., den lip verses auricular
tags, but it does permit quantification of classes of adverse effects.)

      Addressing other areas, we note that the term "population risk" (and related
terms in connection with both cancer and non-cancer adverse effects) is employed
correctly vis-a-vis cancer, but not with non-cancer effects. As discussed in more detail

-------
 in the Appendix, the population risk for cancer is correctly given as an estimate of the
 number of people affected by cancer within the exposed populations.  In contrast, for
. non-cancer effects, estimates of the  number of people who are exposed at levels
 exceeding the Hazard Index (HI > 1) do not necessarily coincide with the numbers
 presumed affected; indeed, the fraction of those actually expected to be affected by
 non-cancer effects among the numbers cited is likely to be very small, possibly zero.
 Also, within the cancer/non-cancer dichotomy,  consideration might be given in some
 way to those autoimmune diseases such as lupus, and certain  genetic diseases
 whose causes may be related to environmental factors.  The Agency should be more
 explicit in distinguishing cancer and other disease conditions with respect to risk and
 "population at risk."  The SAB's Environmental Engineering Committee, in its review of
 the RCRA MMSoils model (EPA-SAB-EEC-94-002) had similar  concerns about the
 EPA practice of using different approaches to cancer and non-cancer risk assessment
 (as well as on some misidentification of critical endpoints for some common landfill
 constituents).

       The benefits of abating disease are not  monetized in the RIA document. At
 some point, the decrease in cancer cases and the decrease in  numbers exposed to
 possible risks of non-cancer adverse effects may need to be balanced, along with
 other benefits (either monetized or not) against the dollar costs of corrective action.
 This is a difficult if not impossible aim to achieve in any objective way.  Alternatively,
 using existing methodologies, the results of the calculations could be used to  estimate
 direct monetary benefits of cancer avoidance (medical costs avoided, lost productivity,
 etc...) as well as indirect  costs (pain and suffering, damage to family relations,
 damage to quality of life, etc...).  However, a similar calculation cannot be made for
 non-cancer effects based  on the results of the  proposed  screening method. (The
 SAB's Environmental Economics Advisory Committee, as part of its review of  the
 RCRA/RIA, is considering the monetization of health benefits.)  Another consideration
 which might be taken into account in estimating impact or monetizing effects is age of
 onset of a fatal  and/or disabling disease;  i.e., cancer in an 80-year old will have
 differerlf societal and  personal impacts than cancer in a 24-year old person.

       It would be useful to estimate  for cancer, the number of individuals "at risk"
 (already done for non-cancer effects) so as to have comparable numbers of people at
 risk for cancer and non-cancer effects.  An estimate of the population exposed at
 levels of exposure of concern for cancer (i.e., levels leading to a lifetime individual risk
 of 10* or greater) would yield such estimates recognizing the fact that what is of
 concern is not identically defined in the two cases.

-------
      Lastly, we urge increased emphasis on the collection and management of good
exposure data as a foundation for this, and other efforts by the Agency. The impor-
tance of good exposure data can not be underestimated. From what we now under-
stand, concentrations predicted via the MMSOILS model (as reviewed by the Environ-
mental Engineering Committee) are subject to large uncertainties, affecting exposure
estimates significantly, and thereby affecting the results of the risk assessment.  More-
over, the screening methodology will produce results which are completely useless
and inaccurate if chemicals released from the subject facilities are not included in the
assessment.  We note that several chemicals which have caused problems around
municipal landfills are not included in Table E-1 - particularly methane and hydrogen
sulfide.  These chemicals may well also be generated at solid waste management
units at  RCRA facilities. Although  not generally considered a toxic problem, methane
has accumulated In houses and caused explosions; this is indeed a public health
problem. Hydrogen sulfide is produced when sulfur containing compounds such as
gypsum are buried and become wet.  Hydrogen sulfide releases from several landfills
have produced documented health effects in people living in  nearby communities.
Although the importance of exposure data is discussed in the Appendix, we believe
that it is such a basic and important consideration that it needs to be highlighted in our
comments especially since  some problematic chemicals have not been Included in the
assessment.

      We look forward to receiving your response to our comments.

                                  Sincerely,
                        Dr. Raymond C. Loehr, Chair
                        Science Advisory Board
                                    iy£
                        Or. Arthur C. Upton, Chair
                        Environmental Health Committee
ENCLOSURE

-------
                                  APPENDIX

 SPECIFIC COMMENTS: CHAPTER 7

 -  Page 7-7. last para.: The SAB/EEC, in its review of the MMSOILS transport model
 as part of the RCRA/RIA review, apparently has reservations about the use of such a
 general model to predict concentrations for a wide variety of different, specific cases,
 along with other reservations about it. It would be desirable to contact the SAB/EEC
 (and the liaison members) to determine what bias or uncertainty this might introduce
 into the risk assessment, what might be done instead, and under what criteria
 MMSOILS might not be adequate, with suitable precautionary considerations, for a
 screening analysis.

 -  Page 7-15. references 17 and 18: It would be useful to review the SAB's reviews
 of these documents, which took place subsequent to the dates on them, to ensure that
 the comments therein are  adequately taken into account in this  proposed methodolo-
 gy; perhaps the SAB's review should be added as a reference.

 -  Page 7-22. top Of the page: The selection  of the nine-year period  needs more
 justification or explanation.

 -  Page 7-23. last sentence of 1st para.: This is an inadequate  risk descriptor (See
 comments re Page 7-41. 2nd para, and further related comments as to why).

 -  page 7-29.1st paragraph: the word "may" appears twice; delete the second occur-
 rence.

 -  Page 7-41. 2nd para.: The statement,"... and 25,000,000 persons experiencing
 non-cancer health effects over the 128 year modeling period..." is an example of an
 inaccurate interpretation of the  extent of the population whose exposure exceeds an
 HI of 1.0. (The next sentence is another example as  is the wording in Exhibit 7-17).
 The 25,000,000 people correspond to the population exposed at levels yielding an  HI
 of 1.0 or higher, not those  "...experiencing non-cancer health  effects...".  Exposure at
 or above' effect levels does not equal effect; only the probability of effect (Suppose a
 news story were to appear asserting that EPA estimates that 25,000,000 people suffer
from etc...!) The number experiencing non-cancer health effects will actually only be a
fraction of the 25,000,000 — 25, 250 or maybe as many as 2,500.  The number of
 people exposed at an HI of 1.0 or greater is therefore not a measure  (or a descriptor)
 of population risk: the number of such people who do actually experience non-cancer
 health effects is (as in the  case of cancer.population risk is the number of people
adversely affected).  The numbers of people so exposed and  the numbers actually
experiencing non-cancer health effects can, in fact,  easily be in reverse order for
different cases so that the  figure of 25,000,000 is, again, not a measure of risk.  This
wording should be changed (as should similar wording elsewhere in the text) to reflect

                                     A-l

-------
 this fact and to emphasize it so that the user/reader will not be misled.  As mentioned
 above, one of the matters the SAB's Environmental Economics Advisory Committee is
, addressing in their portion of the RCRA/RIA review is the question of monetizing the
' health risks so as to calculate the benefits obtained directly from reducing the risks
 through regulation. Use of the 25,000,000 figure in such monetization under the
 impression that it measures the number of people experiencing  non-cancer health
 effects would lead to grossly high dollar values.

 -  Page 7-42.1st para.: This paragraph is a most important observation.  It could
 imply that most of the cost and most of the benefit of Corrective Action could be
 attributed to this  one site: It would be highly desirable to quote the site population
 figures in this paragraph, compared to the relevant totals, and to underline or highlight
 the paragraph to help ensure that its importance for the rest of the analysis is not
 missed by the reader.

 ~  Page 7-42. 2nd para., last two sentences: The last of the two sentences corrects
 the statement of  the first one;  I suggest eliminating the first one and using the sense
 of the second one, instead, to give a correct impression in the first place.

 ~  Pace 7-42. section entitled " Number of Facllffies..."Exhlbit 7-18 and all other
 sections of the material under review where this comment applies; The cancer and
 non-cancer population "risk" figures should not be combined in any way and indicated
 to be somehow of the same type since they are intrinsically differently defined; they
 should preferably be presented separately.  The cancer figures represent an  estimate
 (however uncertain) of the number of cancer cases whereas the non-cancer figures
 represent the number of individuals who merely might become non-cancer health
 effects cases. Any kind of "sum" or statement of "jointness" is meaningless unless
 very  carefully labelled. Stating the number or percentage of sites where the number
 of cancer cases is expected to be insignificant (less than one in a million, say) and in
 which the HI is less than one (and therefore the number of people in which non-
 cancer health effects might not occur in significant numbers if at all) is a
 not-misleading statistic that might be helpful to the reader/user.  The text and Exhibits
 need to be modified to not mix the two types of estimates in a misleading way.  For
 exampte/jri Exhibit 7-18, the wording No Risk might be changed to W Risk and the
wording'	% Risk might be changed to	% Risk and Possibly At Risk. Other
 examples abound; for example, in Exhibit 7-24 (a very useful Exhibit), even though the
 non-cancer ordinate has a parenthetical statement defining what is really meant, the
title of the chart should be changed to CUMULATIVE NON-CANCER POPULATION
 POSSIBLY AT RISK and the title of the ordinate should  be changed accordingly.

       Throughout the entire text the distinction between the two types of population
 estimates should be carefully maintained: population risk, in the case of cancer, and
 population at riskJn the case of non-cancer effects. There are many such instances
 and I suggest the text be carefully edited to find and change each and every one of


                                      A-2

-------
 them (for example, where the term "risk/effects' is used one might use, instead "risk/at
 risk," or "risk/concern'' or "effects/possible effects," etc.

 —  Sections 7.4.1. 7.4.2 and 7.4.3: These two sections are well done and very wel-
 come.

 SPECIFIC COMMENTS: APPENDIX E

 - page 5. first paragraph: Too flat and sharp a distinction is made between carcino-
 gens and non-carcinogens on the basis of the existence or nonexistence of thresh-
 olds. I  would suggest the following words: "It has been the custom to suppose that for
 non-carcinogenic or systemic effects protective physiological mechanisms exist that
 must be overcome before the adverse effect is manifested. This may not, in fact, be
 universally true and for the non-carcinogenic effects of lead, for example, K appears
 not to be true.  Similarly thresholds are thought to be absent in the case of cancer,
 Le.,  any level of exposure, however small, could result in cancer although there are a
 very few instances now known in which this assumption may not hold.  Nonetheless,
 in this method, the existence of thresholds will be assumed as usual in the case of
 non-carcinogens (except for lead, as discussed below) and the lack of thresholds will
 be assumed for carcinogens, also as usual"

      These words describe just what is  being done, and the real assumptions being
 made, for the user of the method.

 - page 5. second paragraph: The use of "benchmark" to describe RfDs could be
 confused with the term "benchmark dose," which quite different from and RfD.  We
 suggest changing the subject sentence to read "For many chemicals the RfD ap-
 proach  has been used as  a basis for regulatory decisions in relation to potential
 impacts on human health."

 — Page 27. first paragraph: This is an entirely sensible way to handle the aggregation
 of risks  from exposures to multiple carcinogens, it should be mentioned here, howev-
 er, that  synergism  (and/or antagonism) is possible (referring to section 7.4.3 of
 Chapter^ -Where it is already mentioned).

 — Page 27. second paragraph:  The second sentence, beginning "Ratios of contami-
 nant level... " should be  added to as follows:"... non-carcinogenic health affect for
 exposure to a particular contaminant.  The hazard quotients for different contaminants,
even If they have equal RfDs, do not necessarily indicate which substance poses the
greater  risk.' (the material in italics is the  added material) - Also, regarding the HI, the
SAB/EHC commented in considerable depth somewhat over a  year ago on the
limitations of the use of the Hi in its review of the "Risk Assessment Guidelines for
Superfund Sites". We suggest that the Agency needs to refer to the SAB/EHC review,
especially its Appendix, and to include mention of the principal  limitations of the use of


                                     A-3

-------
 the HI, here, in Appendix E, since the HI is incorrectly used in identifying populations
 as being significantly exposed to contaminants having non-cancer adverse health
»effects.

 -  paoe 27.last paragraph fE.3.31: Here, again, the comments made on Chapter 7
 need to be taken into account on the subject of the difference between population risk
 and population at risk. The first sentence of this paragraph, as well as further
 statements within it, need modification.

       The problem is that the estimates of population risks associated with carcino-
 gens, despite their well known weaknesses, at least purport to make some kind of
 estimate, using a model which may or may not apply in a given case, of the number of
 people affected (true population risk) as a result of exposure whereas counting the
 number of people with HI > 1 for non-carcinogens estimates the number of people at
 exposure levels such that they are potentially affected but not necessarily affected
 (this is not population risk). Whereas the estimates for carcinogens at least attempt to
 get at the number of people affected, the method for non-carcinogens does not
 attempt to do so since it includes those exposed and affected, plus those exposed but
 not affected in its count.  Unless some effort is made to reconcile the two methods,
 the result of the non-carcinogen procedure  is inconsistent with that for carcinogens
 and the two results are not comparable.  Moreover, as mentioned above, the
 "population-at-risk" result for non-carcinogens does not measure risk and should not
 be said or inferred to do  so here or elsewhere in the report it estimates only one
 factor in characterizing risk, namely, an estimate of whether an exposed population is
 exposed to possibly meaningful levels of the agents involved, but it takes no account
 of the probability  of such a population actually exhibiting adverse effects.  This point
 needs to be made and maintained dearly in the text to avoid any misunderstanding by
 users if the method.
                                      A-4

-------
                              REFERENCES

•Crump, K.S.  1984.  A new method for determining allowable daily intakes. Fund.
      •Appl. Toxicol. (4):854-871.

 EPA. 1990.  Review of proposed developmental guidelines. EPA-SAB-EHC-90-013.

 Kimmel, C.A. and Gaylor, D. W.  1988.  Issues in qualitative and quantitative risk
      analysis for developmental toxicology. Risk Analysis (8): 15-20.
                                    R-l

-------
                 U.S. ENVIRONMENTAL PROTECTION AGENCY
                         SCIENCE ADVISORY BOARD
                    ENVIRONMENTAL HEALTH COMMITTEE

                    RCRA RIA HEALTH BENEFITS REVIEW

CHAIRMAN
Dr. Arthur Upton, Santa Fe, NM

MEMBERS AND CONSULTANTS
Dr. William B. Bunn, Mobile Administrative Services, Company, Inc., Princeton, NJ

Dr. David Gaylor, Department of Health & Human Services, National Center for lexicological
Research, Jefferson, AR

Dr. Rolf Hartung, School of Public Health, University of Michigan, Ann Arbor, Ml

Dr. Rogene F. Henderson, Inhalation Toxicology Research Institute, Albuquerque, NM

Dr. Marshall Johnson, Jefferson Medical College, Philadelphia, PA

Dr. Nancy K. Kim, New York Department of Health, Albany, NY

Dr. David Howe Wegman, University of Lowell,  Lowell, MA

Dr. Bernard Weiss, University of Rochester School of Medicine, Rochester, NY

DESIGNATED FEDERAL OFFICER
Mr. Samuel Rondberg, Environmental Health Committee, Science Advisory Board (1400F),
U.S. Environmental Protection Agency, Washington, D.C. 20460

STAFF SECRETARY
Ms. Mary L. Winston, Environmental Protection Agency, Science Advisory Board (1400F),
Washington, D.C. 20460

-------
                                   NOTICE

    This report has been written as a part of the activities of the Science Advisory
Board, a public advisory group providing extramural scientific information and advice
to the Administrator and other officials of the Environmental Protection Agency. The
Board is structured to provide balanced, expert assessment of scientific matters
related to problems facing the Agency. This report has not been reviewed for
approval by the Agency and, hence, the contents of this report do not necessarily
represent the views and policies of the Environmental Protection Agency, nor of other
agencies in the Executive Branch of the Federal government, nor does mention of
trade names or commercial  products constitute a recommendation for use.

-------
                                Distribution List

Administrator
Deputy Administrator
Assistant Administrators
Deputy Assistant Administrator for Research and Development
Deputy Assistant Administrator for Water
EPA Regional Administrators
EPA Laboratory Directors
EPA Headquarters Library
EPA Regional Libraries
EPA* Laboratory Libraries

-------
EPA-SAB-EEAC-LTR-94-001 'Review of Economic
Aspects  of the Proposed RIA  for  the  RCRA
Corrective  Action Rule"   by the Environmental
Economics Advisory Committee

-------
                UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
                            WASHINGTON, D.C. 20460
                              November 19, 1993
                                         1                    OFFICE OF THE ADMINISTRATOR
                                                              SCIENCE ADVISORY BOARD
 EPA-SAB-EEAC-LTR-94-001

 Honorable Carol M. Browner
 Administrator
 U.S.JEnvironmental Protection Agency
 401 M Street, S.W.
 Washington, DC 20460

 Subject:     Review of Economic Aspects of the proposed RIA for the RCRA
            Corrective Action Rule by the Environmental Economics Advisory
            Committee (also referred to as CV-2)

 Dear Ms. Browner:

      At the October, 1992 meeting of the SAB's Executive Committee (EC) the
 Board was asked by the Office of Solid Waste and Emergency Response (OSWER) to
 review the methodology for the draft Regulatory Impact Analysis (RIA).  This
 cost/benefit analysis is required prior to promulgation of the Agency's final Resource
 Conservation and Recovery Act Corrective Action Rule. The EC, recognizing the
 importance, complexity, and novelty of OSWER's work and its multi-disciplinary
 character, established an ad hoc  RCRA-RIA Steering Committee (RRSC) to assure
 that certain aspects of the RIA -  in both methodology and application - received
 appropriate attention from the relevant SAB committees.

      At a public meeting on January 29, 1993, the RRSC concluded, on the basis of
 presentations by and discussions with OSWER personnel, that four SAB individual
 committees should review the major segments of the RCRA-RIA. Specifically, the
 RRSC agreed to review, a) the contingent valuation (CV) methodology used in the
 RCRA RIA analysis (CV-1, by the Environmental Economics Advisory Committee
 (EEAC)); b) the application of CV in the RCRA-RIA (CV-2, by the EEAC); c) the princi-
 pal fate and transport model (MMSOILS). used in the RCRA-RIA (by the Environmen-
tal Engineering Committee (EEC)); d) the ecological risk assessment portion of the
RCRA-RIA (by the Ecological Processes and Effects Committee (EPEC)); and f) the
human health risk assessment portion of the RCRA-RIA (by the Environmental Health
Committee (EHC)).

-------
       This letter comprises the report of the EEAC, which completed its review of the
 analytical methodology for the draft RIA at its meeting of September 23, 1993.  The
.Committee evaluated four aspects of the proposed economic methodologies, includ-
 ing:

       a)    the desirability of disaggregate information as part of the development
             and presentation of benefit-cost information in the RIA for a large,
             complex national rule

       b)    the exploration and presentation of "human health benefits," including
             both the cancer and non-cancer health effects associated with the
             proposed rule

       c)    the possibility of using the McClelland et a/. (1992) CV analysis as the
             basis for estimating the non-use values for groundwater cleanup on a
             national scale

       d)    the application of hedonic methods to evaluate the effects of contaminat-
             ed sites on residential property values (Hedonic models recognize that
             many commodities within the same broad categories, like houses and
             automobiles, for example, have different features. The models imply that
             the prices of products within each category should be related to the mix
             of features that each type of commodity has. The models are sometimes
             used to estimate people's willingness to pay for dimensions of environ-
             mental quality that are location specific, such as air or water quality.)

       Overall, EPA staff are to be commended for a number of innovations in the
methodologies outlined for use in the full RIA  and  illustrated with the examples in the
draft report  Especially notable from the perspective of the economic analysis were:

       a)    the recognition, as part of the human health benefit analysis (and associ-
             ated risk computations), that policies are not always effective

       b)    careful descriptions of both baseline  and "with-policy" effects in all
             aspects of the evaluation

       c)     attempting to separate the effects of  the sub-part S rule from those of
             other initiatives

       d)    recognition of the importance of  the extent of the market for benefits
             transfer

-------
       e)    full documentation of the econometric analysis associated with the
             hedonic property value models

       With respect to the development and presentation of national benefit-cost esti-
 mates, the Committee recognizes that the RIA must, to the extent possible, represent
 the aggregate benefits estimated to be associated with the proposed regulatory
 alternative.  Nonetheless, this goal does not preclude EPA staff from presenting
 disaggregate information for the benefits and costs associated with different compo-
 nents of the aggregate.  For example, it should be possible to classify Solid Waste
 Management Units (SWMU) by size, location, and other characteristics and to develop
 the estimates according to these categories. This practice would facilitate evaluation
 of the methods used and improve the transferability of disaggregate results to the
 aggregate level.

       Classification of SWMU would also permit evaluation of the likely distributional
 impacts of policy alternatives across different types of facilities and types of communi-
 ties impacted. The Committee recognizes that the limits implied by confidentiality
 requires such disaggregation be conducted at levels that assure sites cannot be
 identified. However, this requirement does not preclude substantive effort to develop
 informative decompositions for the  policy alternatives.

       The Committee also recommends that in developing the benefit-cost analysis,
 EPA staff should provide a more detailed  description of the role of each type of
 analysis, as well as more complete perspectives on the judgements made in develop-
 ing estimates of benefits and costs. For example, the Executive Summary of the draft
 RIA notes that benefits estimates would likely be larger if the health effects were
 monetized.  In an apparent contradiction to the Summary, the Health  Benefits chapter
 (Chapter 7)  indicates that monetization was not undertaken because it was judged to
 overlap the monetary  measures developed using the avoiding cost model; no discus-
 sion or explanation of these differing views are offered. Similarly, the hedonic esti-
 mates are provided without developing their role in the comprehensive benefit-cost
 analysis~nor their implications for further RIA analyses.

       These descriptions would permit a second type of evaluation associated with
the aggregations that must accompany an RIA evaluation at a national level.  Moneti-
zation of multiple components of benefits can lead to double counting if the concepts
being measured and the rationale underlying each method are not well-understood
and well-documented. Discussion of these concepts and rationale can provide a
means of avoiding  this.  Together with the presentation of the estimates, such a
discussion often allows plausibility checks, particularly if one approach produces

-------
 estimates that can be expected to provide an upper*or lower bound on the estimates
 of an alternative approach.

     •  The second major element of our review addressed the RIA's evaluation of the
 benefits from  avoiding adverse human health effects (the Committee prefers the term
 "effects" to the term "benefits" as noted in the Technical Appendix). The RIA offers an
 innovative analysis of exceptionally difficult issues.  The analysis is to be commended
 for recognizing and developing methods to reflect the fact that regulatory policies are
 not always completely effective.  While there are  also notable features in this compo-
 nent of the RIA, the Committee believes that there are also three important limitations.
 First, the Committee concludes that efforts to monetize the mortality risks reduction
 (e.g., reduction in statistical lives) should be included in the RIA. The Agency's
 Guidelines for Preparing Regulatory Impact Analyses discusses these procedures and
 recommends monetization, but the RIA does not  do so. Monetizing of health effects is
 also an integral part of the benefit-cost analyses being undertaken for the Clean Air
 Act evaluation currently  under way.

       The argument that averting cost and monetized health effects would imply
 double counting is correct, but it ignores the role  these estimates could serve as
 bounds on estimates of benefits.  The inability to  monetize all health effects should not
 preclude this effort. It may be possible to offer better resolution about how benefits
 might vary across different classes of facilities, regions, or other characteristics of the
 SWMU through quantification of the health effects than by using  information on
 averting expenditures (because the exposed  population and  response estimates seem
 more firmly grounded in scientific evidence than the averting expenditure information),

       Aggregation of non-cancer health effects based on exceedances of the chronic
 reference dose provides an index that is no more than informative.  The Committee
 has been advised that there are  a number of judgements used in developing the
 hazard indexes from  hazard quotients (i.e., estimates of individual exposures to the
 chronic reference dose)  for specific substances.  The Committee recommends that
 efforts To" aggregate across substances be avoided  and disaggregated information for
 classes of substances be reported instead.

       Finally,  the cost effectiveness analysis and presentation of overall impacts
 aggregates effects over  a  128 year time horizon.  The Committee recommends that
 alternative strategies be investigated for dealing with the effects of this long time
 horizon, including: discounting the measures  of physical effects before gauging cost
 effectiveness;  calculating cost effectiveness based on a year-by-year cost effective-
 ness and these ratios then discounted to a base period or an annualization of all
factors relevant to a comparison.

-------
       (The Committee also identified some further technical issues on the health
 benefits topic, as well as on the next two aspects of our review; a detailed summary is
. enclosed as a Technical Appendix to this letter.)

       The EEAC devoted considerable effort to the issue of using the McClelland et
 al. CV estimates as a basis for making national estimates of non-use values for
 ground water cleanup. Because the focus of the McClelland et al. effort was to
 develop oer-household estimates of these non-use values, a number of issues would
 need to be addressed to use the results for national estimates relevant to the RIA.
 The Committee's detailed findings on these issues are provided in the Technical
 Appendix.

       Based on our earlier review of the McClelland et al. study itself, and two further
 issues raised by the  methodology proposed for adopting the McClelland et al. results
 for use in the RIA, the Committee believes that the McClelland et al. estimates cannot
 be used for the intended purposes.  The RIA's evaluation primarily deals with benefits
 from cleaning up over a long period of time (and in some cases partially) groundwater
 contaminated by industrial facilities.  The McClelland et al. study focuses on municipal
 sources and describes a situation with instantaneous cleanup.  Both issues are very
 important features of the problem and past literature suggests that they are likely to be
 important to people's willingness to pay for cleanup.  Unfortunately, simple adjust-
 ments do not appear to be possible.

       Last,  a key element in the development of aggregate estimates is the determi-
 nation of the number of households which would be willing to pay (a non-use  value)
 for cleanup of the groundwater. This factor is the key determinant of the wide range
 of estimates of aggregate non-use values.  No specific evidence has been developed
 on how to determine the number of these households.  The original McClelland et al.
 study focused on per-household values.  Subsequent work appears very preliminary
 based on the information made available to the Committee.

       "EPA is to  be commended for recognizing the  extent of the market question.
 Nonetheless, the McClelland et al. methodology does not offer an approach to deal
with the problem.

       Overall, the Committee recommends  against using the specific approach
proposed in the draft RIA for developing  the aggregate estimates of non-use values.
Unfortunately, there is no other information in the literature to be used to meet the
needs of the RIA effort.

-------
      The Committee found the hedonic analysis to be a careful and systematic
 evaluation of the effect of proximity of a contaminated site on the prices of nearby
 residential properties.  The Committee has some technical suggestions described in
 the Appendix, and recommends that if the results of the hedonic analysis are included
 in the RIA, two additions be made to the discussion.  First, the relevant chapter should
 describe dearly the analysis as providing a record of studies undertaken and summa-
 rize in greater detail the reasons for not using the results  in the benefit measurement.

      Lastly, a brief discussion of the  potential (at a conceptual level) for using
 hedonic estimates to bound valuation estimates would be desirable.

      In summary, from the perspective of its economic methodology, the draft RIA
 reflects considerable creativity by EPA staff.  Although the Committee had a number
 of detailed suggestions, these can, with the exception of the estimates of aggregate
 non-use values, be addressed with revisions  and extensions to existing methods.

      We look forward to receiving your response to our  comments.

                                   Sincerely,
                        Dr. Raymond C. Loehr, Chair
                        Science Advisory Board
       «= •-
      Dr/Allen Kneese, Co-Chair           Dr/vTWetry^Smith, Co-Chair
      Environmental Economics            Environmental Economics
      Advisory Committee                 Advisory Committee
ENCLOSURES

                                      6

-------
                             TECHNICAL APPENDIX

      The purpose of this Appendix is to summarize some detailed further sugges-
tions elaborating on the Committee's comments regarding human health benefits
estimates, the use of the McClelland et a/, findings as a basis for national non-use
value estimates, and the hedonic analyses.

                          aj Human Health "Benefits"

      The Agency constructed two alternative baseline scenarios as descriptions of
conditions in the absence of the Proposed Rule:

      1)    human exposures would NOT be capped through existing MCLs and
            taste/odor limits
      2)    exposures would be capped through existing MCLs and taste/odor limits.

The agency then estimated the reductions in the numbers of total cancer cases and
non-cancer health effects associated with the Proposed Rule assuming that the rule
would be less than 100 per cent effective in eliminating risks.  The reductions in health
effects are listed as the human health benefits.

      A positive feature of this analysis is the recognition that policies are not always
100 per cent effective and that realistic estimates of benefits should be based on the
expected level of effectiveness rather than the theoretical ideal. However, the
reported numbers are not particularly informative for policy purposes for several
reasons:

      1)    there is no economic valuation of the reductions in either cancer cases
            or non-cancer health effects. This is puzzling, since the Agency's
            Guidelines for Performing Regulatory Impact Analysis discusses ap-
            proaches for estimating economic values for reductions in both morbidity
            and mortality and other Agency RIAs have utilized these approaches to
            monetize health effects.

     ~2)     the measure of non-cancer health effects involves a difficult-to-interpret
            aggregate index that consists of the sum of the numbers of individuals
            with daily intakes in excess of the chronic reference doses (RfD) for the
            substances analyzed plus numbers of individuals whose exposure to
            lead results in  blood lead levels above the threshold.  Two important
            pieces of information are obscured by this form of aggregations:
            i)     Different chemicals cause different types of health effects; and not
                  all health effects are of the same severity.  Thus mild and severe
                                     A-1

-------
                   adverse effects are lumped together and given the same weight in
                   this analysis.

             ii)     This form of aggregation treats a small exceedance of the  RfD as
                   equivalent to a large exceedance.  Also, using the number of
                   exceedances as an indicator of adverse health effects involves the
                   implicit assumption that all exceedances cause an adverse effect.
                   But in general, one would expect that as the average dose in-
                   creases above the RfD, the percentage of the exposed population
                   experiencing adverse health effects would rise, or the severity of
                   the effects would increase, or both.

      3)     there is no systematic effort to describe and quantify the uncertainties in
             the analysis.  In Chapter 13 a range of effects is reported. But this
             range reflects only the two alternative baseline exposure scenarios.

      In addition to these limitations, the Committee also has reservations about
several features of the analysis:

      1)     The uncapped scenario is not plausible as a matter of public policy.  It
             assumes, in effect, that existing regulations affecting human exposures
             through a variety of pathways are ignored.  This scenario undoubtedly
             seriously overstates the number of cases of cancer and non-cancer
             health effects in the absence of the  rule and therefore overstates  the
             health benefits of the rule.

      2)     The report estimates baseline risks for people who move onto sites after
             they are dosed (p. 7-39 to 40). Apparently these risks were not used in
             the calculations for Table 7-23 (see  the first line on p. 7-41).  But  this
             raises the question of why these calculations were done  and reported in
             the first place.  In any case, it seems unlikely that in the  absence  of the
             rule, sites would be used in this way, given the high calculated risks.

     Jl) -   The report states that one big facility dominates the risk estimates.  It
             would be useful to see the results with this facility omitted. It also
             suggests a policy design issue: should the regulations have two tiers
             with stricter regulations for facilities like the one big one included in the
             sample?

      4)     The report uses IRIS cancer slope factors which are mostly 95 percent
             UCLs, not maximum likelihood estimates. The report is also based on
             other standard "conservative" practices in risk assessment Thus, the
             estimated health benefits cannot be considered to be expected values or
             most likely values. Given the level of effort devoted to other parts of the


                                      A-2

-------
             analysis, and given the potential economic impact of the Proposed Rule,
             more effort should be devoted to both expected values and upper and
             lower bounds for the health benefits.

      This decision to measure effects with count variables leads to the failure to take
account of the temporal distribution of health effects.  One hundred twenty-eight years
of health effects are aggregated as if they were equally important and measured with
equal precision. This is especially troublesome when it is recognized that these
aggregates (over 128 years) provide the basis for the cost effectiveness analysis -
comparing discounted costs to undiscounted physical effects. At a minimum, the
comparison should be made year by year and then discounted.

    b)  Feasibility of Using McClelland et al Estimates for National Estimates
                  of Non-use Values for Groundwater Cleanup

      Our earlier report (CV-1) raised a number of questions with the results of the
McClelland et. al contingent valuation study.  The focus of that review was on the per
household estimates of nonuse values for complete cleanup of contaminated ground-
water.  The overall judgment of the Committee was that we had no confidence that the
respondents to the McClelland et. al survey were clear about the commodity they were
being asked to value. This basic failing called into question the usefulness of the
willingness to pay (WTP) estimates as indications of a typical household value for
complete cleanup of contaminated groundwater.

      There are also a number of problems with adopting this perspective, including
serious discrepancies between the context of the commodities to be evaluated as part
of the effects of the planned RIA and what has been described to the respondents in
the McClelland et. al survey.  Equally important,  there are significant unresolved
issues in the benefit transfer procedures used. These problems are independent of
our review of the McClelland  et. al report and  relate exclusively to the use of their
estimate in the draft methodology document for the RIA.  The most important of our
concerns are:

     JU- Technical Questions

      0 At the dose of our review of the survey procedures, questionnaire and  esti-
      mates, it was suggested that the Committee might consider the feasibility of
      using the estimates as an approximate  upper bound on the per household WTP
      to meet the goal of an upper bound evaluation of this source of aggregate
      benefits generated by the Rule. There  is no basis in the McClelland et. al
      study, related literature (see Boyle [1993]) or the analysis presented in the RIA
      to assume the estimates are upper bounds for per household WTP.  The RIA
      does not use the full range of estimates developed by McClelland et. al but
      focuses instead on a comparison of two of the five scenarios, arguing incorrect-


                                     A-3

-------
ty, that such comparisons were judged to be the most reliable in the NOAA
panel report on contingent valuation (see pp 10-14 to 10-15). Nonuse values
were measured by the difference between the complete cleanup and the public
treatment scenarios.  No recommendation of this type was part of the NOAA
panel's report.

ii) The analysis did not rely on the primary data from the survey. The analysis
uses predictions from the Box-Cox model for WTP as if they were primary data,
performs a subsequent linear regression treating these predictions as depen-
dent variables and income plus regional dummy variables as independent
variables. Based on the McClelland et. al report, these variables were argu-
ments of the original Box-Cox models. This exercise has no basis in statistical
methods.  Predictions from this model were adjusted by a single (for all esti-
mates) adjustment factor of .503 to compute the WTP for public treatment.  The
difference was then the basis for the nonuse values.  It is hard to understand
why this strategy was adopted when primary data on the difference that is
sought are  available in version C of the McClelland et. al survey questionnaires.

These are not the only technical problems associated with the economic
methodology used in the benefit transfer.  They are examples and reinforce the
need for appropriate peer review of methods prior to the presentation  of materi-
als  to committees of the SAB.

2)  Benefit Transfer

i) The most fundamental problem arises with the source of the contamination.
All of the McClelland et. al questionnaire variations identify the source of
contamination  as a public landfill The majority of the sites affected by the
RCRA rule would not fit this description. The pretest and design work from the
McClelland et. al study suggest this distinction is very important. Virtually all of
the literature on people's responses to contamination of groundwater finds
similar results. The source matters to people's evaluation of the problem and to
the character of the response.
_«•" *
ii) The extent and timing of cleanup activities presented in the McClelland et. al
survey are completely different from all of the potential RCRA actions  and there
is no basis for gauging the temporal properties of the nonuse estimates in
response to changes identified as critical in the McClelland et. a/ pretest as well
as in the earlier Mitchell-Carson focus group analyses.

iii) The most fundamental issue giving rise to the range in estimates of the
aggregate nonuse values is the extent of the market Estimates range from
$170 million to $18 billion for aggregate nonuse values depending on  the
assumption about extent of the market This variation does not relate to the

                               A-4

-------
       variability in contingent valuation estimates or "to the Committee's concerns
       about the commodity understood by survey respondents. It results exclusively
       from assumptions about which households are actually concerned about each
       site.

       The RIA presents alternative estimates, identifies the fact that McClelland et. al
       use the phrase "in your community" to describe how households were intended
       to infer responsibility. Subsequent research is sketched in four pages of a later
       McClelland et. al [1993] report where it is suggested that the concept of
       community was examined in March 1993 post-testing of the survey.  The
       authors describe  a question added to the survey to elicit information about how
       respondents would evaluate their community. No information was provided in
       that report about  how the results used to propose a definition for the community
       (as a proxy  measure for the extent of the market) were derived - sample size,
       implementation, and correspondence of other variables for the new survey with
       the original survey were not discussed.

       These comments are intended to illustrate the substantive questions in the
       development of the nonuse benefit estimates. Taken together with the ques-
       tions about interpreting the per household estimates of nonuse values, we are
       forced to conclude that there is no basis for accepting any of the EPA estimates
       of nonuse values.

                             c) Hedonic Analyses

       The hedonic chapter and  "sanitized" companion report provide an analysis of
residential property values using a hedonic framework where housing prices are
related to the home and site characteristics including the distance from hazardous
waste treatment, storage and disposal facilities (TSDF) at three distinct sites.

       In each case study, a notable "event" took place at the TSDF during the
observation period. For two of the studies, distance from the TSDF became a
significant explanatory variable in the hedonic price equation only after the publicizing
of the Invent" In the third case study, distance from the site was strongly significant
throughout the entire period. The results from this study are qualitatively reasonable
and promising.  They are based on large samples of housing prices. But, the authors
and the Committee caution against adoption of the quantitative estimates of benefits
of remediation based on these studies at this time. We reiterate that list adding
additional reasons to those reported  and introducing some new concerns:

      1)    Functional form.  The report relies primarily on a linear hedonic model,
            although it  employs a semi-log and a piece-wise (in distance from site)
            linear form. The "benefit" estimates (see point 5 below) obtained seem
            quite sensitive to functional form and the authors admit the need to


                                     A-5

-------
       investigate.  Additionally, the implication of linear in characteristics
       models, i.e. constant marginal prices of characteristics, has not been
       found especially satisfactory in other studies. The fact that housing
       prices are being underestimated in some house size ranges and overes-
       timated in others suggests to them and to us the need to experiment with
       more flexible functional forms with respect to housing characteristics as
       well.

 2)    Explanatory power.  Little of the variation in housing prices is explained
       by the model probably because data are not available on important .
       housing characteristics (especially neighborhood characteristics). Both
       the report and the Committee agree on this point  This is a particularly
       difficult problem in one case study where the effects of one TSDF are
       being estimated, when in fact several such facilities exist in the surround-
       ing area.

       While it should be possible to incorporate some additional neighborhood
       characteristics, others may pose more complicated problems than the
       authors anticipate.  Some neighborhood characteristics might themselves
       be endogenous - brought about by the location of the facility.  The
       report suggests using socio-demographic characteristics of the neighbor-
       hood to explain price differences but these, too, may pose an
       endogeneity problem.

 3)    Outliers.  The report suggests that outliers (i.e. houses that sell for less
       than $50,000 and over $500,000) are quite influential in their estimates.
       Clearly the model is not explaining these very well. If, in addressing
       problems 1 and 2 above, this outlier problem  is not resolved, the Com-
       mittee suggests the authors adopt some of the numerous available
       statistical techniques for dealing with outliers rather than relying on ad
       hoc procedures.

 4)    Distance as a proxy variable. The authors use distance from a TSDF as
^ '   a proxy for the relative damage due to the site.  But, all effects from a
       site are not radially symmetric.  Prevailing winds and topographical
       features can increase/decrease the impact of noxious fumes and other
       aesthetic effects. The significance of these will  be case specific.

 5)    Benefit measurement Without any discussion, the report takes the
       change in the hedonic price function (as if each affected house were
       moved farther from the site) as a measure of benefits. Welfare measure-
       ment in hedonic models is extremely complicated and depends on such
       factors as the extent of the market, whether the market could be consid-
       ered closed or open,  the heterogeneity of participants, etc.  Even if one


                                A-6

-------
      considers only a short run, when indivfduals (and therefore the hedonic
      price functions) do not adjust, the hedonic price function is a locus of
      equilibrium points and not a demand function.

      Under certain circumstances, McConnell (in Kopp and Smith, Valuing
      Natural Assets [1993]) has suggested that the change in the hedonic
      price as predicted by the change in policy could serve as an approximate
      welfare measure. However, the report shows no appreciation  for this
      complex problem and needs to consider the factors that would make this
      a good or bad approximation, or an upper or lower bound.  From our
      understanding of the  problem, this predicted change is likely to be an
      upper bound on the welfare measure sought, but this needs to be
      examined.  Other literature of interest on the subject includes Palmquist
      (in Braden and Kolstad, eds., Measuring Demand for Environmental
      Commodities. [1991]; Kanemoto Econometrics [1988]; Cropper, Deck
      and McConnell REStat [1988]; Bartik and Smith in Mills, eds., Handbook
      of Urban Economics.  [1987].

6)    Repeat sales/time series.  In addition to the hedonic study described
      above, the report analyzed some time series data in a "repeat  sales"
      analysis for a small sample of houses that were sold more than once
      during the time period. Little came of that analysis  and, for a number of
      reasons, we do not recommend that they pursue this part of the study.
                              A-7

-------
                              REFERENCES

Boyle, K. J. A Review of Contingent Valuation Studies of the Benefits of Groundwater
      Protection (report to U.S. Environmental Protection Agency), Research Triangle
      Institute, April 1993).

McClelland, G.H., Schulze, W.D. Lazo, J.K., Waldman, D.M., Doyle, J.K., Elliott, S.R.
      and Irwin, J.R. Methods for Measuring Non-Use Values: A Contingent Valua-
      tion Study of Groundwater Cleanup (Draft report to the U.S. EPA), University of
      Colorado, Center for Economic Analysis,  October, 1992.
                                    R-1

-------
              UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
                            SCIENCE ADVISORY BOARD
               ENVIRONMENTAL ECONOMICS ADVISORY COMMITTEE
                                RCRA RIA REVIEW

 CO-CHAIRS
 Dr. Allen V. Kneese, Resources for the Future, Washington DC

 Dr. V. Kerry Smith, Department of Economics, North Carolina State University, Raleigh NC

 MEMBERS
 Dr. Nancy E. Bockstael, Department of Agricultural and Resource Economics, University of
 Maryland,  College Park, MD

 Dr. A. Myrick Freeman, Department of Economics, Bowdoin College, Brunswick, ME

 Dr. Charles D. Kolstad, Department of Economics, University of Illinois, Urbana, IL

 Dr. William Nordhaus,  Department of Economics, Yale University, New Haven CT

 Dr. Bryan  Norton, School of Public Policy, Georgia Institute of Technology, Atlanta GA

 Dr. Wallace E. Oates,  Department of Economics, University of Maryland,College Park, MD

 Dr. Paul R. Portney, Resources for the Future, Washington DC

 Dr. Robert Repetto, World Resources Institute, Washington, DC

 Dr. Richard Schmalensee, Massachusetts Institute of Technology, Cambridge MA

 Dr. Robert N. Stavins,  Kennedy School of Government, Harvard  University, Cambridge, MA

 Dr. Thomas H. Tietenberg, Department of Economics, Colby College, Waterville, ME

 Dr. W. Kip Viscusi, Department of Economics, Duke University, Durham, NC

 SAB COMMITTEE LIAISONS
 Dr. William Cooper (EPEC), University of Michigan
 Mr. Richard Conway (EEC), Union Carbide Corporation
 Dr. Morton LJppmann (IAQC), Nelson Environmental Institute, New York University
 Dr. Roger McCleUan (CASAC), Chemical Industry Institute of Toxicology

 DESIGNATED FEDERAL OFFICER
 Mr. Samuel Rondberg, Environmental Health Committee, Science Advisory Board (1400F),
 U.S. Environmental Protection Agency, Washington, DC 20460

STAFF SECRETARY
Ms. Mary L Winston, Environmental  Protection Agency, Science Advisory Board (1400F),
Washington, DC 20460

-------
                                   NOTICE

    This report has been written as a part of the activities of the Science Advisory
Board, a public advisory group providing extramural scientific information and advice
to the Administrator and other officials of the Environmental Protection Agency. The
Board is structured to provide balanced, expert assessment of scientific matters
related to problems facing the Agency.  This report has not been reviewed for
approval by the Agency and, hence, the contents of this report do not necessarily
represent the views and policies of the Environmental Protection Agency, nor of other
agencies in the Executive Branch of the Federal government, nor does mention of
trade names or commercial products constitute a recommendation for use.

-------
EPA-SAB-EEC-94-002   'Review   of   MMSOILS
component of the  Proposed  RIA for the RCRA
Corrective  Action Rule"  by the Environmental
Engineering Committee

-------
      United States      Science Advisory        EPA-SAB-EEC-94-002
      Environmental      Board (1400F)         November 1993
      Protection Agency
xvEPA AN SAB REPORT: REVIEW
      OF MMSOILS COMPONENT
      OF THE PROPOSED RIA FOR
      THE RCRA CORRECTIVE
      ACTION RULE
     REVIEW OF THE OSWER & ORD
     DRAFT DOCUMENTATION AND
     USER'S MANUAL AND RIA OF
     THE MMSOILS MULTIMEDIA
     CONTAMINANT, FATE, TRANSPORT,
     AND EXPOSURE MODEL

-------
              UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
                           WASHINGTON. D.C. 20460
                              November 19, 1993            OFFICE OF THE ADMINISTRATOR
                                          '                   SCIENCE ADVISORY BOARD
EPA-SAB-EEC-94-002
Honorable Carol M. Browner
Administrator
U.S. Environmental Protection Agency
401 M Street, S.W.
Washington, D.C. 20460

      Re:   Review of MMSOILS Component of the Proposed Regulatory
            Impact Analysis (RIA) for the RCRA Corrective Action Rule

Dear Ms. Browner:

      The Science Advisory Board (SAB) is pleased to submit its report on review
of the Agency's draft document entitled "MMSOILS: Multimedia Contaminant
Fate, Transport and Exposure Model Documentation and User's Manual," dated
September 1992.  The MMSOILS (Multi-Media Contaminant, Fate, Transport and
Exposure Model) document was developed jointly by the Office of Research and
Development's (ORD's) Office of Health and Environmental Assessment (OHEA),
Exposure Assessment Group  (EAG) and the Office of Environmental Processes and
Effects Research (OEPER).  This  report by the MMSOILS Model Review
Subcommittee (MMRS) was prepared as part of the SAB's review of the "Draft
Regulatory Impact Analysis for the Final Rulemaking on Corrective Action for
Solid Waste Management Units: Proposed Methodology for Analysis."  Our report
resulted from the MMRS public reviews on April 22 and 23 and June 29, 1993.

      The Agency, through the Office of Solid Waste and Emergency Response
(OSWER) asked the SAB to review specific elements of the multi-media
contaminant fate, transport and exposure model,  MMSOILS, with regard to the
methodology used to predict contaminant concentrations in the  environment and
the resultant implications on human health and ecological risk assessments.
Specifically, the review dealt  with:

      a)     the adequacy of methods for using a screening level model where
            there is substantial subsurface heterogeneity and/or where non-
            aqueous phase liquids (NAPLs) are present,

      b)     the appropriateness of the Agency's  approach for aggregating releases
            from solid waste management units  (SWMUs)  in order to estimate
            concentrations at exposure points as a function of time, and
                                                                 Piw»do«upwindo«»m«

-------
       c)     the adequacy of the Agency's approach for developing long-term
             effectiveness and failure scenarios for site remedies.

       The OSW/ORD working group is to be commended for a well-coordinated
and focused effort to develop a regulatory impact analysis (RIA) that will help the
Agency and the Nation better understand the costs and benefits of the proposed
rule. The Subcommittee wishes to recognize the responsiveness and progress made
by the OSW/ORD working group to many of the recommendations made at the
April 22-23, 1993 meeting and candidly displayed in detail at the June 29, 1993
meeting.  The Subcommittee appreciates and recognizes the significant effort
expended to date, the positive attitude and open candor displayed by the
P^W/ORD working group in their presentations and interactions during the two
review meetings.  The Subcommittee considers the intraagency coordination
represented by this RIA to be a "model approach" that the Agency would do well
to adopt in other programs.

      On the positive side, the Subcommittee observes that  MMSOILS uses
simple, conservative, and computationally efficient equations for estimating
chemical transport via ground water, surface water, soil erosion, atmospheric, and
food chain pathways. Pathway documentation is well organized with appropriate
references. Applied mathematical formulae are widely used  and accepted by the
scientific community for use in simple situations. Underlying assumptions have
been identified, clearly stated, and appear to be reasonable yet not overly
restrictive. Given these  strengths, MMSOILS, when applied  to simplified case
studies, might certainly be a valid screening tool for assessing the relative risks
and  costs associated with  alternative regulatory options.

      However, the Subcommittee notes that two problems create unquantifiable
uncertainties that seriously diminish the utility of MMSOILS relative to its use in
the draft Corrective Action Regulatory Impact Analysis (RIA), namely:

      a)  inaccurate input parameters, and

      b)  application of the model to cases outside its range of validity.

      Inadequate input data are a consequence of sparse or  inaccurate
information, poor parameter estimation especially relative to source terms, and
suspected over-reliance upon default parameters.  The Subcommittee recommends
a documented and thorough peer review of all aspects of the data base, focusing
particularly on those parameters  to which the results are most sensitive.

      Equally serious is the inappropriate application of the MMSOILS model to
scenarios  for which it was not intended, such as sites with complex hydrogeological
conditions or sites where NAPLs ar<, ^resent.  To some extent, as discussed in the
Subcommittee's full report, the latter could be addressed by  means of appropriate

-------
 revisions to model formulations.  However, for a significant number of sites, the
 Subcommittee suspects that no generic model is likely to provide answers of
 acceptable quality.  OSWER is generally aware of the limited usefulness of generic
 models for the analysis of complex environmental settings.

       The Subcommittee also observes that uncertainly analysis for the RIA is in
 its infancy and will require much greater effort to  meet the needs of the
 assessment process.  Given the high stakes involved in terms of potential
 commitment of national resources, defensible  estimates of the uncertainties
 associated with risk" and benefits are critical, and the protocol followed to obtain
 such estimates deserves as much forethought  and careful peer review as that
 required to obtain the central estimate.  As a related issue, the Subcommittee is
 concerned that the simple protocol followed to obtain high-end risk estimates may
 be inadequate  in that these estimates  in some cases apparently gave rise to lower
 exposures than did the central tendency estimate.

       Given these serious shortcomings, many of which  were already recognized
 by the Agency, the most basic and pressing concern of the Subcommittee is
 whether the use of a generic model such as MMSOILS is appropriate as a basis
 for the assessment of regulatory costs  and benefits  at the national level, given the
 fate and transport estimates that comprise the model output may be wrong by
 orders of magnitude for many complex sites.  We recommend that the Agency:

      a)     augment its RIA with cost/benefit estimates derived by alternative
             approaches, such as:

             1)    utilizing assessment data generated for Superfund sites,

            2)    using more sophisticated models with better-defined data to
                  develop estimates for representative sets of waste sites, or

            3)    applying site-specific models to analyze that relatively small
                  number of facilities which MMSOILS results indicate dominate
                  the total costs or risks, and

      b)    at a  minimum,  expert review of the latter cases should  be undertaken
            to judge the reasonableness of model outputs.

This augmentation should help validate the present reliance on the screening
studies that use MMSOILS model  output as a starting point.

-------
      The SAB appreciates the opportunity to comment on the EPA's MMSOILS
model.  We are gratified that the Agency has brought this issue before us and look
forward to receiving a summary of the EPA's response, particularly to the points
raised in this letter to you.

                                   Sincerely,
Dr. Raymond C. Loehr, Chair      Mr. Richard A. Conway, Chair
Executive Committee              Environmental Engineering Committee
Science Advisory Board            Science Advisory Board
                                 Dr. C. Herb Ward, Chair
                                 MMSOILS Model Review Subcommittee
                                 Environmental Engineering Committee
                                 Science Advisory Board

-------
                                   NOTICE

      This report has been written as a part of the activities of the Science
Advisory Board, a public advisory group providing extramural scientific
information and advice to the Administrator and other officials of the
Environmental Protection Agency.  The Board is structured to provide a balanced,
expert assessment of scientific matters related to problems facing the Agency.  The
report has not been reviewed for approval by the Agency; hence, the comments of
this report do not necessarily represent the views and policies of the
Environmental Protection Agency or of other federal agencies. Any mention of
trade names or commercial products does not constitute endorsement or
recommendation for use.

-------
                                  ABSTRACT

       The MMSOILS Model Review Subcommittee (MMRS) of the Environmental
 Engineering Committee (EEC) of the EPA Science Advisory Board (SAB) has
 prepared a report on the Agency's Office of Solid Waste (OSW) MMSOILS
 Multimedia Contaminant Fate, Transport, and Exposure Model.  This model and
 guidance document was developed as a technical  resource for estimating potential
 health risks  at sites contaminated by toxic wastes or spills of toxic chemicals.

       The review by the  SAB's MMRS dealt with the adequacy of methods for
 using a screening level model where there is substantial subsurface heterogeneity
 or where non-aqueous phase contaminants are present,  the appropriateness of the
 Agency's approach for aggregating releases from solid waste management units
 (SWMUs) to estimate concentration at exposure points over time, and the
 adequacy of the Agency's approach for developing long-term effectiveness and
 failure scenarios for site remedies.

      The general consensus of the MMRS was that the use of a multimedia
 pathway model for screening purposes could be an appropriate approach for
 developing risk and cost estimates for a national-level Regulatory Impact Analysis
 (RIA), as long- as the input parameters are accurate and the model is  not applied
 outside its range of validity.  Furthermore, the Agency's use of a  single  model, to
 the extent defensible for each facility considered,  was viewed by the MMRS as
 necessary in  order to ensure consistency among model results. The major
 overriding concerns of the MMRS were:  a) application of MMSOILS outside its
 range of validity; b) large uncertainties in input parameters; c) consequent large
 uncertainties in MMSOILS results; d) clear communication of this uncertainty to
 decision-makers; and e) presentation of the results in the draft RIA document in a
 scientifically  defensible manner that communicates the uncertainties of the
 calculations and their implications for the cost/benefit analysis.

      The MMRS recommended that the Agency  augment the MMSOILS results
 with cost/benefit estimates derived by alternative  approaches,  such as  utilizing
 assessment data generated for Superfund sites, using more sophisticated models
 with better-defined data to develop estimates for representative sets of waste sites,
 applying site-specific models to analyze that relatively small number of facilities
 which MMSOILS results indicate dominate the total costs or risks, and submission
 of selected case studies to expert panel review.
Key Words: Mathematical Models, Cleanup, Corrective Action, Regulatory Impact
Analysis, RCRA Models
                                      11

-------
               U.S. ENVIRONMENTAL PROTECTION AGENCY
                           Science Advisory Board
                    Environmental Engineering Committee
                    MMSOILS Model Review Subcommittee
CHAIR
Dr. C. Herb Ward, Professor and Director, Energy and Environmental Systems
Institute, Rice University, Houston, TX

VICE CHAIR
Dr. George F.  Carpenter, Environmental Quality Manager, Environmental
Response Division, Michigan Department of Natural Resources, Lansing MI

           AND CONSULTANTS
Dr. Steven M. Bartell, Vice President, Center for Risk Analysis, SENES Oak
Ridge, Inc., Oak Ridge, TN

Dr. Randall J. Charbeneau, Professor and Director, Center for Research in Water
Resources, Department of Civil Engineering, University of Texas, Austin, TX

Dr. Calvin C. Chien, Principal Consultant, duPont Corporate Remediation Group,
E.I. duPont de Nemours  & Company, Wilmington, DE

Dr. Rolf Hartung, Professor of Environmental Toxicology, Department of
Environmental and Industrial Health, School of Public Health, University of
Michigan, Ann Arbor, MI

Dr. Wayne M. Kachel, Corporate Environmental Management, Martin Marietta
Corporation, Oak Ridge,  TN

Dr. June Fabryka-Martin, Hydrogeologist, Isotope and Nuclear Chemistry Division,
Los Alamos National Laboratory, Los Alamos, NM

Dr. Ishwar P. Murarka, Senior Program Manager, Land & Water Quality Studies,
Environmental Division,  Electric Power Research Institute, Palo Alto, CA

Dr. Bernard Weiss, Professor, Department of Environmental Medicine, University
of Rochester Medical Center, Rochester,  NY

Science Advisory Board Staff
Dr. K. Jack Kooyoomjian, Designated Federal Official, U.S. EPA, Science Advisory
Board, (1400F), 401 M Street, SW, Washington, DC 20460

Mrs. Diana L. Pozun, Staff Secretary

Dr. Donald G. Barnes, Staff Director, Science Advisory Board
                                     in

-------
                          TABLE OF CONTENTS

 1. EXECUTIVE SUMMARY	  1
      1.1 Overall Comments	  1
      1.2 Response to Charge  	  2
      1.3 Additional Observations and Recommendations	  4
            1.3.1 Model Selection, Development, Formulation and
                  Documentation  	  4
            1.3.2  Possible Improvements to Model Formulations for
                  Specific Processes  	  5
            1.3.3 Issues of Parameter Estimation 	  6
            1.3.4 Issues of Range of Model Validity	  7
            1.3.5 Issues Relating to Pathway Model Verification and
                  Validation	   7
            1.3.6 Comments on Remediation Effectiveness	   8
            1.3.7 Issues Relating to Assessment of Uncertainty  	   9
            1.3.8 Comments on Results for Health Risk Analysis	   9
            1.3.9 Comments on Use of MMSOILS in Corrective Action RIA .   10
            1.3.10 Other User Groups for MMSOILS	   11

2. INTRODUCTION  	   12
      2.1  Charge for SAB Review  	   12
      2.2  SAB Review Procedure	   12

3. COMMENTS ON MODEL SELECTION, FORMULATION,
      DOCUMENTATION AND APPLICATION  	   14
      3.1  MMSOILS Selection,  Development, Formulation and
            Documentation  	   14
            3.1.1 Model Selection and Development	   14
            3.1.2 Use of Standard Formulations	   15
            3.1.3 Documentation of Modeled Pathways  	   15
            3.1.4 Documentation of Assumptions Underlying Multimedia
                 Treatment   	   15
      3.2    Possible Improvements to Model Formulations for Specific
            Processes 	   16
            3.2.1 Additional Types of Solid Waste Management Units
                 (SWMUs)	   16
            3.2.2 Recognition of Natural Biodegradation Processes in
                 Ground Water Pathway	   16
            3.2.3 Modeling Transport through the Vadose Zone	   17
            3.2.4 Plume Aggregation in Groundwater Pathway	   17
            3.2.5 Food-Chain Module	   17
            3.2.6 Mass Balance	   18
           3.2.7 Disparity in Relevant Time and Space Scales  for
                 Transport Mechanisms	   18
            3.2.8 Relevant Time Scales for Ecological Risk Assessment ....   18
      3.3  Issues  of Parameter Estimation	   19

                                    iv

-------
              TABLE OF CONTENTS: CONTINUED:

      3.3.1 Source Term Parameters	   19
      3.3.2 Waste Release and Solubility	   20
      3.3.3 Use of Default Values	   21
      3.3.4 Peer Review of Data Base   	   21
      3.3.5 Data Base for Future Related Modeling Efforts	   21
3.4  Issues of Range of Model Validity  	   21
      3.4.1 Extreme Events	   22
      3.4.2 Complex Sites  	   23
      3.4.3 NAPLs   	   23
      3.4.4 Development of Guidelines for Assessing Model
             Applicability to Specific Cases	   23
3.5  Issues Relating to Pathway Model Calibration, Verification and
      Validation	   24
      3.5.1 Ground Water Model   	   24
      3.5.2 Other Pathway Models  	   24
      3.5.3 Guidelines for Calibration, Verification and Validation  ...   24
3.6  Comments on Remediation Effectiveness	   24
      3.6.1 Remediation Times	   24
      3.6.2 Effect of Unknown Presence of DNAPLs on Remediation
             Times   	   25
      3.6.3 Remediation Effectiveness	   25
      3.6.4 Inclusion of Biologically-Based Remediation Technologies .   26
      3.6.5 Risks of Remediation   	   26
3.7  Issues Relating to Assessment of Uncertainty 	   26
      3.7.1 Uncertainty Estimation Protocol	   26
      3.7.2 Development of High-End Risk Estimates	   27
3.8  Interpretation of Results for Health Risk Analysis	   27
      3.8.1 Health Risk as the Assessment Endpoint   	   27
      3.8.2 Empirical Validation of Exposure Estimates  	   28
      3.8.3 Inconsistent Treatment of Cancer and Noncancer Health
             Risks	   28
      3.8.4 Inaccurate Identification of Critical Health Effects	   28
      3.8.5 Questionable Treatment of  Different Waste Classes  	   29
      3.8.6 Other Sources of Hazardous Wastes	   29
3.9  Comments on Use of MMSOILS in  Corrective Action RIA   	   29
      3.9.1 Facility  Selection Process	   29
      3.9.2 Use for  National-Level Screening	   30
      3.9.3 Presentation of Results in Corrective Action RIA	   30
      3.9.4 Presentation of Uncertainty Analysis in RIA	   31
3.10 Other User Groups for MMSOILS  	   31
      3.10.1  Applicability to  Other EPA Program Activities	   31
      3.10.2  Use for State-Level Screening	   31
      3.10.3  Other User Groups	   31

-------
               TABLE OF CONTENTS: CONTINUED:



APPENDIX A - BRIEFING AND REVIEW MATERIALS 	 A-l



APPENDK B - REFERENCES CITED  	 B-l



APPENDK C - GLOSSARY OF TERMS AND ACRONYMS  	 C-l
                           VI

-------
                          1. EXECUTIVE SUMMARY
       In response to a request from the Office of Solid Waste and Emergency
 Response (OSWER), the Science Advisory Board (SAB) has reviewed several
 aspects of the draft Regulatory Impact Analysis (RIA) prepared in support of the
 Resource Conservation and Recovery Act (RCRA) Corrective Action Rule.  At the
 October 1992 meeting, the SAB Executive Committee, recognizing the importance,
 complexity, and creativity of OSWER's work and its multi-disciplinary nature,
 established an ad hoc Steering Committee to assure that certain significant aspects
 of the RIA - both methodology and application - received appropriate attention
 from the relevant SAB standing Committees.

      At a public meeting on January 29, 1993, the Steering Committee concluded,
 based on presentations by and discussions with OSWER staff, that four SAB
 committees, with appropriate  inter-committee liaison participation, should review
 major segments of the RCRA  Corrective Action RIA as follows:  the Environmental
 Economics Advisory Committee (EEAC) would review the Contingent Valuation
 (CV) methodology and its application in the RIA;  the Environmental Engineering
 Committee (EEC) would review the MMSOILS multi-media contaminant fate,
 transport and exposure model; the Ecological Processes and Effects Committee
 (EPEC) would review the ecological risk  analysis; and the Environmental Health
 Committee (EHC) would review the human health risk assessment.  In addition,
 the Steering Committee agreed to prepare an overview report to accompany the
 individual committee reports.

      The MMSOILS Model Review Subcommittee (MMRS) of the EEC reviewed
 the Agency's  draft document entitled "MMSOILS: Multimedia Contaminant Fate,
 Transport and Exposure Model Documentation and  User's Manual,"  dated
 September 1992 (See Appendix B, Reference 6), as well as the supporting RIA and
 Appendices (See Appendix B, References  7 & 8). The draft  documentation and
 user's manual was developed jointly by the Office of Research and Development's
 (ORD's) Office of Health and  Environmental Assessment (OHEA)  and Office of
 Environmental Processes and  Effects Research (OEPER). The MMRS report
 resulted from a review of the  above draft documents and briefing materials at
 meetings on April 22-23, 1993 and June 29,  1993 (See Appendix A, and Appendix
 B; References 6 through 8).

 1.1  Overall Comments

      The OSW/ORD working group is to be commended for a well-coordinated
 and  focused effort to develop a regulatory impact analysis (RIA) that will help the
Agency and the Nation better understand the costs  and benefits of the proposed
 rule. The Subcommittee wishes to recognize the responsiveness and progress made
by the OSW/ORD working group to many of the recommendations made at the
April 22-23, 1993 meeting and candidly displayed in detail at the June 29, 1993
 meeting. The Subcommittee appreciates  and recognizes  the significant effort

-------
expended to date, the positive attitude and open candor displayed by the
OSW/ORD working group in their presentations and interactions during the two
review meetings. The Subcommittee considers the intraagency coordination
represented by this RIA to be a "model approach" that the Agency would do well
to adopt in other programs.

      The  consensus of the MMSOILS Model Review Subcommittee (MMRS) is
that the use of a multimedia pathway model for screening purposes could be an
appropriate approach for developing risk and cost estimates for a national-level
RIA as long as the input parameters are accurate and the model is not applied
outside  its  range of validity.  The Agency's use of a single model, to the extent
defensible,  ensures consistency among model results.

      The  major overriding concerns of the MMRS are the application of
MMSOILS  outside its range of validity; large uncertainties in input parameters;   .
consequent large uncertainties in MMSOILS results; clear communication of this
uncertainty to decision-makers; and the generation of credible guidance on
exposure, risk, costs, and benefits.  Consequently, the recommendations contained
in this report are focussed at efforts to decrease the level of uncertainty, to
validate the MMSOILS results by comparison with alternative estimation methods,
and to ensure that the results of the modeling exercise are expressed in the RIA
background documents in a scientifically defensible manner that communicates the
uncertainties of the calculations and their implications  for the cost/benefit analysis.

1.2  Response to Charge

      The  following issues were presented in the charge  to the Subcommittee.
(Please  note that numbers following specific observations and recommendations
refer the reader to more detailed discussion in Section  3  of this review report.):

Issue 1.     The adequacy of methods for using a screening level model to
            characterize situations where there is a substantial subsurface
            heterogeneity or where  non-aqueous phase contaminants are present.

      While the Subcommittee (the MMRS) agrees that a screening-level model
may be appropriate for developing risk and cost estimates for  a national-level RIA,
the MMRS recommends that the current version of MMSOILS not be applied to
the characterization of contaminant  distributions in ground water in complex
hydrogeological settings or where Non-Aqueous Phase Liquids (NAPLs) may be
present. For these facilities, the MMRS recommends that alternative approaches
to characterization should be used.  Such approaches include modification of the
MMSOILS ground water module to more accurately model contaminant movement
in complex hydrogeologic settings; utilization of assessment data generated at
Superfund sites; application of more sophisticated models with better-defined data
to develop estimates for representative sets of waste sites; application of site-
specific models to analyze that relatively small number of facilities which
MMSOILS results indicate dominate the total costs or risks;  and submission of

-------
 these case studies to expert panel review to develop estimates of contaminant
 migration.

 Issue 2.    Appropriateness of the Agency's approach for aggregating releases
            from solid waste management units (SWMUs, the source terms for
            the contaminant modeling) to estimate concentration at exposure
            points over time.

      The Subcommittee is concerned that the method of aggregation used to
 obtain the concentration distributions for application to individual wells may not
 conserve mass. Rather, it appears that the resulting apparent mass and the
 average concentrations for each  concentric ring downgradient from the SWMUs
 will always exceed that from the untransformed plumes.  Such an approach may
 nonetheless be defensible for the purposes of the RIA because  it is conservative;
 however, the MMRS recommends that the degree of conservatism be evaluated
 through comparison with a number of simulations which do not use this method
 of aggregation. Unduly conservative estimates can cause the inappropriate
 prioritization of risks.  In addition, the MMRS recommends that the Agency
 evaluate whether movement of the contaminant plumes could result in a decreased
 concentration  for population wells.  The required transformations from cartesian
 to cylindrical coordinates should not require much computational effort compared
 with that  required for the model to begin with.

 Issue 3.     Adequacy of the Agency's approach for developing long-term
            effectiveness and failure scenarios for site remedies.

      The Subcommittee observes  that the annual time scale for exposure
 estimates  produced by MMSOILS may be inappropriate for many ecological
 applications.  Typical organisms  of concern exhibit short life spans, or critical
 stages in their complex life histories that occur at time scales substantially shorter
 than one year. Thus, the Subcommittee recommends the modification of
 MMSOILS to produce more realistically-scaled  exposures for meaningful  inputs to
 ecological  risk analysis (Recommendation #13; also Section 3.2.8). The
 Subcommittee recommends that  the ecological risk assessment  component be
 constructed using the principles  for ecological risk  assessment as suggested by the
 Risk Assessment Forum. (Recommendation #9; also Section 3.2.5; See also
Appendix  B, Reference 23). The  Subcommittee further recommends that
 ecologically relevant exposure scenarios be modified so as to be capable of
 simulating acute impacts from waste sites on aquatic environments due to surface
 run-off after major rain events. (Recommendation #10; also Section  3.2.5).

 Issue 4.     The implications of the fate and transport modeling assumptions on
            the ecological and human risk assessment.

      The Subcommittee observes  that certain hazardous agents are not easily
 controlled and may pose health risks beyond the substances discussed in the RIA.
 (Observation #43; Section 3.8.6). The Subcommittee recommends that the Agency

-------
 revise its practices for assessing cancer and noncancer health risks so as to make
 them more consistent with one-another (Recommendation #40; Section 3.8.3). The
 Subcommittee also recommends that the Agency review its assumption of
 additivity of Hazard Indices, and additivity of risks across Class A (known) and
 Class C (suspected) carcinogens (Recommendation # 42; Section 3.8.5).  The
 Subcommittee further recommends that the Agency review its discussion of critical
 health effects and correct any inaccurate information (Recommendation #41;
 Section 3.8.4).

      The Subcommittee recommends that the Agency consider how the general
 validity of its exposure estimates  might be tested by comparison with empirical
 field data.  This is being recommended as a result of the observation by the
 Subcommittee that the translation of contaminant concentrations to estimates of
 exposure necessarily involves a long chain of assumptions and  requires the
 adoption of parameter values of variable uncertainty (Recommendation #39;
 Section 3.8.2).

 1.3 Additional Observations and  Recommendations

 1.3.1  Model Selection, Development, Formulation and Documentation

      Recommendation 1. The MMRS recommends that the criteria and rationale
for the selection of MMSOILS be  more fully documented in the RIA so that the
scientific and strategic bases for the selection will be clear to all concerned -
regulator, regulated, and scientific/risk assessment/economic communities at large.
 (3.1.1)

      Observation 2.  The model  uses simple, conservative, and computationally
efficient equations for estimating  chemical transport via ground water, surface
water, soil  erosion, the atmosphere, and foodchains.  Mathematical formulae used
to estimate transport rates for each pathway are widely used and accepted by the
scientific community for application to simple situations. Underlying assumptions
for each pathway model have been identified, are clearly stated, are reasonable and
are not overly restrictive.  However, for a significant number of sites, the MMRS
suspects that no generic model is  likely to provide answers of acceptable quality.
OSWER is generally aware of the limited usefulness of generic models for the
analysis of complex environmental systems, including aquifers.  (3.1.2)

      Recommendation 3. While  documentation of the formulations for individual
pathway models is well organized with appropriate references, the manual would
benefit from another round of editing.  (3.1.3)

      Recommendation 4. Documentation for MMSOILS would benefit from a
concise and explicit presentation «r the model's basis, assumptions and limitations
in a central location at the beginning of the user's manual. (3.1.4)

-------
 1.3.2   Possible Improvements to Model Formulations for Specific Processes

       Recommendation 5.  MMSOILS should be modified to make it capable of
 handling other, potentially more costly, types of SWMUs such as leaky sewer
 systems which are currently excluded from the RIA.  Eventually these problematic
 SWMUs will be impacted by the proposed corrective action rule, so an estimate of
 their cost contribution to the implementation of the proposed rule should be
 developed. (3.2.1)

       Recommendation 6.  Because of the long time periods involved, the MMRS
 believes that it is critical that the role of natural biodegradation processes be
 explicitly incorporated into the ground water fate and transport pathway by the
 use of an appropriate biodegradation coefficient  value.  This function is essential
 and provides realism for actual mechanisms taking place.  Even a small
 biodegradation coefficient would have a big impact.(3.2.2)

      Recommendation 7.  The MMRS recommends that the unsaturated-zone
 transport module be replaced with a simple kinematic model in order to make its
 treatment consistent with the other transport process models.  (3.2.3)

      Recommendation 8.  The MMRS recommends that the Agency quantitatively
 assess the degree of conservation introduced by its method of plume aggregation
 through a comparison with simulations which do not use this method. As a part
 of this exercise, the Agency should  quantitatively evaluate whether movement of
 the plumes could result in a decreased concentration for population wells. (3.2.4)

      Recommendation 9.  The MMRS recommends that the ecological risk
 assessment component be constructed using the principles for ecological risk
 assessment as suggested by the Risk Assessment Forum.  (3.2.5)

      Recommendation 10.  The MMRS recommends that the ecologically relevant
 exposure scenarios be  modified so as to be capable of simulating acute impacts
 from waste sites on aquatic environments due to surface run-off after major rain
 events.  (3.2.5)

      Observation 11.  The MMRS is concerned  that the MMSOILS model may
not effectively estimate long-term consequences of remediation alternatives due to
a suspected breakdown of mass balance as a result of model output post-
processing.  (3.2.6)

      Observation 12.  The MMRS notes that a  major problem that must be
confronted in the development of any multimedia model, such as MMSOILS, is the
forcing of differently scaled environmental transport processes into a single model
construct. Attempts to force disparate scales into a single model by selecting a
compromise in time step will necessarily result in a loss of accuracy in model
predictions.  (3.2.7)

-------
      Recommendation 13. The Subcommittee recommends the modification of
MMSOILS to produce more realistically-scaled exposures for meaningful inputs to
ecological risk analysis. This recommendation results from the observation that
the annual time scale for exposure estimates produced by MMSOILS may be
inappropriate for many ecological applications.  Typical organisms of concern
exhibit short life spans, or critical stages in their complex life histories that occur
at time scales substantially shorter than one year.   (3.2.8)

1.3.3  Issues of Parameter Estimation

      Recommendation 14.  The Subcommittee recommends that the Agency
ensure that the uncertainty estimates in the RIA fairly reflect the uncertainties in
quantification of the source term of the model input. The MMRS believes that the
largest single source of uncertainly in the risk analysis is probably that related to
quantification of the source term. Problems include sparse or inaccurate
information on identification of types of wastes present (e.g., presence of NAPLa),
on quantification of waste quantities, and on estimation of waste  distribution.
(3.3.1)

      Recommendation 15. The MMRS recommends that the Agency consider
the quantity and quality of waste information  as a reasonable criterion or
requirement for the inclusion of a particular facility in the facility selection
process.  The Subcommittee believes that the expected improvement of the
confidence in the modeling results is obvious.   (3.3.1)

      Observation 16.  The MMRS observes that the uncertainty  of the waste
transport calculations may be increased by the fact that the  existing data that
have been developed for SWMUs were generally not constructed or collected for
the purpose of estimating risks to humans or to ecosystems, but rather for the
purpose of defining the extent of contamination at a site rather than defining the
exposures at or near the site. (3.3.1)

      Recommendation 17. The MMRS recommends that the solubility models
used for metals and organics be submitted to peer review to assess their scientific
basis and limitations. (3.3.2)

      Recommendation 18. The MMRS recommends that the input data for the
case studies undergo peer review in order to evaluate a suspected over-reliance on
the use of default parameter values.  (3.3.3)

      Recommendation 19. The MMRS recommends a documented and thorough
peer review of all aspects of the data base, focusing particularly on those
parameters to which the results are most sensitive.  (3.3.4)

-------
       Recommendation 20.  The MMRS recommends that the Agency build upon
 the extensive data base it has accumulated for the Corrective Action RIA, to begin
 the development of an extensive data base that could be tapped for other EPA
 programmatic efforts, such as for a comparable assessment of the risks associated
 with NORM wastes and radiologically contaminated sites. The intraagency
 modeling task force, the Ad Hoc Agency Task Force on Environmental Regulatory
 Modeling (AFTERM), may be an appropriate vehicle for organizing and
 coordinating such an effort in a manner that would be most beneficial to the
 potential users.  (3.3.5)

 1.3.4  Issues of Range of Model Validity

      Recommendation 21.  The MMRS recommends that the Agency evaluate the
 validity of each pathway model to assess the extent to which extreme events might
 be expected to contribute to the bulk of contaminant releases, and the extent to
 which the model may under- or over-estimate transport. (3.4.1)

      Recommendation 22.  For facilities in complex hydrogeological settings
 outside the range of validity of the MMSOILS model, the MMRS recommends that
 alternative  approaches to characterization be used. Examples include the
 following: modification of the ground water module in MMSOILS to more
 accurately model contaminant movement under these conditions; utilization of
 assessment data generated for Superfund sites; application of more sophisticated
models with better-defined data to develop estimates for representative sets of
waste sites; application of site-specific models to analyze that relatively small
 number of facilities which MMSOILS results indicate dominate the total costs or
 risks;  and submission of these case studies to expert panel review to develop
 estimates of contaminant migration.  (3.4.2)

      Recommendation 23.   The MMRS strongly endorses ORD's recommendation
that the Agency develop an improved screening-level model for non-aqueous phase
liquid (NAIL) transport, either by modification of the existing MMSOILS model or
by conducting independent modeling exercises.  (3.4.3)

      Recommendation 24.   The MMRS recommends that the Agency develop
guidelines - perhaps including a requirement for peer review for key case studies -
in order to assess the applicability of MMSOILS to specific cases. (3.4.4)

1.3.5  Issues Relating to Pathway Model Verification and Validation

      Recommendation 25.   The MMRS recommends that the Agency prepare a
documented comparison of model predictions of chemical transport to field data
that would strengthen the scientific credibility of the results and provide a basis
for  readers to evaluate the model validity and magnitude of uncertainty.  For
similar reasons, the MMRS recommends that, for a subset of SWMUs where
ground water plume predictions are made by using MMSOILS, NATIVEPACMS
models also be exercised so as to permit comparison of plume predictions. (3.5.1)

-------
      Recommendation 26. The MMRS recommends that documented validation
exercises be undertaken for the remaining environmental transport pathways, e.g.t
aerosolization, volatilization, surface water runoff and bioaccumulation, in order to
assess the ability of these pathways models to provide meaningful input to the
RIA,  (3.5.2)

      Recommendation 27. The MMRS recommends that the Agency, perhaps
through AFTERM, develop generic guidelines for model calibration, verification
and validation, including  criteria for judging whether or not discrepancies among
alternative modeling results or between calculated and measured field data are
significant  In the case of MMSOILS, it recommends that the Agency undertake a
root-cause analysis for discrepancies, where significant, in order to evaluate the
potential for systematic bias in the modeling approach.  (3.5.3)

1.3.6  Comments on Remediation Effectiveness

      Recommendation 28.  The MMRS recommends that the sensitivity of the
RIA conclusions to these estimated remediation clean-up times be evaluated.  This
recommendation is made from the observation that experience gained from the
Superfund program with respect to remediation effectiveness and time has shown
that time estimates are commonly overly optimistic for ground water extraction
systems.  Because of unidentified sources, vadose zone contamination,
heterogeneities, and the unknown presence of NAPLs, remediation has gone  on at
a number of sites for periods well in excess of initial estimates.   (3.6.1)

      Recommendation 29.  The MMRS recommends that the Agency discuss the
implications of unknown presence of NAPLs in the Corrective Action RIA. The
MMRS observes that NAPLs are not always recognized during site
characterization, and that this oversight may result in selection of a remediation
system that  is not appropriate for NAPLs, resulting in excessive remediation times
and associated costs, and  possibly in remediation goals not being achieved. (3.6.2)

      Recommendation 30.  The Subcommittee recommends that the Agency
evaluate the sensitivity of the RIA analysis to assumptions about remediation
effectiveness. The MMRS believes that, for some cases, especially cases in which
NAPLs are present or those sites located in fine-grained soils and fractured or
karst rock, the assumed extent of remediation effectiveness may be too high.
(3.6.3)

      Recommendation 31.  The MMRS recommends that a closer review be made
of the derivation and scientific basis of the soil-water partition coefficient (K
value) used in the post-processing of model results to calculate the change in
concentrations at the exposure location.  (3.6.3)

      Recommendation 32.  The MMRS recommends that the suite of remediation
technologies used in the analysis be expanded to include biologically-based
treatment technologies. The Subcommittee observes that a significant advantage

                                      8

-------
 of these treatment technologies is that, where applicable, they may provide a more
 cost-effective treatment approach than other currently available remedial
 technologies.  (3.6.4)

      Recommendation 33.  The MMRS recommends that the risk analysis be
 modified to recognize risks that may be incurred through the remediation process.
 (3.6.5)

 1.3.7 Issues Relating to Assessment of Uncertainty

     ""Recommendation 34.  The MMRS recommends that guidance be provided in
 the MMSOILS user m*n"«l concerning why and how the user should obtain
 qualitative or quantitative estimates of the uncertainties associated with each
 pathway.  (3.7.1)

      Recommendation 35. The MMRS  recommends that any numerical results
 emanating from the RIA analysis be presented as a range.  The MMRS stresses
 that presenting results as "a number" tends to give the reader a false sense of
 accuracy which, in this instance, is particularly dangerous given the
 incompleteness of the input data set and our incomplete comprehension of the fate
 of hazardous constituents in the environment.  (3.7.1)

      Recommendation 36. The MMRS recommends that the MMSOILS model
 and results be subjected to more thorough, formal and comprehensive sensitivity
 and uncertainly analyses in order to identify the critical parameters associated
 with predictions of contaminant concentrations  along various pathways.  This
 information can then be used to determine what the  critical data are  for
 improving model predictions, and possibly to simplify the model structure without
 sacrificing accuracy or precision of model results.  (3.7.1)

      Recommendation 37.  The MMRS recommends that the Agency review its
 risk estimation protocol. The MMRS is concerned that the simple protocol
 followed to obtain high-end risk estimates may be inadequate, in that these
estimates in some cases apparently give rise to lower exposures than those
generated using the central tendency estimate.  (3.7.2)

 1.3.8  Comments on Results for Health Risk Analysis

      Observation 38.  The MMRS observes that  since health risks are the
predominant focus of current environmental protection initiatives, the adequacy of
risk estimates has to serve as the ultimate criterion of model relevance and
accuracy.  (3.8.1)

      Recommendation 39. The MMRS recommends  that the Agency consider how
the general validity of its exposure estimates might be tested by comparison with
 empirical field data.  This  is being recommended as a result of the observation by
 the MMRS that the translation of contaminant concentrations to estimates of

-------
 exposure necessarily involves a long chain of assumptions and requires the
 adoption of parameter values of variable uncertainly.    (3.8.2)

       Recommendation 40.  The MMRS recommends that the Agency revise its
 practices for assessing cancer and noncancer health risks so as to make them more
-consistent with one another. (3.8.3)

       Recommendation 41.  The MMRS recommends that the Agency review its
 discussion of critical health effects and correct any inaccurate information.  (3.8.4)

       Recommendation 42.  The MMRS recommends that the Agency review its
assumption of additivity of Hazard Indices, and additivity of risks across class A
(known) and Class C (suspected) carcinogens.  (3.8.5)

       Observation 43. The MMRS observes that certain hazardous agents are not
easily  controlled and may pose health risks beyond the substances discussed in the
RIA.  (3.8.6)

1.3.9  Comments on Use of MMSOILS in Corrective Action RIA

       Recommendation 44.  The MMRS recommends that the word "random" be
deleted from any reference to the sample; the fact that various facilities were
eliminated from the analysis for various reasons, some of which are quite valid,
belies the concept of the sample being "random."

      Observation 45.  The MMRS agrees that MMSOILS  may be appropriate to
use as a screening-level model at the national level, but observes that the model is
actually used beyond screening in estimating the fate and transport of
contaminants.  The MMRS observes that the acceptability of uncertainties
associated with model  predictions must be evaluated in the context of model use.
The model use determines the objective of any validation effort.  If the model is
used in a screening mode, then greater  uncertainties on the model outputs can be
tolerated in making a coherent decision and validation efforts should focus on how
well the model screens. If the model is to be  used in estimating spatial-temporal
values of contaminant  concentrations, for example, to feed  into a site-specific risk
assessment, then validation requires comparisons with these kinds of data which
are highly likely to need greater accuracy and precision, if the model is to
effectively contribute to these estimations.  Care must be taken in not confusing
the two different uses  of the model and that such a distinction be  made to the
model  users. (3.9.2)

      Observation 46.  The MMRS observes that, as an alternative to attempting
to estimate a national  average by aggregating the 38 site-specific applications of
MMSOILS, it might be just as valid to use as  much data as possible from the
5,800 sites to construct an "average" national waste site and apply the model to
this single hypothetical site.  This approach might be particularly effective given
that the validity of each site-specific simulation is not held to be very accurate.

                                       10

-------
 Analyzing the hypothetical site with the model might be more in-line conceptually
 with the notion of screening.  (3.9.2)

       Observation 47.  The MMRS commends the Agency for having drafted such
 a well-organized and well-written report for such a highly complex issue as that
 for the Corrective Action RIA.  However, the MMRS observes that the major goal
 of the RIA is to provide a quantitative estimate of the cost with incremental
 benefit for corrective actions, and it is commonly recognized - and accepted as a
 necessary reality - that the MMSOILS model application could derive exposure
 estimates no better than "order(s) of magnitude". Although the results may still
 be valuable for the purposes of screening, e.g., for assessing relative clean-up costs
 or cost versus incremental benefits between various sites, their utility is brought
 into question when the results are intended to be used for evaluating remediation
 costs, i.e., how meaningful it is when a cost estimate is given with a built-in
 uncertainty of one or more orders of magnitude, considering the total cost at the
 national level would probably involve hundreds of billions of undiscounted dollars?
 (3.9.3)
      Recommendation 48.  The MMRS recommends that the Agency give
priority to highlighting the uncertainties in the MMSOILS model screening effort
and the propagation and perhaps magnification of that uncertainty in the
subsequent estimates of exposure, risk, costs, and benefits, because of the critical
importance of this aspect of the RIA. The MMRS observes that a major deficiency
with the draft RIA relates to an inadequate representation of the magnitude of the
uncertainties associated with the cost and benefit estimates.  The MMRS further
recognizes that communication of the relevance and implications of uncertainty
analysis to decision-makers is a difficult and challenging problem.  (3.9.4)

1.3.10 Other User Groups for MMSOILS

      Recommendation 49.  Because of the potential utility of MMSOILS for
estimating ecological risks in relation to other EPA programmatic efforts, the
MMRS recommends that this modeling construct continue to receive attention,
both in terms of review and in resources, to ensure that it has utility beyond
RCRA.  (3.10.1)

      Recommendation 50.  On a longer-term perspective, the MMRS recommends
that the Agency consider what might be its role in providing guidance to states as
to the appropriate types of models to use for state-level screening calculations.
(3.10.2)

      Recommendation 51.  The MMRS observes that the model documentation
makes clear that MMSOILS is meant to be used by non-specialists. Consequently,
the MMRS recommends that the manual be revised to contain stronger statements
that emphasize the model limitations to such users, to recommend alternative
models, and to emphasize the inapplicability of the model to site-specific
evaluations.  (3.10.3)

                                      11

-------
                             2. INTRODUCTION

2.1  Charge for SAB Review

      In accordance with the charge to the MMSOILS Model Review
Subcommittee (MMRS), [See Appendix A, Reference 1 (memorandum dated March
26, 1993 from Richard Guimond to Donald Barnes regarding Charge for SAB
review of Regulatory Impact Analysis Supporting the Corrective Action
Regulation), as well as Appendix A, Reference 2 (a jointly-signed memorandum
dated June 26, 1992 from Richard Guimond and Peter Preuss requesting a SAB
review of the RCRA Corrective Action RIA), and Appendix A, Reference 3 (Federal
figflgter, Vol. 58 (April 9, 1993), pg. 18395, which states the charge to the SAB)]
the MMRS focussed on the technical aspects of the MMSOILS model. The specific
issues that the Subcommittee was asked to address include:

Issue 1.     The adequacy of methods for using a screening level model to
            characterize situations where there is a substantial subsurface
            heterogeneity or where non-aqueous phase contaminants are present.

Issue 2.     Appropriateness of the Agency's approach for aggregating releases
            from solid waste management units (SWMUs, the source terms for
            the contaminant modeling) to estimate concentration at exposure
            points over time.

Issue 3.     Adequacy of the Agency's approach for developing long term
            effectiveness and failure scenarios for site remedies, and

Issue 4.     The implications of the fate and transport modeling assumptions on
            the ecological and human risk assessment.

2.2 SAB Review Procedure

      The primary review document is the EPA/ORD report, MMSOILS:
Multimedia Contaminant Fate, Transport, and Exposure  Model - Documentation
and User's Manual (September 1992 draft; See Appendix A, Reference 4, as well as
Appendix B, Reference 6).  The Subcommittee also relied heavily upon the
EPA/OSW Draft Regulatory Impact Analysis for the Final Rulemaking on
Corrective Action  for Solid Waste Management Units: Proposed Methodology for
Analysis (including its Appendices) (March 1993 draft; See Appendix A, References
5 and 6, as well as Appendix B, References 7 and  8).  These documents  were
addressed at a meeting of the MMSOILS Review Subcommittee (MMRS) of the
Environmental Engineering Committee (EEC) in Arlington, VA on April 22-23,
1993, at which time the MMRS was also briefed by Agency staff on the  selection,
development and application of the MMSOILS models in the RIA (See Appendix A,
References 7 through 12).
                                     12

-------
      A second meeting was held on June 29, 1993 with some of the MMRS and
Environmental Engineering Committee (EEC) members and consultants who
wished to focus of the MMSOILS model and data [See Appendix A, Reference 13
(Federal Register. Vol. 58, No. 108, June 8, 1993, pg. 32122). Also, see Appendix
A, References  13 through 19, for a listing of the briefing materials for the June
29, 1993 meeting].  The specific purpose of this meeting was to further discuss
with the Agency staff the selection of the corrective action sample facilities, data
sets, data  acquisition and facility conceptualization process, as well as progress by
the OSW staff on verification and validation of the MMSOILS model.
                                     13

-------
          3. COMMENTS ON MODEL SELECTION, FORMULATION,
                   DOCUMENTATION AND APPLICATION

3.1  MMSOILS Selection, Development, Formulation and Documentation

      The Subcommittee would like to note that, while a number of
recommendations are made in this report to improve the MMSOILS model and
overall approach, such an approach lends itself well toward the context of better
understanding and dealing with reducing risk concepts (See Appendix B,
References 9, 10, 14 through 17 and 19). This approach also is a systematic way
of characterizing and assessing ground water oriented and multi-media oriented
approaches for grappling with the admittedly very complex, difficult, demanding
and challenging risk assessment concepts that are being applied in a national
context to deal with the Corrective Action RIA (See, for instance, Appendix B,
References 1 through 5, 10 through 13 and  19 through 22).

   3.1.1 Model Selection and Development

      The OSW/ORD working group is to be commended for a well-coordinated
and focused effort to develop a regulatory impact analysis (RIA) that will help the
Agency and the Nation better understand the  costs and benefits of the proposed
rule for the final corrective action for solid waste management units.  The
Subcommittee considers  the intraagency coordination represented  by this RIA to
be a "model approach" that the Agency would do well to adopt in  other programs.

      The consensus of  the MMSOILS Model Review Subcommittee (MMRS) is
that the use of a multimedia pathway  model for screening purposes  could be an
appropriate approach for developing risk and cost estimates for a  national-level
RIA, as long as the  input parameters are sufficiently accurate and the  model is not
applied outside its range of validity. The Agency's use of a single model, to the
extent defensible ensures consistency among model results.  The rationale for the
selection of MMSOILS for the corrective action RIA was explained during a
briefing to the MMRS by OSW. The MMRS recommends that the criteria and
rationale for the selection  of MMSOILS as expressed in that briefing be more fully
documented in the RIA so that the scientific and strategic bases for the selection
will be clear to all concerned - regulator, regulated,  and scientific/risk
assessment/economic communities.

      The major overriding concerns of the  MMRS  are: a) the application of
MMSOILS outside its range  of validity; b) large uncertainties in input parameters;
c) consequent large  uncertainties in MMSOILS results; d) the lack of clear
communication of this uncertainty to decision-makers; and e) the  generation of
credible guidance on exposure, risk, costs, and benefits.  Consequently, the
recommendations contained in this report are focused at efforts to decrease tbe
level of uncertainty, to validate the MMSOILS results by comparison with
alternative  estimation methods, and to ensure that the results of the modeling
exercise are expressed in the  RIA in a  scientifically  defensible manner  that

                                     14

-------
 communicates the uncertainties of the calculations and their implications for the
 cost/benefit analysis.

 3.1.2  Use of Standard Formulations

       MMSOILS  was selected by EPA for use as a screening model for simulation
 of contaminant transport from waste management units through multiple
 environmental pathways, and for evaluation of potential exposures and associated
 risks.  The model uses simple, conservative, and computationally efficient
 equations for estimating chemical transport via ground water, surface water, soil
 erosionj the atmosphere, and foodchains.  Mathematical formulae used to estimate
 transport rates for each pathway are widely used and accepted by the scientific
 community for application to simple situations. Underlying assumptions for each
 pathway model have been identified, are clearly stated, are reasonable and are not
 overly restrictive.  For the mc^L part, the representations used in MMSOILS for
 the various exposure pathways are consistent in their level of treatment.
 However, for a significant number of sites, the MMRS suspects that no generic
 model is likely to  provide answers of acceptable quality. OSWER is generally
 aware of the limited usefulness of generic models  for the analysis of complex
 environmental systems, including aquifers.

 3.1.3  Documentation of Modeled Pathways

      Documentation of the formulations for individual pathway models is well
 organized with appropriate references. However, the manual would benefit from
 another round of editing;  a distracting aspect of the review draft is that several
 terms in the equations and figures have not been  defined in the text (e.g., Cwl,
 Cdwl in equation 3-11, page 3-14 of the Users Manual). The MMRS recommends
 that each term be defined in three places in the manual: (a) in the  beginning or
 end of each chapter in which it appears,  (b) at the time each is first used in a
 given chapter, and (c) in the compilation of terms in Chapter 12. The terms
 should also be reviewed for internal consistency.  In at least one  case, two
 different symbols have been used for the same parameter (Q^ qm).  In another
 case, the same symbol (DF) has been used for two different parameters.

3.1.4  Documentation of Assumptions Underlying Multimedia Treatment

      Basic underlying assumptions, such as the assumption in the ground water
transport model of an "idealized homogeneous, uniform porous media," are
dispersed throughout the manual.  Documentation for MMSOILS would benefit
if om a concise presentation of the model's basis, assumptions and limitations in a
central location at the beginning of the user's manual.  The presentation should
include descriptions of key aspects of the overall model structure, such as:
modeling time frames for each pathway, whether the model assumes finite or
infinite sources, whether the model is steady-state or dynamic, and how the model
deals with competing mechanisms or pathways.
                                      15

-------
       This introductory section should be sufficiently comprehensive that a
 knowledgeable reader could use it to quickly assess the level of sophistication of
 each pathway component in the MMSOILS model and thus develop a level of
 comfort with the predictions provided by the model.  Chapter 2.0 of the user
 manual, "Applications and Limitations of the Methodology," does not  fulfill this
 requirement, although it could be revised to do so. The cursory discussion
 presented in the section of questions that should be asked may be interesting but
 does not inform the user of the assumptions that were made in the model
 construction.  Assumptions and their resulting model simplifications are crucial to
 the evaluation of the model.  The brief section on model limitations also fails to
 discuss what the true limitations are.
      Possible Improvements to Model Formulations for Specific Processes

3.2.1 Additional Types of Solid Waste Management Units (SWMUs)

      As currently constructed, MMSOILS is only capable of handling the more
traditional SWMUS such as landfills and surface impoundments.  Other,
potentially more costly, types of SWMUs such as leaky sewer systems have been
excluded from the analysis.  Although they have been identified as SWMUs, few of
these less traditional SMWUs have been remediated largely because no one really
knows how to deal with them.  Their remediation  could be quite  costly and result
in disruption of industrial operations.  Eventually these problematic SWMUs will
be impacted by the proposed corrective action rule and will have  to be addressed.
Thus, an estimate of their cost contribution to the implementation of the proposed
rule should be developed.

3.2.2 Recognition of Natural Biodegradation Processes in Ground Water Pathway

      Approximately 130 years of simulations are made using the MMSOILS
model to predict the existence,  development or dissipation of ground water plumes.
Because of the long time periods involved, it is critical that the role of natural
biodegradation processes be explicitly incorporated into the ground water fate and
transport pathway by the  use of an appropriate biodegradation  coefficient  value.
This function is essential and provides realism for actual mechanisms taking place.
Even a small biodegradation coefficient would have a big impact.  For instance, a
value of only 0.0001/yr for the  degradation coefficient of organic constituents in
ground water will  have a very large effect on the distribution of contaminants
when taken over simulation periods much less than 130 years.

      Neglecting the role  of biodegradation processes in the transport model could
result in overestimation of exposure concentrations.  This omission might  not be
critical in screening applications where bias towards overestimation may provide
appropriately conservative results. However, the literature continues to increase in
terms of estimates of biodegradation of many organic contaminants and these data
should be examined for possible use in MMSOILS.  (See also comment 3.6.4 on
biologically-based remediation technologies.)

                                      16

-------
 3.2.3  Modeling Transport through the Vadose Zone

       The finite element numerical model used for the partially saturated zone is
 adequate for the job, but it may not necessarily be consistent with the precision of
 other model components. This is especially true in light of other assumptions
 made in the RIA. For example, only a single layer (or small number of layers
 with only roughly estimated properties) was used to represent the unsaturated
 zone and the net infiltration rate was taken as piece-wise constant.  Given these
 assumptions, simple kinematic models are appropriate and consistent with the rest
 of the transport process models.  Kinematic models for flow and transport will
 conserve mass, can address step-wise constant infiltration rates, and will provide
 simple algebraic equations for use in the model.

 32.4 Plume Aggregation in Groundwater Pathway

      The method of aggregation used to obtain the concentration distributions for
 application to  individual wells does not conserve mass.  Rather, it appears that the
 resulting  apparent mass will always exceed that from the untransformed plumes.
 While this approach is conservative, the degree of conservation should be  evaluated
 through comparison with a  number of simulations which do not use the method of
 aggregation.  In addition, the question of whether movement of the plumes could
 result in a decreased concentration for population wells should be  evaluated.  The
 required transformations from cartesian to cylindrical coordinates  should not
 require much computational effort compared with that required to operate the
 model.

 32.5 Food-Chain Module

      The food-chain module in MMSOILS is  very synthetic and unrealistic.  Food
 chains are highly site-specific and depend upon the gathering of the contaminant
 into the receptor environment, the structure of the ecological community,  and the
 ultimate receptor of interest (humans or eagles or others).  It makes considerable
 difference in the risk estimate whether the ecological community is terrestrial or
 aquatic and to what extent the contaminated food contributes to the total food in
 each trophic level.  Furthermore, the efficiency of contaminant transfer from  one
 trophic level to the next varies, and is dependent in part upon the octanol-water
 partition coefficient (KQW) and molecular size of specific organic compounds.

      The ecological risk assessment component should be  constructed using
recommendations structured in the SAB's report of the Ecology and Welfare
 Subcommittee of the Relative Risk reduction Strategies Committee of the SAB [See
Appendix B, Reference 10 in particular, as well as Appendix B, References 9,  11
 and 12, and the principles for ecological risk assessment as suggested by the Risk
Assessment Forum, Appendix B, Reference 23.]
                                      17

-------
 3.2.6 Mass Balance

      The effectiveness of the model for estimating long-term consequences of
 remediation alternatives appears to be weakened by the breakdown of mass
 balance as a result of post-processing of model outputs (See also Section 3.2.4
 regarding issues related to the method of aggregation and conservation of mass).
 The exponential decay of ground water contaminants as described represents one
 particular example of this apparent breakdown.  Similar concerns for overall mass
 balance in MMSOILS remain. For example, if the combined degradation or
 transport along all pathways accounts for more contaminant mass than is
 available, the flux along each pathway is merely normalized according to the
 "demand."  This procedure  may provide results of questionable accuracy given the
 different time scales applied to contaminant flux along different pathways.

 3.2.7 Disparity in Relevant Time and Space Scales for Transport Mechanisms

      One major problem that must be confronted in the development of any
 multimedia model, such as  MMSOILS, is the forcing of differently scaled
 environmental transport processes  into a single model construct. For example, the
 temporal dynamics of volatilization of organics may vary on the order of hours or
 less, given  changes in microclimatic conditions that drive this process (e.g.,
temperature, wind velocity). Surface water runoff leading to transport of soils and
dissolved contaminants is strongly  event-driven, that is, local precipitation patterns
and strong storms can move significant  amounts of chemicals in time scales of
hours. These dynamics contrast markedly with the slow movement of
 contaminants in ground water, where years to decades may be the relevant time
scale.

      Attempts to force the above  disparate scales into a single model by selecting
a compromise in tune step  will necessarily result in a loss of accuracy in model
predictions. Parallel considerations apply in the spatial domain. Representing the
spatial distribution of ground water plumes of contaminants may allow a
comparatively coarse spatial description of the waste site and surrounding region.
To achieve comparable accuracy  in estimating atmospheric transport, a more
 detailed spatial representation of the system may be necessary.  Again, attempting
to force disparate spatial scales into a single model can produce inaccuracies in
 model results.  Finally, the combination of time and space scale selections required
 in establishing a single model  can compound problems outlined above.

 3.2.8 Relevant Time  Scales for Ecological Risk Assessment

      At present the  ecologically relevant exposure scenarios are inadequate.  The
 known major source for acute impacts from waste sites on aquatic environments is
 surface run-off after major  rain events.  The MMSOILS model cannot simulate
 this  in its present version.  On a long-term basis, major effects can be due to
 biomagnification, changes in biodiversity, etc.
                                      18

-------
       Methods for estimating ecological risk continue to evolve, e.g., the EPA
 Framework for Ecological Risk Analysis.  Nonetheless, it is quite apparent that
 the methods will surpass simple multiplicative models of food chain accumulation
 and comparison of exposure concentrations to acute or chronic toxicity
 benchmarks.  The annual time scale for exposure estimates produced by MMSOILS
 seems inappropriate for many ecological applications.  Typical organisms of
 concern exhibit short life spans, or critical stages in their complex life history that
 occur at time scales substantially shorter than one year.  Thus, an average annual
 concentration is not particularly useful as an input to many ecological risk
 assessments.  Strong seasonal constraints are typically important, especially for
 temperate surface waters, e.g., streams and lakes.  Thus, MMSOILS may have to
 be  modified to produce more realistically-scaled exposures for meaningful inputs to
 ecological risk analysis.

 3.3 Issues of Parameter Estimation

      The MMRS notes that the use of inaccurate input parameters is a suspected
 major source of unacceptable errors and unreasonable  magnitudes of uncertainties
 in MMSOILS results relative to their use in  the draft  Corrective Action  RIA.
 Inappropriate input data are a consequence of sparse or inaccurate information,
 poor judgement in parameter estimation,  and suspected overreliance upon default
 parameters.  Regardless of the high quality of the model formulation,  the quality
 of the outputs will  retain the deficiencies of the inputs, and may well amplify
 them further.  Agency personnel are clearly aware of problems in this area.  Below
 are highlighted some of the parameters of greatest concern.

 3.3.1  Source Term Parameters

      Most members of the MMRS believe that the largest  single source of
 uncertainty in the risk  analysis  was probably that related to quantification of the
 source term.   Problems include sparse or inaccurate information on identification
 of types of wastes present (e.g., presence  of NAPLs), on quantification of waste
 quantities, and on estimation of waste distribution. Given the time and  budget
 constraints under which it is operating, it is highly questionable whether the
Agency could significantly improve upon the  extensive  and thorough job  it has
 already done in compiling the source-term data.  Nonetheless, at a minimum, the
Agency should ensure that the uncertainty estimates in the RIA fairly reflect the
uncertainties in this aspect of the model input.

      The MMRS believes that  consideration of the quantity and quality of waste
information should be a criterion for the  inclusion of a particular facility in the
facility selection process.  As explained by the Agency in its presentations, the 79
sites were selected randomly from approximately 5,800 sites nation-wide; for many
 of these sites, information on the wastes  is often sketchy or non-existent. Because
a computer model cannot provide results  that are any more precise or accurate
than the input data used, there may be no issue more  important than ensuring
that the model input has the most accurate information possible on waste

                                      19

-------
 characteristics and history. An appropriate criterion for facility selection might be
 a requirement for "waste information with both reasonable quality and quantity."
 Evaluation of the waste information against such a criterion should not take much
 additional effort or affect the purpose of the statistical selection process,
 particularly when less than 2% of the sample sites are chosen from 5,800 sites.
 The expected improvement of the confidence in the results from this study is
 obvious.

      Another problem that may increase the uncertainty of the waste transport
 calculations is that the existing data that have been developed for SWMUs were
 generally not collected for the purpose of estimating risks  to humans or to
 ecosystems.  The data were collected to define the extent of contamination  at a
 site, rather than defining the  exposures at or near the site.  The data  are often
 deficient in providing information on environmental properties that influence the
 dynamics of releases to air, si^ace water, or ground water.  In  addition, the
 structure (dimensional distribution of materials, physical characteristics) and
 processes occurring at a site are only partially understood.  These short-comings
 significantly affect the utility of input data.

 3.3.2 Waste  Release and Solubility

      Predictions of ground water contamination and future growth in the  plume
 are directly proportional to the mass of leachate assumed to be released  to  the
 subsurface from a SWMU. The  release models used for metals and organics,
 while being good starting points, could be improved.

      With regard to metals release, the model's use of the maximum observed
 concentrations  near known source terms might over- or underestimate metal
solubility depending on the environmental context of the waste site, e.g., pH,
 redox, aerobic/anaerobic, soil type, organic content of soils  and ground water, etc.
 Perhaps some of the chemical speciation models e.g., MINEQL, might be examined
 to see if they can provide more meaningful estimates of the solubility, complexed,
 adsorbed, etc. fractions of metals for use in MMSOILS calculations.

      With regard to organics release, it appears that using a multiplier of 100 to
 estimate organic leachate concentration is arbitrary at best.  The choice of the
 100 value incidently corresponds  to the generic Dilution Attenuation Factor (DAF)
 of 100 used in  the Toxicity Characteristic (TC) rule promulgated by the Agency in
 1991.  The ideal or the most reliable method for choosing  leachate concentrations
 might involve choosing the most  important solid phases for metals and perhaps
 Aqueous solubilities based on Raoult's Law for most of the organic compounds.  It
 is also recommended that only a  fraction of the total mass of a chemical in the
 SWMUs be allowed/available for leaching instead of the entire 100%.  [NOTE:  It is
 recognized that there should bt ~*gnificant attention paid to what the fraction of
 total mass that would be allowed to leach might be.]
                                      20

-------
 3.3.3  Use of Default Values

       The MMRS suspects that the modelers relied more heavily than warranted
 on the use of default values, although this aspect is difficult to judge from the
 information provided by the Agency.  Clearly, the OSW/ORD working group needs
 to address the issue of use of default values, making it very clear when they are
 used and why the default value makes sense for the particular application, in the
 absence of better data.

 3.3.4  Peer Review of Data Base
     •«.
       The MMRS recommends a documented and thorough peer review of all
 aspects of the data base, focusing particularly on those parameters to which  the
 results are most sensitive.  Such a peer review does not need to involve the SAB,
 but occasional consultation with the SAB or the interagency modeling task force,
 ATFERM, on the approach and issues to be grappled with might be a useful
 exercise.

 3.3.5 Data Base for Future Related Modeling Efforts

      The MMRS recommends that the Agency build upon the extensive data base
 it has accumulated for the  Corrective Action RIA, to begin the development of an
 extensive  data base that could be tapped for other EPA programmatic efforts, such
 as for a comparable assessment of the risks associated with NORM wastes and
 radiologically contaminated sites.  The intraagency modeling task force, AFTERM,
 may be an appropriate vehicle for organizing and coordinating such an effort in a
 manner that  would be most beneficial to the potential users.

 3.4  Issues of Range of Model Validity

      MMSOILS  was not designed to estimate contaminant transport and  fate for
 chemicals in sites with complex hydrology, nor to assess the environmental
 behavior of non-aqueous phase contaminants in ground water.  However, these
 limitations should not necessarily be considered as weaknesses of the model.
 Complex  sites and Non-Aqueous Phase Liquids (NAPLs) need to be addressed for
 the RIA,  but  remain outside the domain of applicability of the  current MMSOILS
 construct. The solution lies either in developing separate models to examine  these
 issues or modifying the current MMSOILS so as to extend its applicability. Given
the current state-of-the-art in our understanding and ability to model either
complex hydrogeology or NAPLs,  it may be some time before these aspects can be
realistically introduced into MMSOILS.  For example, stochastic modeling of
ground water may contribute toward addressing the complex hydrogeology issue.
The literature on this subject continues to grow; however, the complexity of these
kinds of models may preclude their easy incorporation into a scheme such as
MMSOILS. This  is not to undermine the importance of these issues, but merely
to emphasize that we are at the cutting edge of science in the development and
 applications of MMSOILS (albeit that MMSOILS deals with simple, conservative

                                      21

-------
 and computationally efficient equations for estimating chemical transport) and
 progress may be slow.  Some additional questions about model validity and some
 suggestions for how the Agency might deal with this issue are provided below.

       The Subcommittee offers the following comment on use of stochastic models
 and Monte Carlo models.  Stochastic models are usually for simple hydrogeology,
 but with complex parameterization, and it is doubtful that they are or will be
 considered useful for a screening analysis.  Further, these (the stochastic models)
 should be distinguished from Monte Carlo models which generally are simple
 deterministic models with random input data.  The later are appropriate for
 uncertainty analysis with a screening model.]

 3.4.1  Extreme Events

       How well does the model deal with processes that are event-driven?  These  .
 applications may be outside the range of validity of the model. Two examples are
 offered.  First, the water balance approach is not expected to work well for sites in
 arid regions in which average precipitation and evapotranspiration are
 approximately equal.  Use of the balance approach would lead  one to expect no
 recharge.   While some infiltration and recharge in arid regions does occur as a
 result  of extreme rainfall events, extreme rainfall events are typically associated
 with flash flooding.  Recharge is more likely due to confluence of flows along
 arroyos, and can occur with "normal" rainfall events.  Such recharge is localized,
 and probably should not be considered in the screening model calculations, unless
 a facility is placed along the ephemeral stream.]

       The model appears to under-estimate waste transport via surface runoff.
 Net infiltration is calculated from precipitation less runoff and evapotranspiration.
 Runoff is calculated using the curve number method, which is based on daily
 rainfall amounts. Precipitation is provided through monthly rainfall amounts.
 Daily rainfall amounts are calculated by dividing the monthly rainfall by the
average number of days with rainfall greater than 0.01" per month.  The "wet"
days are then distributed uniformly though the month to calculate infiltration and
runoff.  This approach increases the likelihood of infiltration and decreases the
potential for runoff. Runoff tends to occur when storms of greater than normal
precipitation occur, or when storms occur on consecutive days.  The approach may
be conservative because the ground water pathway appears  to dominate the
exposures and risks.  However, it is not clear that this was the intent. Actual
daily rainfall data should be used for specific locales  and regions where available.

      The MMRS recommends that the Agency evaluate the validity of each
pathway model to assess the extent to which extreme events might be expected to
contribute to the bulk of contaminant releases, and the extent to which the model
may under- or over-estimate transport.
                                      22

-------
 3.4.2  Complex Sites

       The Subcommittee observes that applicability of MMSOILS is limited by its
 inability to deal with complex sites.  It is likely that critical investigations of the
 applicability of MMSOILS will turn up other complex  sites beyond karst (e.g., a
 limestone region marked by sinks, abrupt ridges, irregular protuberant rocks,
 caverns and underground streams) and fractured rock. Sites in glacial till, such as
 former gravel pits, may also exhibit more complexity than MMSOILS could
 reasonably address.

       For complex sites, the MMRS recommends that the Agency pursue
 alternative approaches and should develop a strategy that combines MMSOILS
 with an appropriate monitoring strategy.  The outputs from such an effort can
 also be used to modify and improve  the MMSOILS model.

 3.4.3   NAPLe

       The MMRS strongly endorses ORD's recommendation that the Agency
 conduct a separate modeling exercise and obtain expert opinion to develop  an
 improved screening-level modeling of NAPLs.  For dense non-aqueous phase
 liquids (DNAPLs), a possible modification might be to  locate the source term in
 the saturated zone.  For light non-aqueous phase liquids (LNAPLs), a possible
 modification might be to use an alternative  volatilization model. Four
 volatilization models are available, representing two conceptualizations of the
 process.  One approach is based on Fick's first and second law, and gives classical
 solutions to the diffusion equation. The second approach combines Fick's first  law
 with a moving  boundary model for continuity. This is the "Landfarming Equation"
 of Thibodeaux  and Hwang.  A comparison between these approaches shows that
 the moving boundary model predicts fluxes that are smaller than the classical
 diffusion model by a factor of about 0.8 (square root of 2 divided by pi).

      Given the level of accuracy expected from a screening model, these
 approaches provide essentially the same result.  By choosing the moving boundary
 model, one can represent cases with  a finite region of contamination, both with
 and without a cover, using simple  algebraic  equations.  Use of an effective
 diffusion coefficient in soil  along with appropriate partitioning relationships will
also allow one to account for the presence of air, NAPLs, and water within the
pore space, and partitioning of the constituent within the air, water, soil, and
NAIL system.

3.4.4  Development of Guidelines for Assessing Model Applicability to Specific
Cases

      The MMRS recommends that the Agency develop guidelines in order to
assess  the applicability of MMSOILS to specific cases.
                                      23

-------
3.5 Issues Relating to Pathway Model Calibration, Verification and Validation

3.5.1  Ground Water Model

      The ground water flow module has been verified by comparison of results to
those of numerical models.  However, documented comparison of model predictions
of chemical transport to field data would strengthen the scientific credibility of the
results and provide a basis for readers to evaluate the model validity and
magnitude of uncertainty.

      Another possible means to provide a limited validation of the ground water
pathway component of MMSOILS is to compare its output to  that of other peer-
reviewed EPA models.  The EPA Office of the Solid Waste (OSW) has developed
and used EPA Composite Model for Landfills (NATD/EPA Composite Model for
Surface Impoundments (EPACMS) model for ground water transport  and fate of
contamination for the purpose of regulating RCRA wastes on the national level.
MMSOILS is similar in many respects to the NATL/EPACMS models. Therefore,
the MMRS recommends that, for a  subset of SWMUs where ground water plume
predictions are made by using MMSOILS, NATL/EPACMS models also be
exercised so as to permit comparison of plume predictions.

3.5.2  Other Pathway Models

      The MMRS strongly recommends that documented validation exercises be
undertaken for the remaining environmental transport pathways, e.g.,
aerosolization, volatilization, surface water runoff, and bioaccumulation, in order to
assess the ability of these pathway  models to provide meaningful input to the RIA.

3.5.3  Guidelines for Calibration, Verification and Validation

      The MMRS recommends that the Agency, perhaps through AFTERM,
develop generic guidelines for model calibration, verification and validation,
including criteria for judging whether or not discrepancies among alternative
modeling results or between calculated and measured field data are significant. In
the case of MMSOILS, it recommends that the Agency undertake a root-cause
analysis for discrepancies, where significant, in order to evaluate the potential for
systematic bias in the modeling approach.

3.6 Comments on Remediation Effectiveness

3.6.1  Remediation Tunes

      As part of the RIA process, experts have estimated the  time  for clean-up of
contaminated sites.  Experience cited in EPA's "19 Sites" (a 1989 publication; See
Appendix  B, References 24 and 25) and "24 Sites" (a  1992 publication; See
Appendix  B, References 26 and 27) documents has shown that these time
estimates  may be  overly optimistic  for ground water extraction systems.  Even less

                                     24

-------
is known about the operation and efficiency of other remedial systems.  Because of
unidentified sources, vadose zone contamination, heterogeneities, and the unknown
presence of NAPLs, remediation has gone on at a number of sites for periods well
in excess of initial estimates.  In addition, for a number of these sites, the
remediation goal has changed from site clean-up to plume containment. The
impact of underestimating the clean-up time is that the clean-up costs will be
underestimated, and the benefits associated with clean-up will be overestimated.
The sensitivity of the RIA conclusions  to these estimated remediation clean-up
times should be evaluated.

3.6.2 Effect of Unknown  Presence of DNAPLfi on Remediation Times

      MMSOILS and the  RIA do not adequately address the presence of NAPLs.
It is well recognized that NAPLs are not always recognized during site
characterization.  This may result  in selection of a remediation system that is not
appropriate for NAPLs, resulting in excessive remediation times and associated
costs, and possibly in remediation  goals not being achieved.  NAPLs strongly
influence the source term. Effective solubility and partition coefficients are far
different when NAPLs are present, compared to their absence. The degree and
timing of contamination events are different when NAPLs are  mobile, as compared
to cases with only dissolved phase  contamination.  These issues should be
addressed in the Corrective Action RIA.

3.6.3 Remediation Effectiveness

      According to the RIA documentation, most of the risk is associated with
the ground water pathway. To calculate the benefits of corrective action, the
MMSOILS model  is run for each site to develop the extent and concentration of
the plume over the time period of  interest.  The plumes are then aggregated to
obtain the baseline or pre-remediation conditions.  For  sites requiring clean-up,  a
remedial action plan is identified, and the effectiveness of each remedial scheme is
assumed.  The extent of remediation is applied to each plume,  and the plumes are
aggregated to provide the  post-remediation concentration distributions.  Exposures
and  risks are then calculated for the two conditions to  evaluate the benefits of
corrective action.

      At least two questions  arise with regard to this procedure.  The first
concerns the assumed level of remediation achieved with each applied technology.
For some cases, especially NAPLs, the  assumed extent of remediation may be too
high. In fine-grained soils and fractured or karst rock, it is doubtful that complete
remediation of LNAPLs is achievable.  For DNAPLs the assumed levels of
containment and remediation are also in question.  The sensitivity of the RIA
analysis to these assumed levels of remediation should be evaluated.

      The remediation effectiveness predictions are, at times, made by using the
MMSOILS model directly. But at  other times the Agency has  used post-processing
through the use of a decay or a percent removed value. For simplicity reasons,

                                      25

-------
 several assumptions regarding failure of caps, liners and barriers etc., have been
 used.  A simplified decay rate coefficient (K) has been employed to calculate the
 change in concentrations at the exposure location.  In the case of ground water
 pump and removal remedy, it is very unclear as to how the K value is derived or
 how realistic this approach is in representing the appropriator of the response.  It
 is recommended, therefore, that a closer review of the K value for choosing the
 quantitative number for sites be made.

 3.6.4  Inclusion of Biologically-Based Remediation Technologies

      ••The suite of remediation technologies used in the analysis should be
 expanded to reflect the realities of emerging technologies. The principal defect is
 the parsimonious use of biologically-based treatment technologies.  Although these
 may not at this time be considered as proven technologies, the scientific principles
 upon which the technologies are based are sound.  It is therefore entirely
 reasonable to assume that over the 128-year tune frame spanned by the analysis
 that these technologies will become widely used.  The  magnitude of uncertainty
 associated with these technologies is certainly no greater than that associated with
 the data base of source terms and transport parameters  upon which the EPA has
 based its analysis.

      A significant advantage of the biologically-based treatment technologies is
that they will likely provide a more cost-effective treatment approach than other
currently  available remedial technologies.

3.6.5  Risks of Remediation

      The MMRS recommends that the  risk analysis be  modified  to recognize
risks that may be incurred through the remediation process.   It is conceivable that
the very act of remediation could incur a higher level of risk than what would be
reduced through remediation. This potential trap should be avoided by estimating
the risks of remediation and including them in the analysis.

3.7 Issues Relating to Assessment of Uncertainty

3.7.1  Uncertainty Estimation Protocol

      No guidance is provided to the user as to how one can obtain a qualitative
or quantitative estimate of the uncertainties associated with each pathway. Given
the high stakes involved in terms of potential commitment of national resources,
such an estimate is as important as, or possibly even more important than, the
final result.

      The MMRS recommends that any numerical results emanating from the RIA
analysis must be presented as a range. Presenting results as "a number" tends to
give the reader a false sense of accuracy which in this instance is particularly
dangerous given the incomplete level of the input data set and our incomplete

                                       26

-------
 comprehension of the fate of hazardous constituents in the environment.  Adding
 on the -uncertainties associated with risk analysis even further expands the error
 bands.

       The model should be subject to formal, comprehensive sensitivity and
 uncertainty analysis.  Some initial efforts have been completed by the Agency,
 emphasizing the ground water pathway.  Similar analyses should be completed for
 the other pathways in the model, and finally for the entire model.

       Sensitivity analyses can be used to identify the critical parameters
 associated with predictions of contaminant concentrations along various pathways.
 This  information can be used to determine what the critical data are for improving
 model predictions, and to simplify the model structure without sacrificing accuracy
 or precision of model results.  Both applications of sensitivity results are
 important in increasing the capabilities of the model for site-specific use and in
 improving its utility in the RIA process.

 3.7.2  Development of High-End Risk Estimates

      The MMRS is concerned that the simple protocol followed  to obtain high-
 end risk estimates may be inadequate in that this estimate in some cases
 apparently gave rise to lower  risks than did the central tendency estimate. The
 Subcommittee recommends that the Agency review its protocol.

 3.8 Interpretation of Results  for Health Risk Analysis

 3.8.1  Health Risk as the Assessment Endpoint

      The model produces outputs which estimate concentrations in various
 environmental media over time.  These concentrations in various  media are then
 applied to various exposures gathered as an initial phase of risk assessment for
 the protection of humans and ecosystems.  The result of the risk  assessments
 comparing no-action and remedial action scenarios are then used  as major inputs
 into a cost-benefit analysis.  The use  of various assumptions in exposure
 assessments as part of the risk assessment process has been highlighted previously
 as a major source for uncertainties in risk assessments.

      Three documents refer to health risk assessment implications: MMSOILS
 (Documentation and User's Manual, the Corrective Action RIA draft, and
appendices for the latter.  See Appendix B; References 1 through  3). These
 implications are drawn from exposure analyses based on applying the models
described in MMSOILS.  Because health risks are the predominant focus of
current environmental protection initiatives, the adequacy of risk  estimates has to
serve  as  the ultimate criterion of model relevance and accuracy.
                                      27

-------
3.8.2 Empirical Validation of Exposure
      Section 9.0 of the MMSOILS document (See Appendix B; Reference 6)
provides equations for determining exposure due to inhalation, ingestion, and
dermal  contact. (Section E of the RIA document addresses some of these as well.)
The relevant variables come from the list on page 9-4, and includes 21 parameters,
some of which are given default values in tables.  These values, however, are
simply guides to translating air, water, and soil concentrations into  exposure
estimates.  These  presumed concentrations are connected to human  exposures
through a long chain of assumptions.  Because of this lack of direct coupling,
questions arise about how the chain might be shortened.  Are there any empirical
data by which the models might be tested?  Is there any way in which they might
be acquired? Measures as simple as uptake by plants might be useful, for
example. Or, analyses of animal carcasses at particular sites to which the
modeling has been applied.  Is empirical validation, the process by which theories
and models are tested in science, out of the question?

3.8.3  Inconsistent Treatment of Cancer and Noncancer Health Risks

      The  other aspect of health risk assessment treated in these documents is
described in the draft report on Regulatory Impact Analysis (See Appendix B;
Reference 7) in the chapter  on human health benefits (Chapter 7).  Here, the
ingenuity of the modeling effort encounters barriers that the Environmental
Health Committee has noted in some of its reports, the most recent of which is
entitled Superfund Site Health Risk Assessment Guidelines. February,  1993 (See
Appendix B; Reference 20).  One of these barriers is the EPA practice of adopting
radically different approaches to cancer and noncancer risk assessment.  Cancer
risk is given as a probability; systemic endpoints are described by Reference Doses
(RfDs).  One example of the confusion this causes appears on page 7-15. Footnote
21 notes that cancer risk is  calculated by averaging intake over a lifetime; that is,
dose distribution is ignored.  For noncancer risk, intake is averaged  over exposure
duration, with an  averaging time of 9 years.

      These assumptions can be disputed and might even be reversed  in some
instances.  Perhaps more important, the averaging of assumptions could easily
distort risk.  For example, for both categories of risk, exposure during early and
prenatal development might be the  crucial values,  and exposure peaks  the major
source of adverse  consequences.

3.8.4  Inaccurate Identification of Critical Health Effects

      Some of the other material also arouses suspicion about the adequacy of the
modeling effort. The table in Exhibit 7-19a (See Appendix B; Reference 7) lists
some of the agents driving noncancer effect levels.  The critical health effects
noted there are perplexing.  For example, the critical health effect for  chromium
VI is given as central nervous system effects, which is contrary to the  commonly
known effects ascertained by neurotoxicologists; for nickel, it is reduced body and

                                      28

-------
 organ weight rather than hypersensitivity reactions; for toluene, a prototypical
 neurotoxicant, it is given as liver and kidney pathology.  There are several other
 peculiar entries.

 3.8.5  Questionable Treatment of Different Waste Classes

       Another difficult problem is how to handle the kinds of mixtures found at
 RCRA sites.  The properties of the Hazard Index, offered  as a solution, cannot be
 applied automatically. For  example, if 101 agents are identified, each with a
 Hazard Quotient of 0.01, the additivity assumption will yield a Hazard Index
 above 1.0.  Would such a situation really present a bothersome risk?

      Another additivity problem arises with cancer agents.  The MMRS disagrees
 with the treatment noted on pages 7-56  to 7-57 (See Appendix B; Reference 7), in
 which agents with different weight-of-evidence classes are combined by adding
 risks, because the MMRS feels that it is misleading  to add risks across class A
 (known) and class C (suspected) carcinogens.  Despite the warning contained
 within the RIA that such a  procedure may overstate the true cancer risk, the
 presentation  of such combined results is bound to have an impact.

 3.8.6  Other  Sources of Hazardous Wastes

      Finally, going beyond the contents of the current documents,  it would be
useful to point out, not just the intrinsic limitations of this rather ingenious and
extensive modeling effort, but also where it might coincide with the full scope of
 EPA's responsibilities. Certain agents are not easily controlled and may pose
 health risks beyond the substances discussed in these reports.  Methylmercury, for
 example, is partially a product of fossil fuel combustion.  Inorganic mercury is
 discharged when  coal and oil are burned, ascends into the atmosphere, travels  in a
 global mercury cycle, returns to earth in rain, and is transformed to the toxic
 methyl form  by organisms in the bottom sediment of bodies of water.
 Methylmercury travels up the food chain in a continuous  cycle of bioconcentration
 to lodge  in fish that then reach human consumers.  In keeping with the spirit  of
Reducing Risk  (See Appendix B; References 9 through 12, and  particularly
Reference 10), such scenarios should be included in efforts such as the current
reports.

3.9 Comments on Use of MMSOHS  in Corrective Action RIA

3.9.1  Faculty Selection Process

      The facility sample used for the analysis is incorrectly characterized as
being a "stratified, random sample."  The fact that various facilities were
eliminated from the analysis for various  reasons, some of which are quite valid,
belies the concept of the sample being "random." The MMRS suggests that the
word "random" be deleted from any reference to the sample.
                                      29

-------
 3.9.2  Use for National-Level Screening

       The model is emphasized as a screening tool.  However, it is also clear that
 the model is used beyond screening in estimating the fate and transport of
 contaminants. If the model is used in a screening mode, then validation efforts
 should focus on how well the model screens.  If the model is to be used in
 estimating spatial-temporal values of contaminant concentrations, then validation
 requires comparisons with these kinds of data.  Care must be taken in not
 confusing the two different uses of the model. Similarly, the uncertainties
 associated with model predictions must be evaluated in the context of model use.
 If screening is the emphasis, then perhaps greater uncertainties on the model
 outputs can be tolerated in making a coherent decision.  If detailed
 characterizations of contaminant concentrations are needed,  for example, to feed
 into a risk assessment, then it is likely that greater accuracy and precision will be
 required if the model is to effectively contribute to these estimations. The  SAB has
 given modeling-related advice in a number of instances to the Agency, and the
 Subcommittee refers the staff to this (See Appendix B; References 11, 12,  13, 18,
 19, 21, and 22).

       Instead of attempting to estimate a national  average by aggregating the 38
 site-specific applications of MMSOILS, it might be just as valid to use as much
 data from the 5,800 sites to construct an "average" national waste site and apply
the model to this single hypothetical site.  This may be particularly  effective given
that the validity  of each site-specific simulation is not held to be very accurate.
Analyzing the hypothetical site with the model in this fashion might be  more in
 line conceptually with the notion of screening. The Subcommittee wishes to
 remind and caution  the Agency that a screening model can estimate the spatial-
 temporal values of concentrations; however, it is limited in the scenarios it can
 consider. Further, it (the screening model) is not suitable for site-specific
 applications where site  details should be included in the model.

       Subject to  the reservations stated above, the  MMRS agrees that MMSOILS
 may be an appropriate to use as a screening-level model at the national  level.

 3.9.3  Presentation of Results in Corrective Action RIA

       The Agency is to be commended for having drafted such a well-organized
 and well-written  report for  such a highly complex issue as that for the Corrective
Action RIA.

       The MMSOILS model adopted for the study, as clearly stated by the
 Agency, was intended to be used as a screening tool.  The  assumptions made and
 approaches followed in  the development of this model make the use of it
 appropriate for the screening exercise, but the current data base needs significant
 improvement. However, the major goal of the RIA is  to estimate the cost  with
 incremental benefit  for corrective actions.  Therefore a quantitative result  is
 required from the study.  It is commonly recognized - and accepted as a reality -

                                      30

-------
 that the MMSOILS model application could derive exposure estimates no better
 than "order(s) of magnitude". Although the results may still be valuable for the
 purposes of screening, e.g., for assessing relative clean-up costs or cost versus
 incremental benefits between various sites, their utility is brought into question
 when the results are intended to be used for evaluating remediation costs, i.e., how
 meaningful it is when a cost estimate is given with a built-in uncertainty of one or
 more orders of magnitude, considering the total cost at the national level would
 probably involve hundreds of billions of discounted dollars.  In other words, how
 can results at this level of confidence be used in the policy/regulation decision-
 making process?

 3.9.4 Presentation of Uncertainty Analysis in RIA

      The major concern of the MMRS with respect to the contents of the RIA
 relate to an inadequate representation of the magnitude of the uncertainties
 associated  with the cost and benefit estimates.  For the RIA, how much
 uncertainty can  you live with and still make an  intelligent decision regarding the
 efficacy of RCRA clean-up. This issue again pertains to the use of MMSOILS as a
 screening model, versus a realistic process model for estimating exposure and fate
 concentrations.

 3.10 Other User Groups for MMSOILS

 3.10.1  Applicability to Other EPA Program Activities

      The  utility of MMSOILS for estimating ecological risks may loom in future
 importance, for example hi relation to CERCLA.  Therefore, the MMRS
 recommends that this modeling construct should continue to receive attention,
 both in terms of review and in resources, to ensure that it has utility beyond
 RCRA.

 3.10.2  Use for State-Level Screening

      On a longer-term perspective, the MMRS recommends that the Agency
consider what might be its role in providing guidance to states as to the
appropriate types of models to use for state-level screening calculations.

3.10.3  Other User  Groups

      The  documentation makes it clear that MMSOILS is meant to be used by
nc ^-specialists.   The manual  needs  stronger statements to emphasize the model
limitations to such users, to recommend alternative models, and to emphasize the
inapplicability of the model to site-specific evaluations.  With regard to model
limitations, the MMRS recommer J   that each pathway model include a summary
table listing assumptions, and clearly stating the known parameter sensitivity
alongside site characteristics which might invalidate model results or which would
be expected to lead to results with order-of-magnitude uncertainties.

                                      31

-------
            APPENDIX A - BRIEFING AND REVIEW MATERIALS

 Review Materials for the April 22. 1993 SAB/EEC/MMRS Review Meeting:

 1)    U.S. EPA/OSWER, A memo entitled "Charge for SAB Review of Regulatory
      Impact Analysis Supporting the Corrective Action Regulation," signed by
      Richard J. Guimond, Assistant Surgeon General, U.S. Public Health Service
      and Deputy Assistant Administrator, Office of Solid Waste and  Emergency
      Response to Dr. Donald G. Barnes, Director, Science Advisory Board, March
      26, 1993

 2)    U.S. EPA/OSWER & ORD, A jointly signed memo entitled "Request for SAB
      Review of RCRA Corrective Action RIA," from Peter W. Preuss, Director,
      Office of Technology Transfer and Regulatory Support, and Richard J.
      Guimond, Assistant Surgeon General, U R  Public Health Service and Deputy
      Assistant Administrator, Office of Solid Waste and Emergency Response to
      Dr. Donald G. Barnes, Director, Science Advisory Board, June 26, 1992

 3)    U.S. EPA/SAB, Environmental Engineering Committee, MMSOILS Model
      Review Subcommittee, Notice of Subcommittee Open Meeting, Federal
      Register. Vol. 58, No. 67, Friday, April 9, 1993, p. 18395

 4)    U.S. EPA, "MMSOILS: Multimedia Contaminant Fate, Transport, and
      Exposure Model: Documentation and User's Manual," Office of Research and
      Development [Prepared by the Exposure and Assessment Group, Office of
      Health and Environmental Assessment and the Office of Environmental
      Processes and Effects Research], Washington, D.C. 20460, EPA XXX/X-
      XX/XXX draft document dated September 1992

 5)    U.S. EPA, "Draft Regulatory Impact Analysis for the Final Rulemaking on
      Corrective Action for Solid Waste Management Units: Proposed  Methodology
      for Analysis," Office of Solid Waste, March  1993

6)    U.S. EPA, "Draft Regulatory Impact Analysis for the Final Rulemaking on
      Corrective Action for Solid Waste Management Units: Proposed  Methodology
      for Analysis," APPENDICES, Office of Solid Waste, March 1993
Briefing Materials from the April
7)    U.S. EPA/OSW briefing entitled "Overview: Corrective Action Regulatory
      Impact Analysis, Proposed Methodology," Presented to the MMSOILS Model
      Review Subcommittee (MMRS) of the Environmental Engineering
      Committee (EEC), Science Advisory Board (SAB), by the Office of Solid
      Waste (OSW), April 22, 1993
                                    A-l

-------
     APPENDIX A - BRIEFING AND REVIEW MATERIALS: CONTINUED

Briefing Materials from the April 22. 1993 SAB/EEC/MMRS Review Meeting:
Continued:

8)    U.S. EPA/OSW, briefing entitled "Application of the MMSOILS Model in the
      Corrective Action Regulatory Impact Analysis," Presented to the MMRS of
      the SAB's EEC by the Office of Solid Waste (OSW), April 22, 1993

9)    U.S. EPA/OSW briefing entitled "Risk Assessment in the Corrective Action
     "Regulatory Impact Analysis," Presented to the MMRS of the SAB's EEC by
      the Office of Solid Waste (OSW), April 22, 1993

10)   U.S. EPA/OSW briefing entitled "Simulation of Remedy Effectiveness in the
      Corrective Action Regulatory Impact Analysis," Presented by the Office of
      Solid Waste (OSW) to the MMRS of the SAB's EEC, April 22, 1993

11)   U.S. EPA/ORD briefing entitled "RCRA Corrective Action RIA: ORD Input
      on Significant Technical Issues," Prepared and Presented by Stephen G.
      Schmelling of the Robert  S. Kerr Environmental Research Laboratory to the
      MMRS of the SAB's EEC, April 22, 1993

12)   U.S. EPA/ORD briefing entitled "RCRA Corrective Action RIA: ORD
      Participation (Fate & Transport), Presented to the MMRS of the SAB's
      EEC, April 22, 1993
Briefing Materials from the June 29. 1993 SAB/EEC/MMRS Review Meeting:

13)   U.S. EPA/SAB, Environmental Engineering Committee  (EEC), MMSOILS
      Model Review Subcommittee, Notice of Subcommittee Open Meeting for
      June 29, 1993 on MMSOILS Review, as well as EEC Open Meeting of June
      30 - July 1, 1993, Federal Register. Vol. 58, No.  108, Tuesday, June 8, 1993,
      p.32122

14)   U.S. EPA/OSW briefing entitled "Model Selection for the Corrective Action
      Regulatory Impact Analysis," Presented to the MMRS and selected
      specialists of the SAB's EEC by the Office of Solid Waste (OSW), June 29,
      1993

15)   U.S. EPA/OSW briefing entitled "Sample Selection for the Corrective Action
      Regulatory Impact Analysis," Presented to the MMRS and selected
      specialists of the SAB's EEC by the Office of Solid Waste (OSW), June 29,
      1993
                                   A-2

-------
     APPENDIX A - BRIEFING AND REVIEW MATERIALS: CONTINUED

Briefing Materials from the June 29. 1993 SAB/EEC/MMRS Review Meeting:
Continued:

16)   U.S. EPA/OSW, an untitled summary sheet, which lists SAB
      Recommendations and the OSW/ORD Next Steps, one page, both sides, June
      29,  1993

17)   U.S. EPA/OSW briefing entitled "Facility Characterization in the Corrective
      Action Regulatory Impact Analysis," Presented to the MMRS and selected
      specialists of the SAB's EEC by the Office of Solid Waste (OSW), June 29,
      1993

18)   U.S. EPA/ORD briefing entitled "SAB Presentation on Corrective Action
      RIA: MMSOILS Model Validation and RIA Uncertainty Assessment," a
      presentation by Mr. Gerry Laniak of ORD-Athens, June 29, 1993

19)   U.S. EPA/ORD untitled briefing comparing site-specific parameters by
      modeling teams, using central tendency and high-end estimates, and plots
      comparing the results of monitoring and modeling data by concentration
      (ppm), as well as plots on modeling and monitoring minimum curve and
      maximum error, a presentation by Mr. Gerry Laniak of ORD-Athens, June
      29, 1993
                                   A-3

-------
                    APPENDIX B - REFERENCES CITED

1)    Kecly, J.F., 'The Use of Models in Managing Ground-Water Protection
      Programs," EPA/600/8-87/003, ORD Ada, OK, 1987, 72p.

2)    Kezsbom, A. and A.V. Goldman, 'The boundaries of groundwater modeling
      under the law: Standards for excluding speculative expert testimony," Tort
      & Insurance Law Journal. Vol. XXVII, No. 1, 1991, pp. 109-126

3)    Konikow, Leonard F. and John D. Bredehoeft, "Ground-Water Models
      Cannot be validated," Advances in Water Resources 15, 1992, pp. 75-83

4)    National Research Council, "A review of ground water modeling needs for
      the U.S. Army," Washington, D.C., 1992

5)    National Research Council, Ground Water Models;  Scientific and Regulatory
      Applications, National Academy Press, Washington, D.C., 1990, 303 pp.

6)    U.S. EPA, "MMSOILS: Multimedia Contaminant Fate, Transport, and
      Exposure  Model: Documentation and User's Manual," Office of Research and
      Development [Prepared by the  Exposure and Assessment Group, Office of
      Health and Environmental Assessment and the Office of Environmental
      Processes and Effects Research], Washington, D.C.  20460, EPA XXX/X-
      XX/XXX draft document dated  September  1992

7)    U.S. EPA, "Draft Regulatory Impact Analysis for the Final Rulemaking on
      Corrective Action for Solid Waste Management Units: Proposed Methodology
      for Analysis," Office of Solid Waste, March 1993

8)    U.S. EPA, "Draft Regulatory Impact Analysis for the Final Rulemaking on
      Corrective Action for Solid Waste Management Units: Proposed Methodology
      for Analysis," APPENDICES, Office of Solid Waste, March 1993

9)    U.S. EPA, Expert Panel on the Role of Science at EPA (Loehr, Goldstein,
      Nerode and Risser), "Safeguarding the Future: Credible Science, Credible
      Decisions," EPA/600/9-91-050, March 1992

10)   U. S. EPA, Memorandum entitled, "EPA Definition of 'Pollution
      Prevention,"1 from F. Henry Habicht II, Deputy Administrator,  to all EPA
      Personnel, May 28, 1992
                                    B-l

-------
            APPENDIX B - REFERENCES CITED: CONTINUED:

 11)   U.S. EPA/SAB, "Review of the EPA Office of Solid Waste's (OSW)
      Unsaturated Zone Code for the OSW Fate and Transport Model (FECTUZ),"
      Report of the  Unsaturated Zone Code Subcommittee of the Environmental
      Engineering Committee (EEC), Science Advisory Board (SAB), EPA-SAB-
      EEC-88-030, July 12, 1988

 12)   U.S. EPA/SAB, "Resolution on Use of Mathematical Models by EPA for
      Regulatory Assessment and Decision-Making," Report of the Modeling
     -Resolution Subcommittee of the Environmental Engineering Committee
      (EEC), Science Advisory Board (SAB), EPA-SAB-EEC-89-012, January 13,
      1989

 13)   U.S. EPA/SAB, "Review of the CANSAZ Flow and Transport Model for Use
      in EPACMS," Report of the Saturated Zone Model Subcommittee of the
      Environmental Engineering Committee (EEC), Science Advisory Board
      (SAB), EPA-SAB-EEC-90-009, March 27, 1990

 14)   U.S. EPA/SAB, "Reducing Risk: Setting Priorities and Strategies for
      Environmental Protection," The report of the Science Advisory Board (SAB),
      Relative Risk Reduction Strategies Committee, EPA-SAB-EC-90-021,
      September 1990

15)   U.S. EPA/SAB, "Relative Risk Reduction Project: Reducing Risk, Appendix
      A," The Report of the Ecology and Welfare Subcommittee of the Relative
      Risk Reduction Strategies Committee (RRRSC) of the Science Advisory
      Board  (SAB), EPA-SAB-EC-90-021A, September 1990

16)   U.S. EPA/SAB, "Relative Risk Reduction Project: Reducing Risk, Appendix
      B," Report of the Human Health Subcommittee of the Relative Risk
      Reduction Strategies Committee (RRRSC) of the Science Advisory Board
      (SAB), EPA-SAB-EC-90-021B, September 1990

17)   U.S. EPA/SAB, "Relative Risk Reduction Project: Reducing Risk, Appendix
      C," Report of the Strategic Options Subcommittee of the Relative Risk
      Reduction Strategies Committee (RRRSC) of the Science Advisory Board
      (SAB), EPA-SAB-EC-90-021C, September 1990

18)   U.S. EPA/SAB, "Review of OSWER's Draft Report on Usage of Computer
      Models in the Hazardous Waste/Superfund Programs  and Proposed Pilot
      Study," Report of the Modeling Project Subcommittee of the Environmental
      Engineering Committee (EEC), Science Advisory Board (SAB),  EPA-SAB-
      EEC-91-016, September 6, 1991
                                   B-2

-------
             APPENDIX B - REFERENCES CITED: CONTINUED

 19)   U.S. EPA/SAB, "Leachability Phenomena: Recommendations and Rationale
      for Analysis of Contaminant Release," A Self-Initiated SAB Report on
      teachability Phenomena by the Environmental Engineering Committee
      (EEC), EPA-SAB-EEC-92-003, October 29, 1991

 20)   U.S. EPA/SAB, "Superfund Site Health  Risk Assessment Guidelines: Review
      of the Office of Solid Waste and Emergency Response's (OSWER) Draft
      Risk Assessment Guidance for Superfund Human Health Evaluation
      Manual," A report of the Environmental Health Committee (EHC) of the
      Science Advisory Board (SAB), EPA-SAB-EHC-93-007, February 22, 1993

 21)   U.S. EPA/SAB, "Review of the Office of Solid Waste and Emergency
      Response (OSWER) Assessment Framework  for Ground-Water Model
      Applications," A report of the Modeling  Project Subcommittee of the
      Environmental Engineering Committee (EEC), Science Advisory Board
      (SAB), EPA-SAB-EEC-93-013, June 21,  1993

22)   U.S. EPA/SAB, "Review of Draft Agency Guidance for Conducting External
      Peer Review of Environmental Regulatory Modeling," A letter report of the
      Modeling Peer Review Subcommittee of the Environmental Engineering
      Committee (EEC), Science Advisory  Board (SAB), EPA-SAB-EEC-LTR-93-
      013, August 12, 1993

23)   U.S. EPA, Risk Assessment Forum, "Framework for Ecological Risk
      Assessment, EPA/630/R-92/001, February 1992

24)   U.S. EPA, "Evaluation of Ground-Water Extraction Remedies," Office of
      Emergency and Remedial  Response (OERR), EPA/540/2-89/054, Volume 1:
      Summary Report, 1989

25)   U.S. EPA, "Evaluation of Ground-Water Extraction Remedies," Office of
      Emergency and Remedial  Response (OERR), EPA/540/2-89/054b, Volume 2:
      Case Studies 1 -19, 1989

26)   U.S. EPA, "Evaluation of Ground-Water Extraction Remedies: Phase II,
      Office of Emergency and Remedial Response (OERR), Publication 9355.4-05,
      Volume 1: Summary Report, 1992

27)   U.S. EPA, "Evaluation of Ground-Water Extraction Remedies: Phase II,
      Office of Emergency and Remedial Response (OERR), Publication 9355.4-
      05A, Volume 2: Case  Studies and Updates (on 24 Sites), 1992
                                   B-3

-------
           APPENDIX C - GLOSSARY OF TERMS AND ACRONYMS
 ATFERM   (Ad Hoc) Agency T_ask Force on Environmental fiegulatory Modeling
 CANSAZ   Combined Analytical-numerical Saturated Zone (Flow and transport
                  model for use in EPACMS)
 Class A     Known Human Carcinogen
 Class C     Suspect Human  Carcinogen
 Cwl        Concentration of Chemical in Waste Layer
            (milligrams/kilogram) (For instance, see Section 3.1.3)
 GJ.i        See Kdw|, below
 Cv         Contingent Valuation Methodology
 DAF        Dilution and Attenuation factor (For instance, see Section 3.3.2)
 DF         Fraction of Day  During Which Exposure Occurs (hours/24 hours) (For
            instance, see Section 3.1.3)
 DNAPLs    Dense non-Aqueous P_hase Liquids
 EAG        Exposure Assessment Group (U.S. EPA/ORD/OHEA)
 EEAC      Environmental Economics Advisory Committee (SAB/EEAC)
 EEC        Environmental Engineering Committee (SAB/EEC, also referred to as
            'The Committee")
 EHC        Environmental Health Committee (SAB/EHC)
 EPA        U.S. Environmental Protection Agency (U.S. EPA, or "The Agency")
 NATL      EPA Composite Model for Landfills
 EPACMS    EPA Composite Model for Surface Impoundments
 EPEC       Ecological Processes and Effects Committee (SAB/EPEC)
 FECTUZ    finite-Element Code for Simulating Water Flow and Solute Transport
            in the  Unsaturated Zone (Variably saturated porous media)
 K          First order Coefficient, Which Measures Losses of Contaminant Due
            to Pumping (Also referred to as Simplified Decay Rate Coefficient)
            (For instance, see Section 3.6.3)
            Decay  Rate/Distribution Coefficient. Also Referred to as Soil-Water
            Partition Coefficient (milliliters/gram)
            Solid-Water Partition Coefficient Between the Solid Waste and the
            Liquid Leachate  (liters/kilogram). (Also referenced as Cdwl ) (For
            instance, see Section 3.1.3)
            Water-Phase Mass Transfer Coefficient. (Also referred to as the
            Octanol-Water Partition Coefficient.) (For instance, see Section 3.2.5)
LNAPLs     Light non-Aqueous Phase Liquids
MMSOILS   Mathematical Model for Soils (A Multi-Media Contaminant, Fate,
            Transport and Exposure Model.)
MINEQL    A Chemical Speciation Model (For instance, see Section 3.3.2)
MMRS      MMSOILS Model Review Subcommittee (U.S. EPA/SAB/EEC; Also
            referred to as "the Subcommittee")
NAPLs      non-Aqueous P_hase Liquids
NORM      naturally-Occurring Radioactive Material
OEPER     Office  of Environmental Processes and Effects  gesearch (U.S.
            EPA/ORD)
K
 •dwl
                                   C-l

-------
    APPENDIX C - GLOSSARY OF TERMS AND ACRONYMS: CONTINUED

OHEA     Qffice of Health and Environmental Assessment (U.S. EPA/ORD)
ORD       Qffice of Research and Development (U.S. EPA)
OSW       Qffice of Solid Waste (U.S. EPA)
OSWER    Qffice of Solid Waste and Emergency Response (U.S. EPA)
pH        Negative Log of Hydrogen Ion Concentration
Qm        Monthly Net recharge (cubic meters/month) (Also Referenced as qm)
           (For instance, see Section 3.1.3)
qm        Monthly Net Recharge (cubic meters/month) (Also Referenced as Qm)
           (For instance, see Section 3.1.3)
RCRA      Resource Conservation and Recovery Act
RfDs       Reference Boseg
RIA        Regulatory Impact Analysis
SAB       Science Advisory Board (U.S. EPA)
SWMUs    Solid Wjaste Management Units       EPA-SAB-EC-LTR-94-OQ2
TC        loxicity Characteristic               Overview of  SAB  comments on
U.S.        United States                       the proposed  RIA for RCRA
                                             corrective  action rule
                                                 ^ 1 r
                                                          -002
                                              - r
                                          	- - - - 1"
                                               t   _ _.„ —-
                                                 ENVIRONMENTAL
                                                   PROTECTION
                                                     AGENCY
                                                  DALLAS, TEXAS

                                                  LIBRARY
                                 C-2
                                                                       DEMCO

-------
                           DISTRIBUTION LIST

 Deputy Administrator
 Assistant Administrators
 EPA Regional Administrators
 EPA Laboratory Directors

 Deputy Assistant Administrator for Office of Research and Development (ORD)
   Director, Center for Environmental Research Information (CERI)
 Director, Office of Environmental Processes and Effects Research (OEPER)
   Director, Office of Health and Environmental Assessment (OHEA)
 Director, Exposure and Assessment Group (EAG)

 Deputy Assistant Administrator for Office of Solid Waste and Emergency Response
 (OSWER)
   Director, Office  of Emergency and Remedial response (OERR)
 Deputy Director, OERR
 Director, Office of Solid Waste (OSW)
 Deputy Director, OSW

EPA ATFERM Co-Chairs (Agency Task Force on  Environmental Regulatory
Modeling)
EPA Headquarters Library
EPA Regional Libraries
EPA Laboratory Libraries

-------