United States Science Advisory EPA-SAB-EEAC-94-001
Environmental Board (1400F) November 1993
Protection Agency Washington DC
&EPAAN SAB REPORT:
CONTINGENT VALUATION
METHODOLOGY (CV1)
REVIEW OF THE CONTINGENT
VALUATION METHOD FOR THE
PROPOSED RIA FOR RCRA
CORRECTIVE ACTION RULE BY THE
ENVIRONMENTAL ECONOMICS
ADVISORY COMMITTEE
-------
NOTICE
This report has been written as a part of the activities of the Science Advisory
Board, a public advisory group providing extramural scientific information and advice to
the Administrator and other officials of the Environmental Protection Agency. The
Board is structured to provide balanced, expert assessment of scientific matters related
to problems facing the Agency. This report has not been reviewed for approval by the
Agency and, hence, the contents of this report do not necessarily represent the views
and policies of the Environmental Protection Agency, nor of other agencies in the
Executive Branch of the Federal government, nor does mention of trade names or
commercial products constitute a recommendation for use.
-------
ABSTRACT
The EEAC addressed the design, conduct, and results of the contingent
valuation study (undertaken for the EPA Office of Solid Waste by Drs. McClelland,
Schulze, et a/.), focusing on a Charge organized around five major questions: a) the
survey respondents' understanding of groundwater resources; b) selection of the best
method for estimating non-use values from the survey responses; c) use of the Box-
Cox econometric procedure to address large bids; d) the problems of embedding, non-
bids, and scenario rejection; and e) the applicability of the valuations obtained in this
study as a basis for EPA to determine the non-use values of groundwater. The
Committee commends EPA staff for supporting exploratory research of this nature.
There is little doubt that this study represents a substantive contribution, extending our
understanding of the issues associated with contingent valuation estimation of non-
market values. Addressing the last, but most encompassing element of the Charge
first, the Committee can not endorse the McClelland et al. study as a means of generat-
ing valid and reliable estimates of the nonuse values associated with cleaning up
contaminated groundwater. Specifically, the Committee has no confidence that the
respondents were clear about what it is they were being asked to value. Although the
study was innovative in a number of respects, this most basic failing gives the Commit-
tee no choice but to question the validity of the findings. Addressing other aspects of
the Charge: a) The Committee does not believe that the pre-testing and survey design
techniques offer convincing evidence that a well-defined groundwater commodity was
understood properly by all the respondents; b) The Committee does not believe that
any of the three possible methods for separating the non-use or passive use values
from total values can be established as preferred at this time; c) the Committee deems
it impossible to judge whether the Box-Cox econometric estimates alone provide an
acceptable and defensible method for dealing with the scenarios and the large bids
associated with them; and d) the Committee does not believe that the approaches for
treating embedding, scenario rejection, and the potential effects of non-bidding
responses can be assessed for their reliability on the basis of the information provided
in the report. The EEAC feels that the problems in using the study results to meet the
needs of the RIA effort arise from requirements imposed on the research by the EPA,
including the need for separate estimate of nonuse value and for a method that
abstracted from the specific features of the local conditions associated with each
specific case of groundwater contamination. The approaches taken to deal with these
requirements have no basis in the theory of non-market valuation, nor precedent in
practice, and were never subjected to peer review. The Committee's report offers
specific suggestions for further research to help resolve the questions raised by this
study, including the criteria for deciding which households would be among the groups
demanding increases in the amount available of specific commodities or values and
study of the sensitivity of CV outcomes to the survey methods used
KEYWORDS: contingent valuation; groundwater; hedonic models; nonuse values;
RCRA RIA
-------
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
SCIENCE ADVISORY BOARD
ENVIRONMENTAL ECONOMICS ADVISORY COMMITTEE
RCRA RIA REVIEW
CO-CHAIRS
Dr. Allen V. Kneese, Resources for the Future, Washington DC
Dr. V. Kerry Smith, Department of Economics, North Carolina State University, Raleigh NC
MEMBERS
Dr. Nancy E. Bockstael, Department of Agricultural and Resource Economics, University of
Maryland, College Park, MD
Dr. A. Myrick Freeman, Department of Economics, Bowdoin College, Brunswick, ME
Dr. Charles D. Kolstad, Department of Economics, University of Illinois, Urbana, IL
Dr. William Nordhaus, Department, of Economics, Yale University, New Haven CT
Dr. Bryan Norton, School of Public Policy, Georgia Institute of Technology, Atlanta GA
Dr. Wallace E. Dates, Department of Economics, University of Maryland,College Park, MD
Dr. Paul R. Portney, Resources for the Future, Washington DC
Dr. Robert Repetto, World Resources Institute, Washington, DC
Dr. Richard Schmalensee, Massachusetts Institute of Technology, Cambridge MA
Dr. Robert N. Stavins, Kennedy School of Government, Harvard University, Cambridge, MA
Dr. Thomas H. Tietenberg, Department of Economics, Colby College, Waterville, ME
Dr. W. Kip Viscusi, Department of Economics, Duke University, Durham, NC
SAB COMMITTEE LIAISONS
Dr. William Cooper (EPEC), University of Michigan
Mr. Richard Conway (EEC), Union Carbide Corporation
Dr. Morton Lippmann (IAQC), Nelson Environmental Institute, New York University
Dr. Roger McClellan (CASAC), Chemical Industry Institute of Toxicology
DESIGNATED FEDERAL OFFICER
Mr. Samuel Rondberg, Environmental Health Committee, Science Advisory Board (1400F), U.S.
Environmental Protection Agency, Washington, DC 20460
STAFF SECRETARY
Ms. Mary L. Winston, Environmental Protection Agency, Science Advisory Board (1400F), Wash-
-------
ington, DC 20460
IV
-------
TABLE OF CONTENTS
1 EXECUTIVE SUMMARY 1
2 INTRODUCTION 6
2.1 Background 6
2.2 Charge 6
3 DETAILED DISCUSSION 9
3.1 Organization of the Report 9
3.2 Outline of Committee's Review Procedures for the Preparation of this
Report 9
3.3 Committee Evaluation and Response to the EPA Charge 11
3.3.1 Comparison with the NOAA Panel Guidelines 11
3.3.2 Specific Issues of Concern 14
3.3.2.1 Definition of the Commodity 14
3.3.2.2 Embedding 17
3.3.2.3 Partitioning of Total Values 17
3.3.2.4 Consistency with Mitchell-Carson Existence Value
Study 19
3.3.2.5 Box-Cox Estimation and Treatment of Zero Bids 20
3.3.2.6 Statistical Analysis of Pre-test and Final Samples .... 21
3.3.3 Responses to Specific Charges 23
4 CONCLUSIONS AND RECOMMENDATIONS 26
4.1 Summary 26
4.2 Recommended Research 27
5 REFERENCES R-1
-------
1 EXECUTIVE SUMMARY
The EPA Office of Solid Waste (OSW) has been developing methodologies to
assess the benefits of cleaning up (corrective action) contamination at active hazard-
ous waste facilities operating under the authority of the Resource Conservation and
Recovery Act. OSWs definition of potential benefits includes non-use values of clean
groundwater. The OSW (along with the Office of Policy, Planning, and Evaluation) has
supported a Cooperative Agreement with the University of Colorado to conduct a
contingent valuation study (McClelland et al., 1992) to estimate these non-use values.
The OSW requested that the SAB Environmental Economics Advisory Commit-
tee (EEAC) address the design, conduct and results of the contingent valuation study
with particular attention to methodological limitations that have been identified in the
literature on contingent valuation. The Committee was asked to review the study in the
context of the state of the art for contingent valuation methodology, but to be aware of
the practical constraints under which the effort had been carried out (e.g., budget
limitations). The Committee devoted three meetings (in whole or in part) to this review -
- December 18, 1992, April 30, 1993, and July 13, 1993, completing a detailed review
of the research. The Charge for the EEAC review was organized around five major
questions posed by the OSW, and summarized below:
a) Did survey respondents understand groundwater sufficiently to give a
meaningful value to the commodity in question?
b) Which of three similar methods should be used for estimating non-use
values from the survey responses?
c) Is the Box-Cox econometric procedure an acceptable and defensible
method for addressing large bids?
d) Embedding has been identified as a problem in contingent valuation
studies. Were the methods used to address the issue of non-bids,
scenario rejection and embedding reasonable?
e) Are the valuations obtained in this study sufficiently accurate and repro-
ducible so as to be used, in part, as a basis for EPA to determine the non-
use values of groundwater? Can the SAB advise the Agency on
-------
analyses of the survey results to improve the estimation of non-use
values for groundwater?
Addressing the last, but most encompassing, element of the charge first, the
Committee can not endorse the McClelland et al. study as a means of generating valid
and reliable estimates of the non-use values associated with cleaning up contaminated
groundwater. Moreover, the Committee questions the study's applicability within the
context of the forthcoming RCRA Corrective Action Rule Regulatory Impact Analysis
(RIA). This CV study design flows from the premise that the values people place on
cleaning up groundwater that they are told will have no effect on their own water
supplies can be measured independently of the specific circumstances associated with
each contaminated groundwater resource. The benefits (or value) provided to people
from cleaning up contaminated groundwater are measured by what people would be
willing to give up to obtain that improvement. With most commodities, these "sacrific-
es" can be measured from market transactions. For many environmental commodities
this is not possible. Indeed, in situations where people experience satisfaction from the
knowledge that a resource is restored or preserved irrespective of any direct use they
might make of it, there is a particular need for the type of innovative research strategies
attempted in this McClelland et al. study. Their approach used survey methods to ask
a representative sample of adult decision makers what they would be willing to pay to
clean up a contaminated groundwater resource. While the survey approach is not
accepted by all economists, it has been accepted in principle for natural resource
damage assessment by the recent (Federal Register, January 15, 1993) report of the
National Oceanographic and Atmospheric Administration (NOAA) panel of distin-
guished social scientists (including two Nobel Laureates) composed of five economists
and a survey researcher.
The Committee's concerns about the McClelland etal. findings relate primarily to
specific details of this study's application of the CV technique. It was apparent to
Committee members that people answering the study's survey instrument could have
interpreted the services provided by cleaning up contaminated groundwater in a
number of different, conflicting ways. There is no way to know which of these multiple
meanings these respondents adopted in answering the valuation questions.
Addressing other aspects of the Charge:
a) The Committee does not believe that the pre-testing and survey design
techniques offer sufficient evidence to ensure that a well-defined ground-
water commodity was understood properly by all the respondents. The
sample sizes for the pre-tests were not large enough to permit
-------
multivariate analysis that would allow evaluation of some of the judgments
made by the principal investigators.
b) The Committee does not believe that any of the three possible methods
for separating the non-use or passive use values from total values can be
established as preferred at this time. Each should be regarded as an
innovative research technique and not one of proven reliability for esti-
mating non-use values from survey responses.
c) It is not possible to judge whether the Box-Cox econometric estimates
alone provide an acceptable and defensible method for dealing with the
scenarios and the large bids associated with them. Comparative evalua-
tions of these results, and the results of other approaches for dealing with
the skewness in the valuation responses across a number of surveys,
should be undertaken before accepting the Box-Cox results. Because
there are several different hypotheses about the source of large bids, the
literature does not provide one set of criteria for discriminating among
these approaches. Comparative evaluations provide one means of
evaluating whether the specific approaches used to address skewed bid
distribution influences the summary statistics used for policy analysis or
the conclusions drawn or based on survey findings.
d) We do not believe that the approaches for treating embedding, dealing
with scenario rejection, and reflecting the potential effects of non-biding
responses can be assessed for their reliability on the basis of the informa-
tion provided in the report.
The above problems notwithstanding, the Committee wishes to commend EPA
staff for supporting exploratory research of this nature. There is little doubt that this
report represents a substantive contribution, extending our understanding of the issues
associated with contingent valuation estimation of non-market values. It provides new
research insights to the evolving literature on contingent valuation. However, such
innovation is not the criterion that the Committee was asked to use in evaluating the
report, nor the foundation for use of the study by EPA. In many respects, the Commit-
tee feels that the problems in using the findings of the study to meet the needs of the
RIA effort arise from requirements imposed on the research - the need for a separate
estimate of non-use value and for a method that abstracted from the specific features of
the local conditions associated with each specific case of groundwater contamination.
After reviewing the full record, the Committee feels that this strategic decision on EPA's
-------
part is the cause of many of the study's problems as well as the difficulties of imple-
menting the research findings in the RIA context. The approaches taken to deal with
these requirements have no basis in the theory of non-market valuation, nor precedent
in practice, and were never subjected to peer review.
The Committee's report offers specific suggestions for further research to help
resolve the questions raised by this study. Five general areas seem especially relevant
for EPA's activities associated with the corrective action RIA:
a) For the most part, economic research has focused on developing esti-
mates of the values of the typical or representative household demanding
increases in the amount available of specific commodities (or improve-
ments in them) and not the criteria for deciding which households would
be among the groups having such demands or values. Addressing this
question is fundamental to the task of measuring aggregate values for the
cleanup of groundwater resources and may well have a much greater
quantitative impact on these aggregate estimates than do refinements in
estimates of the representative household's values for changes in some
environmental commodities.1
b) Detailed study of the sensitivity of CV outcomes to the survey methods
used - whether telephone, in-person, or mail surveys is needed. The evi-
dence provided to the Committee suggests that there remains some
questions in the profession about the results provided by telephone or
mail surveys in comparison with in-person interviews. More research on
this topic is clearly warranted.
c) Research addressing issues associated with defining changes in envi-
ronmental commodities is central to understanding whether the contingent
valuation responses are internally consistent. The recommended
research would involve developing methods and practical protocols that
could be used to understand how environmental commodities are best
measured and how those measures relate to the descriptions offered in
the framing of contingent valuation questions.
Given the extremely limited nature of the available research on the value of groundwater cleanup, the Committee feels that in this
case both types of research are exceptionally important.
-------
d) Research on the need for, and the ability of, respondents separating their
total value of increases in environmental commodities (such as the
cleanup of contaminated groundwater) into use and non-use components.
e) Research on the development of methods for gauging the potential
implications of using a hypothetical setting to elicit valuations in relation-
ship to real choices is important to the ongoing development of the
contingent valuation method. Research evaluating whether or not
contingent valuation results can be transferred from one setting to others
is also critical. The Committee believes this should be a central compo-
nent of the future research that EPA undertakes in support of its policy
evaluation.
The Committee assumes that there will be a continuing need to evaluate
analyses intended to appraise household's willingness to pay for improvements in
environmental commodities. There is simply not sufficient information on the proverbial
"research shelf" to address in a meaningful way many of the questions that have been
posed to the EEAC about specific decisions in the design of contingent valuation
surveys and in the transfer of results from such surveys. EPA staff have attempted to
do their best to meet policy needs in the presence of this limited information. Their
Charge to the EEAC reflected a desire to have the Committee's judgement, based on
its collective research experience, substitute for a documented research record on
these issues. Our conclusions reflect the fact that the problem is not simply that EPA
has been unable to sustain a research program to address these questions. Overall,
there does not exist a set of research available for the Committee to use in forming its
own judgements, EPA should begin to sponsor the research required to evaluate the
methods used in its policy evaluations and to develop more experience in valuing
changes to important environmental resources.
-------
2 INTRODUCTION
2.1 Background
The EPA Office of Solid Waste (OSW) is working to complete a final rule
requiring the clean-up of contamination of active hazardous waste facilities operating
under the authority of the Resource Conservation and Recovery Act (RCRA). A
significant area of benefits from the rule is expected to be the clean-up of groundwater
contamination, as well as other environmental media. To provide decision makers with
a complete assessment of the benefits of corrective action, OSW has adopted a broad
definition of potential benefits that includes non-use values of clean groundwater.
The OSW has provided support to a Cooperative Agreement with the University
of Colorado to conduct a contingent valuation study to estimate these non-use values.
The contingent valuation study is moving toward completion, under the direction of Drs.
McClelland, Schulze, Lazo, Waldman, Doyle, Elliot, and Irwin.
The OSW requested that the SAB address the design, conduct and results of
the contingent valuation study. The researchers were aware of the need to cope with
four potential sources of error that have been identified in the literature on contingent
valuation. These sources include: (1) large bids not indicative of willingness-to-pay;
(2) scenario rejection or refusals to bid; (3) embedding; and (4) effects of context on
bids.
Although the study was conducted to place emphasis on pretesting, instrument
design, sampling, and econometric analysis of CV survey data, there were also a
number of practical constraints. The survey instrument length was limited to insure a
reasonable response rate from a mail survey. A mail survey was used because
estimates on a national scale were needed; the sample size had to be large enough to
allow comparison among survey variants and tests of several key questions. Lastly,
the budget was limited. The Committee was asked to review the study in the context of
the state-of-the-art for contingent valuation methodology, but to be aware of the con-
straints noted above.
2.2 Charge
The specific charge was organized around five questions. They are summarized
below:
-------
a) The survey instrument design was developed through focus groups using
a cognitive survey design approach and different survey variants. Two
levels of context were included among the survey variants to test the
effect of context. This was particularly important because groundwater is
an exotic commodity. There was concern during pretesting that ground-
water would not be understood well enough by respondents to properly
give a value. Were the pretesting and subsequent survey design
techniques reasonable methods to use? Does OSWER have
sufficient evidence and support, after using these methods, to insure
that groundwater was understood enough to properly give a value?
b) The survey was designed to allow estimation of non-use values for
groundwater by three possible methods. The three methods produced
similar results. Does the SAB have a preference regarding which
method should be used for estimating non-use values from the
survey responses?
c) The study used the econometric technique of Box-Cox transformations to
address large bids. OSWER believes that the information content of
large bids needs to be considered in the results. Is the Box-Cox econo-
metric procedure an acceptable and defensible method for ad-
dressing large bids?
d) The survey had to address the issue of non-bids or scenario rejection by
using careful survey design techniques. Scenarios that people would
accept were pretested extensively. The cognitive survey design approach
was instrumental in facilitating the pretesting. Embedding has also been
identified as a problem in contingent valuation studies. Embedding was
addressed by posing explicit questions that ask the respondent to rethink
a previous answer and reduce their bid based on how much the values
previously given for groundwater cleanup were actually for other
environmental problems. Were the methods used to address the issue
of non-bids, scenario rejection and embedding reasonable?
e) In developing regulations under its several legislative mandates, the
Agency is required to produce analyses that determine the benefits of
regulations intended to protect groundwater. Are the valuations
obtained in this study sufficiently accurate and reproducible so as to
-------
be used, in part, as a basis for EPA to determine the non-use values
of groundwater? Can the SAB advise the Agency on analyses of the
survey results to improve the estimation of non-use values for
groundwater?
8
-------
3 DETAILED DISCUSSION
3.1 Organization of the Report
The Committee's report on the review of this contingent valuation study is
organized into three sections. In the first section we describe our evaluation process.
We include this section because the steps we took in our review are quite relevant to
our judgment on the report
In the second section, we discuss the implications of the NOAA Contingent
Valuation Panel's Report for the evaluation of McClelland, et al., as well as several
issues that were not addressed in the charge. We summarize our conclusions with
respect to each of the issues identified in the charge. And we present our overall
reactions to this report.
In the third section, we identify three major areas of substantive research
concerning the use of contingent valuation methods for the valuation of nonmarket
resources such as groundwater. We recommend that EPA initiate a program of
research in each of these areas.
3.2 Outline of Committee's Review Procedures for the Preparation of this Report.
The McClelland et al. contingent valuation study of groundwater reviewed was
prepared as part of the research necessary for EPA's Regulatory Analysis for its
Corrective Action Rule for RCRA sites. Our review was undertaken in several stages.
A detailed review was required because the Office of Solid Waste did not provide any
information indicating that the McClelland et al. contingent valuation report had been
subjected to a peer review process prior to submitting it to the Committee.2 Indeed, it is
our understanding that the research design for the benefit transfers planned as part of
the Regulatory Impact Analysis was not subjected to a comprehensive external review.
We believe this was a mistake. It had a significant effect on the design of the
McClelland et al. research, and therefore we will return to it below. In the absence of a
documented peer review of the draft final report, the Committee believed it was
essential to conduct a detailed peer review of the research as well as to evaluate the
issues associated with the Charge given to the Committee.
Subsequent to the preparation of this report, peer review materials were provided to one of the Committee Chairs. These materials
addressed an interim report, not the draft final report. Moreover, there was no indication of how the research design was modified to
respond to the comments.
-------
The first stage of our review consisted of a briefing on December 18, 1992 by
EPA staff on the goals specified for the contingent valuation study. In this presentation
EPA staff outlined its needs in relationship to the Regulatory Impact Analysis. The staff
identified a pre-defined set of research goals that were imposed on the principal
investigators and that conditioned their decisions concerning research strategies. One
of the goals was to obtain an estimate of the non-use values (or "passive use" value,
using the terminology of the 1989 Court of Appeals decision and the NOAA Contingent
Valuation Panel Report) for the cleanup of groundwater resources that could be applied
as a separate component to any use values estimated separately on a site specific
basis. It is our understanding, based on that briefing, that this goal was specified by
EPA staff. It was not based on a recommendation of the principal investigators as a
component of their research design. Nor was it a goal that was developed as a result
of existing research findings on the measurement of use and non-use values (i.e., the
earlier Mitchell-Carson (1989) study using focus groups to evaluate how people
conceived of groundwater resources and whether they might have non-use values for
them).
Following the overview of EPA goals, the Committee Co-Chair (Smith) presented
an overview of contingent valuation methods and an initial summary of the key
elements in the McClelland et al. contingent valuation survey. Because the Committee
was aware of the ongoing NOAA Contingent Valuation Panel's assessment of the CV
method, the Committee decided to wait until the report of that panel was available
before undertaking a detailed review of the McClelland et al. study (The report was
subsequently published in the Federal Register for January 15, 1993).
The second stage of our review process involved assembling a detailed set of
questions for the principal investigators of the CV study. These questions summarized
the issues that arose in the Committee's initial review of the McClelland etal. study.3
The questions posed to the authors also outlined the concerns that the Committee felt
needed to be addressed by the principal investigators in a subsequent meeting.
As part of the preparation of the equivalent of a peer review, the Committee
requested three sets of consultative activities. First, the Committee asked Dr. Robert
C. Mitchell to evaluate the background information forming the basis for the design of
the contingent valuation survey. Mitchell, along with Dr. Richard T. Carson, had
conducted for EPA qualitative research on the issues associated with attempting to
The questions were transmitted to the authors in a memorandum from Kerry Smith and Allen Kneese to William Schulze January
11, 1993.
10
-------
measure non-use or existence values for groundwater prior to the McClelland et al.
report cited above). The McClelland et al. report indicates that it intended to build upon
the findings of the Mitchell-Carson study. The charge given to Mitchell (Smith and
Kneese memorandum, March 23, 1993) proposed that his review focus on the
interpretation given to this earlier Mitchell-Carson work by McClelland etal.
The Committee asked Drs. Richard Bishop and Gregory Poe to evaluate the
implications of their recent study of the influence of information on individual's valuation
of groundwater cleanup. These researchers were asked to consider their findings in
relation to the information that was provided to respondents in the McClelland et al.
contingent valuation study. Finally, Dr. Kevin Boyle was asked to prepare a detailed
review of the literature including the McClelland et al. study. This review was to include
a comparative evaluation of all aspects of the earlier literature on the contingent
valuation method to value cleanup of groundwater resources (K. J. Boyle, 1993).
The third stage in the Committee's review process involved a second meeting of
the Committee (a) to: discuss the report of the NOAA Panel on contingent valuation
methods; (b) to allow the principal investigators to describe their research design in
more detail and to provide detailed responses to the January comments and questions;
and (c) to hear the reports of the three groups of commissioned consultants to the
Committee. Following this meeting, a draft report of the Committee was prepared and
circulated to the Committee members. The Committee's third and final meeting
evaluated and revised this report and finalized the Committee's recommendations
about the study. During the final meeting, Dr. William Schulze presented some
clarifying information to the Committee. A summary of the authors' written responses to
Committee questions (McClelland, Schulze, and Lazo, June 23, 1993) was circulated
shortly before the final meeting.
3.3 Committee Evaluation and Response to the EPA Charge
3.3.1 Comparison with the NOAA Panel Guidelines.
The Committee decided to delay its review of the McClelland et al. report until
after the NOAA Panel's report on the contingent valuation method was available. This
was done because the Committee did not have the resources or time to undertake its
own review of the CV method. The NOAA Panel concluded that "contingent valuation
studies can produce estimates reliable enough to be the starting point for a judicial or
administrative determination of natural resource damages - including lost passive use
value." After reviewing the objectives of the NOAA Panel and its recommendations, the
11
-------
EEAC acknowledges that the mandate requested of the NOAA panel differs from that
associated with using the results of a CV analysis for general policy purposes.
Moreover, the Committee does not intend to endorse or agree with all aspects of the
NOAA report. Rather, we have simply used the NOAA Panel's guidelines as one of
several starting points for an evaluation of the McClelland et al. study.
The NOAA Panel's guidelines for the design and implementation of CV survey
methods included4: use of in-person interviews; use of a referendum (or discrete
choice) format in asking contingent valuation questions; obtaining a high response rate
to the questions; investigating the responsiveness of the "willingness to pay responses"
to changes in the amount of the environmental commodity offered to respondents;5
attempting to determine whether respondents understood the tasks, considered their
budgets in the process of answering contingent valuation questions, and believed the
scenario presented to them. With discrete choice questions, the NOAA Panel
recommended a follow-up to any yes/no questions with specific attempts to identify
responses that indicated concern over available resources to make the stated pay-
ments. Scenarios should demonstrate that respondents considered seriously the
private and public substitutes for the commodities offered as well as these budget
constraints.
Because the McClelland et al. study was conducted long before the NOAA report
was available, it is unrealistic to expect a direct correspondence between the
procedures used in the McClelland etal. report and the NOAA Panel's recommenda-
tion. It also should be noted that not all of the members of this Committee fully endorse
all of the NOAA Panel's guidelines. Nonetheless, the EEAC started its evaluation by
4
These comments paraphrase the specific requirements identified in the NOAA Panel's summary of conditions required to satisfy
their burden-of-proof requirement. They also include the recommendations made about specific implementation procedures. The
specific citation for text of the report is: Federal Register, January 15, 1993, Vol. 58, #10, pp. 4601-4614.
The NOAA Panel report uses the term "scope" to refer to changes in the environmental commodities that are intended to represent
the injuries to one or more specific environmental resources. It is identified as "scope" in their burden of proof requirements and that
specific patterns of responsiveness in willingness to pay acknowledges that the predictions from economic theory about the properties of
the willingness to pay are limited. This is especially true when there may be more than one environmental resource affected and the
reductions in injuries (i.e., increases in the environmental commodities involved) that are being valued may be different across resources.
The specific text of the NOAA Panel's report that seems to be identified as being associated with the scope requirement describes the
condition as follows: Rationality in its weakest form requires certain kinds of consistency among choices made by individuals. For
instance, if an individual chooses some purchases at a given set of prices and income, then if some prices fall and there are no other
changes, the goods that the individual would now buy make him or her better off. Similarly, we would expect an individual's preferences
over public goods (i.e. bridges, highways, air quality) to reflect the same kind of consistency.
Common notions of rationality impose other requirements which are relevant in different contexts. Usually, though not
always, it is reasonable to suppose that more of something regarded as good is better so long as an individual is not
satiated. This in general translated into a willingness to pay some what more for more of a good as judged by the individual.
Also, if marginal or incremental willingness to pay for additional amounts does decline with the amount already available, it is
usually not reasonable to assume that it declines very abruptly (emphasis added) pp 4604.
12
-------
considering the McClelland etal. study's design in relationship to the NOAA Panel
recommendations. Several notable differences were identified. McClelland etal. used
mail surveys and not in-person interviews, a fact that was a concern of some members
of the Committee. The interview format did not allow evaluation of whether the
questionnaires were understandable to respondents of the final survey and did not
incorporate a test for the effects of changes in the scope of the commodity offered
respondents, as recommended by the NOAA panel. While considerable attention was
paid to a specific set of substitutes, the set of substitutes for contaminated groundwater
was narrowly defined and the set of substitutes available with scenario rejection was
not made clear. The NOAA Panel did not discuss the procedures employed in the
McClelland et al. report for dealing with disembedding and for partitioning the total
willingness to pay into use and passive use components. Additionally the study did not
utilize the close-ended or referendum format for the contingent valuation questions,
using instead a payment card approach. While the discussion of the pre-test results
suggests that a closed-ended contingent valuation was evaluated, the questions used
were not in the same format as recommended by the NOAA Panel (based on informa-
tion given in the report, the principal investigator's comments, and the Boyle (1993)
review). The authors' pre-test did not evaluate the discrete choice approach recom-
mended by the NOAA Panel. Thus, the questioning format did not correspond to that
recommended by the NOAA Panel.
Overall, the McClelland etal. study does not satisfy the burden-of-proof
standards described by the NOAA Panel for reliable CV assessments of the passive
use values associated with natural resource damages. It is important to acknowledge,
however, that the NOAA Panel report did not describe these features of CV studies as
absolute criteria for reliability. The report acknowledges that reliable estimates could
be realized without full adherence to all the recommendations. As the NOAA Panel
report clearly indicates, it is possible for CV studies to achieve comparable levels of
reliability without adhering to their recommendations, but the burden of proof must be
satisfied by other means. Accordingly, given the Committee's position that it does not
necessarily endorse the NOAA Panel's recommendations, our evaluation of reliability
and validity was based on the general definitions of these concepts as they have been
used in the CV literature (R. C. Mitchell and R. T. Carson, 1989). As such, this
evaluation is a professional judgment that took the comparison with the NOAA criteria
as an approximate template in the Committee's first stage review of the McClelland et
al. study.
The McClelland et al. report exhibited several commendable features. Although
13
-------
it used mail questionnaires, it did achieve a relatively high response rate (63.4 percent
of total questionnaires were returned; 43.7 percent of the total were available for
regression analysis). The report included an extensive amount of qualitative analysis
associated with the design of their survey instrument and several pre-tests to evaluate
its performance. At the early design stage of questionnaire development, there was
extensive attention to verbal protocol analysis, focus groups, and other qualitative
approaches to evaluate the extent to which individuals were comprehending the
commodity that was being offered to them. Equally important, a series of pre-tests
were used to evaluate the sensitivity of responses to the structuring of the valuation
questions. While concerns were raised about the relative sizes of the samples
underlying these pre-tests, the Boyle (1993) appraisal and other reviewers suggested
that this amount of work was among the most extensive ever conducted in a contingent
valuation study for groundwater cleanup. Unfortunately, there does not appear to have
been evaluation of the final survey instrument to determine whether the judgements
from the pretests were consistent with an independent set of respondent's
understanding of the CV questions.
3.3.2 Specific Issues of Concern
The Committee found difficulties with several elements of the design and
analysis presented in the McClelland etal. report. Some of these are highlighted in
brief terms in our summary judgment. We discuss a few of them in more detail below.
3.3.2.1 Definition of the Commodity
The first of these concerns the definition of the commodity to be valued by
respondents. The Committee found that there were multiple interpretations of what was
offered to respondents based on the information conveyed by the questions comprising
the valuation task. The most direct explanation of the Committee's reasons for concern
stems from the reactions of Committee members to the questionnaires. Careful
readings of the survey instrument by several members of the Committee resulted in
different interpretations of the commodity that was being valued. These differences
(along with similar questions about the survey instruments associated with the
experimental variations included as different versions) raised questions about the
ability of lay persons to understand the nature of the commodity they were being asked
to value.
The Committee found that the sample sizes and evaluation of the information
materials and pre-testing and design techniques were not sufficient to statistically test
14
-------
whether respondents had enough information and understood the commodities offered
to them. Equally important, it was not possible to discriminate between the various
interpretations of the commodity offered to respondents. In order to present a
comprehensible choice to respondents, the CV survey instrument must clearly specify
the nature of the environmental commodity they will receive if they choose not to
purchase what is offered and the modifications to that commodity, the access condi-
tions and the nature of payment(s) if they choose to purchase what is offered. The no-
purchase condition will be referred to as the default condition and all aspects of the
change in the commodity as the commodity change.
There is a basic problem in interpreting the valuation question (Q11), primarily
associated with an ambiguity in the problems that can arise from the contaminated
water. In the base scenario, forty percent of the water used by the community was
described as coming from groundwater contaminated as a result of a leaking public
landfill. In the discussion of the context for the commodity, contamination increased the
risk of dying from cancer by about ten additional deaths per million per year among
people who drink the water. An event with comparable risk was described for respon-
dents.6 However, it is subsequently stated that the water must not be used for drinking
and cooking. Thus, individuals may not perceive health risk to be a feature of the
default conditions of water supply and risk perception may not enter into the individual's
decision process. In this case the only implication of contamination is the possibility of
water shortage.
In fact, it is not clear what respondents view as the default scenario for their
valuation. That is, in order to place a value on the primary scenario of complete
cleanup, respondents must clearly understand the reference point for their valuation,
that is, the condition of their water supply if complete cleanup did not occur. This is
important because acceptance/rejection of a proposed clean-up plan relative to a
default option characterizes the commodity purchased. The scenario could be
interpreted as describing an increased risk and the prospect that a substantial fraction
of their water supply would be contaminated. But since the scenario also specified that
it could not be used for drinking or cooking, it is not clear that those answering this
question would perceive any risk as being present. Furthermore no information was
provided on alternative supplies or response patterns. Other options are described
before the contingent valuation scenario. But there is no statement about whether
these alternatives would be able to be used for the particular groundwater
The term risk is used here to mean the lifetime probability of death.
15
-------
contamination being valued. Under the most optimistic interpretation of questions Q6
through Q10, individuals might have selected one of four alternative base cases to
define the default scenario in considering their valuation of complete cleanup. The
default scenario constitutes the reference point for their valuation estimate. The
difficulty is that we do not know which of these alternatives (if any) would function as
the method of choice for any particular respondent would choose in the event that they
did not select complete cleanup.
In the base survey instrument, the respondent must somehow judge whether the
uncontaminated 60% of the water supply will be adequate. There seems to be an
implicit assumption that the community must find an additional water source. This
implication appears to arise from Q10 and is not stated elsewhere. If the respondent
perceives a water shortage, all that is being asked is the demand for water. Of course,
this interpretation makes the further assumption that respondents have accepted the
conclusion offered in the instrument that risk has been precluded by the water policy in
effect.
Respondents' satisfaction ratings for each of the alternative approaches (i.e.,
containment, public treatment, home treatment, or water rationing) do not help resolve
the ambiguities because they do not provide a complete description of the adjustments
to their values associated with each of these reference points. Versions C and D of the
questionnaire allow comparison of public treatment with complete treatment, and
Version D elicits valuation adjustments in response to varying percentages of domestic
water supply contamination. Nonetheless, this would not allow us to completely
decompose (by scenario comparisons) the values that were offered for anyone who did
not receive one of these questionnaires.
The method of eliciting valuation estimates from some versions of the question-
naire as a percentage of the complete cleanup valuation caused some concern. When
individuals are offered changes in the scenario, they are offered adjustments to the
base case (i.e. a complete cleanup scenario) with specific percentages of value offered
as the responses. Where we have a priori expectations that the values given should be
larger, the percentages are scaled at a larger rate; where there are a priori
expectations that the values should be smaller, the percentages are scaled over a
smaller range. This formatting of the questions preconditions responses that individu-
als can give. The literature provides no basis for judging the reliability of these
percentage adjustments. Moreover, there does not appear to have been a test of
using alternatives to determine whether the formatting influenced the plausible
16
-------
response.
3.3.2.2 Embedding
Another source of concern is the question associated with disembedding.
Embedding is a term that has had a variety of interpretations in the economics
literature, with widespread use of the concept generally associated with research
reported by Kahneman and Knetsch (1992). In the context of this report, we have
interpreted embedding as a situation where survey respondents report values for
commodities that are more inclusive than what is desired by those analysts or policy
makers intending to use their responses. This can arise because respondents interpret
the commodity differently from the analyst or because they assume that other commodi-
ties would be provided along with the one that has been offered (even though the
survey instrument does not imply this to be the case). The McClelland et al.
disembedding question (Q12) asks individuals to reconsider the dollar amount they
stated (in response to the preceding question, Q 11) they would be willing to pay for
complete groundwater cleanup and asks them to characterize this amount as: (a) "Just
for the stated groundwater program;" (b) "somewhat for the groundwater program and
somewhat a general contribution to all environmental causes;" (c) "basically a contri-
bution to all environmental or other worthwhile causes;" or (d) "other. " In question 13,
a percent of a dollar amount is requested. If individuals are asked in question 11 to
describe the most they would be willing to pay each month on top of their current water
bill for each of the next ten years for complete groundwater cleanup program, the
Committee wondered why they would immediately thereafter be willing to admit that this
response was actually for other things. It would be difficult to interpret how any
household would respond to this question if they did not know it in advance, i.e. if the
questions were conducted at an in-person interview or a telephone interview where
respondents do not know what is coming next in the question sequence. When these
questions are presented in a mailed questionnaire (where it is assumed that respon-
dents can read ahead), the response to this type of question is even more difficult to
interpret and there is no basis in the literature for assessing this issue. Thus the
methods used in this study to adjust for embedding have not been evaluated in the
literature.
3.3.2.3 Partitioning of Total Values
One of the objectives of the study was to estimate the non-use values of
cleaning up groundwater contamination. These estimates were to be used in the
benefit assessment required for the Regulatory Impact Analysis of RCRA corrective
17
-------
action rule. The Committee concludes that the report does not provide evidence that
respondents adequately distinguished the use and non-use values for cleaning up
contaminated groundwater. Three methods were used to estimate non-use values:
direct questions requiring allocation of total bids over composite categories; compari-
son of "willingness to pay" (WTP) responses for different commodity scenarios; and
extrapolation by fitting an assumed functional form to WTP bids for different levels of
water shortage.
The first method was implemented through Question 14 which asked respon-
dents to decompose their complete groundwater cleanup bid into motivational catego-
ries. A limited number of investigators have used this approach to isolate use and
non-use values, but there is no evidence in the literature about the method's perfor-
mance.7 The maintained hypothesis that respondents to CV questions can decompose
their bids into constituent parts is difficult to accept and there is little evidence to
support its use. For example, it seems reasonable to suggest that people would have
difficulty decomposing their willingness to pay for something as familiar as an ice cream
sundae into percentages they would associate with texture, temperature, looks, taste,
etc. At the minimum we would expect that this type of detailed information would need
to be elicited very carefully - in effect "coaxed" from them in a very detailed but logical
way. The reliability of the second method is questionable due to confusion over the
definition of the commodity and default scenarios, as discussed above. With regard to
the third method, we have no past experience with the relationship between this
method and the tasks requested of respondents, nor any reason to believe that the
intercept of a fitted function can be interpreted as a non-use value. Finally, there is
little basis provided in the study for gauging the transferability of such results from
other situations requiring the estimation of non-use values for cleaning up contaminat-
ed groundwater resources .
By criticizing the techniques used in this study to separate use from non-use
values, the Committee does not intend to address the broader issue of the validity of
that distinction. Some members of the Committee questioned whether it is appropriate
to ask respondents to partition their willingness to pay into discrete elements that were
associated with components of those respondents' total values. This should be a
matter for further research.
See Boyle [1993] for discussion and evaluation.
18
-------
3.3.2.4 Consistency with Mitchell-Carson Existence Value Study
The Committee found some inconsistency between the lessons learned from the
earlier Mitchell-Carson evaluation of existence values as reported by Mitchell and the
McClelland etal. interpretation of these results. For example, McClelland etal.
summarize the objectives of their study:
The aim of our current study is to estimate non-use values for groundwa-
ter cleanup. This commodity, of great interest to the U.S. EPA, also
appeared to be ideal for a methodology study since in early development
work undertaken for U. S. EPA by Mitchell and Carson, it was apparent
that (1) people were generally poorly informed about groundwater con-
tamination, and (2) people resisted non-use scenarios used for evaluation
in which groundwater was preserved but never used. In other words, the
scenario was rejected by respondents. Delighted with our exotic
commodity, groundwater cleanup, our strategy was to apply two new
methods in designing this survey instrument (page 22).
By contrast, the Mitchell-Carson report concluded somewhat differently from
what the authors indicated. It notes:
We believe it is possible to use the contingent valuation method to obtain
a credible dollar measure of the existence values of groundwater by using
a scenario of the type described in this chapter. The device of having
people evaluate hypothetical GNN (groundwater not needed for human
use) groundwater (aquifer) that can be plausibly isolated from other aqui-
fers and which lies at a great distance from those who are being inter-
viewed should effectively eliminate any use values. The concrete barrier
plan promises to provide a credible protective option for the GNN aquifers
so that those who choose it will be stating how much they would be willing
to pay to preserve the aquifer defined as GNN from contamination. These
containment features should minimize any influence on people's
willingness to pay that might result if they continue to believe that con-
tamination in groundwater travels at a much higher velocity than is
actually the case. . . . Given the difficulty of convincing people that the
aquifer will never be needed for human use, a portion of the values will
include the utility people get from vicarious protection. The vicarious
protection values will be minimized by the scenario features that are
intended to protect others from inadvertently using groundwater in the
19
-------
basin. . . . There is no valid way to obtain separate measures of the
several types of existence value in this study. First, it is too difficult to
overcome people's belief about future use by others to design a scenario
that would only capture stewardship values. Likewise, we see no way to
design a scenario that would only capture bequest or inherent values for
groundwater. Second, it is cognitively unrealistic to ask respondents to
state what proportion of the total value they ascribe to each of these three
types of existence values. However, it will be possible to assess in a
qualitative fashion the degree to which respondents are influenced by
these several types of value by using the follow-up motivational questions
(pp 83-85, emphasis added).
Both studies identify difficulties in separating use and non-use values. In part
because of the Mitchell-Carson work, and in part because of their own pre-test and
verbal protocol analysis, it was clear to the principal investigators that it would be
extraordinarily difficult to explain the nature of groundwater resources that would not
have any foreseeable uses to them or other people, to respondents. Mitchell and
Carson recommended a separation by distance as well as an identification of other
aquifers to try to deal with this problem. McClelland et al. adopted a different frame-
work. Concern about substitutes in the form of alternative aquifers and about the
importance of describing plans for protecting groundwater that would be believable to
respondents did not appear to influence the way in which McClelland et al. structured
their scenarios. The allocation format used in the McClelland etal. report does not
conform to the follow-up motivational questions as described by Mitchell and Carson.
3.3.2.5 Box-Cox Estimation and Treatment of Zero Bids
The Box-Cox regression methods analyzed what is referred to as the "reduced
willingness to pay," the willingness to pay bid for complete groundwater cleanup
adjusted by the reported disembedding percentage. The results reported by
McClelland-SchuIze in the second of the Committee meetings suggest that the Box-Cox
modelling approach used to deal with a skewed bid distribution was quite sensitive to
the authors' treatment of zero bids. The magnitude of the Box-Cox parameter exhibited
substantial variation with the substitutions imposed for the zero values of the
willingness to pay bids. Despite the sensitivity to zero values, there was no discussion
of the treatment of zero bids or scenario rejection in the present report. Moreover, the
Box-Cox transformation is only one of a number of ways of dealing with skewed error
distributions and no comparative evaluation of the methods was undertaken. Thus, the
Committee concludes (in response to the third question of the Charge) that the report
20
-------
does not provide clear-cut evidence that the Box-Cox procedure adequately addresses
the issues posed by large bids, zero bids, and scenario rejection. In addition, an
evaluation of the zero responses, non-responses, large bids, and non-responses to the
valuation scenarios in total are essential for judging the reliability of the analysis. The
existing contingent valuation literature offers an array of
possibilities for dealing with each of these issues, and no single method has been
identified as preferred. In view of this work, it seems reasonable to suggest that
comparative evaluations of the different methods for treating outlying observations,
zero responses, and non-responses would be important for gauging the comparative
performance of the contingent valuation scenarios offered in this study.
The Committee recognizes that there are a variety of statistical methods that can
be used to address large bids, zero bids, and other anomalies. For example, one might
employ some robust regression approach. Using a statistical procedure to address
extreme observations does not, however, solve the more fundamental problem of why
such extreme responses arose. Do these responses reflect legitimate heterogeneity in
valuations, or do they reflect a failure by respondents to understand the survey and
give meaningful responses? Choice of the appropriate statistical solution depends
largely on the factors generating these extreme responses. As a result, the Committee
views these extreme bids as a matter of continuing concern that would warrant further
research in future studies.
3.3.2.6 Statistical Analysis of Pre-test and Final Samples
The design of the McClelland et al. survey instrument was intended to include a
set of activities that would evaluate the amount of information that respondents needed
to value groundwater cleanup. These activities included the verbal protocol analyses
and pretests conducted in the initial stage of the research to permit judgments about
the final success of the form of the survey instruments in communicating the informa-
tion people required to decide about a groundwater cleanup policy offered to them.
Because the pretests played a key role in the survey instrument's design, the Commit-
tee considered the nature of the empirical analyses undertaken to develop the
conclusions on the instrument used in the final survey.
The empirical analysis included summary statistics, primarily means and
standard deviations, and some frequency plots of responses to the willingness to pay
questions against a few of the attitudinal variables for the pre-test and full samples
together. There were also comparable statistics for a variety of subsets of the full
sample. There was little multivariate analysis of the other willingness to pay questions.
21
-------
While there is a much smaller sample associated with each of these variations in the
questions, a comparative evaluation of multivariate functions estimated for each of the
willingness to pay responses using the final sample would contribute information to
help assess the reliability of the valuation estimates. It would provide indirect evidence
on the effects of changes in the scope of the commodity; for example, larger
percentage shortfalls would be expected to be associated with greater use-related
motives, where scenario changes from complete cleanup to containment or public
treatment would be regarded as smaller amounts of the "cleanup" commodity.
Table 7.5 in the McClelland etal. report provides suggestive evidence, both in
the full sample and the regression sample, of a response to variations in the commodity
design . However, it is not clear whether these differences are associated with the
actual characteristics of the commodities or with particular features of the sub-sample
(i.e. lower income level, socio-economic status, or other factors), so that these factors
would need to be taken into account in appraising whether these were consistent
differences in the valuation responses across scenarios. The McClelland et al.
evaluation of the effects of changes in the commodity (across the versions of the
survey questionnaire) did not independently evaluate the change in respondents'
willingness to pay to the features of the commodities. It was conditioned on accepting
the Box-Cox model (and its predictions) as a maintained hypothesis. These predictions
were then used, together with the percentage responses to commodity changes elicited
from each respondent based on which version of the questionnaire they received. No
attempt was made to model the adjustments and to incorporate this in a broader
description of the adjusted valuations. The Committee believes that such models, both
for the base case valuation and for other valuation responses that relate them to the
characteristics of the respondent, would provide indirect evidence of the reliability of
the survey methods.
Concerns were also raised with respect to the responsibility variable (Q 15).
While the briefing from the principal investigators suggested that the results are not
markedly changed by removing this variable from the willingness to pay models, the
performance of the variable as a gauge of acceptance of the scenario and its relation-
ship to other characteristics of respondents could be quite important.
More complete treatment of the sub-samples that include a) individuals living in
areas with sites on the National Priority List; b) individuals who do not receive water
bills - i.e. individuals with private wells; and c) individuals with varying experience with
groundwater contamination, would offer opportunities for additional insights into the
reliability of the responses. Because this work was not undertaken as part of the
22
-------
analysis of the survey and remains to be done, it is difficult to reach an overall
conclusion on the reliability of the willingness to pay responses interpreted as total
values for groundwater cleanup. Preliminary analysis undertaken by Committee
members suggests that there may be further insights from this type of empirical
evaluation.
3.3.3 Responses to Specific Charges
The Committee's response to each of the specific questions associated with the
charge is as follows:
a) We do not believe that the pre-testing and survey design techniques used
in this study offer sufficient evidence to ensure that a well-defined
groundwater commodity was understood properly by all the respondents.
The results of these analyses support the conclusion that groundwater
issues were important to respondents. However, the sample sizes for the
pre-tests were not large enough to permit multivariate analysis that would
allow evaluation of some of the judgments made by the principal
investigators. In particular, the process of reducing the information set
that was provided to individuals, the decisions on eliminating the risk
information, the selection of a valuation payment card over alternative
methods, and the definition of the commodity itself were not sufficiently
tested in the pre-test and verbal protocol analysis in a way that would
enable us to judge whether these were the most understandable ap-
proaches for dealing with these issues.
b) The Committee does not believe that any of the three possible methods
for estimating the total willingness to pay for non-use or passive use
values can be established as most preferred on the basis of the evidence
that is currently available in the literature. This is not to suggest that the
methods are incorrect or that they might not ultimately provide a basis for
estimating non-use values in some cases. The evidence to date however
provides no basis for judging them. They should be regarded as
innovative research techniques and not verified methods for estimating
non-use values from survey responses.
The Committee remains skeptical of the method of asking respondents to
decompose reported total values into use versus non-use or various
motivational categories. These procedures have not been verified in the
23
-------
literature. There fore we recommend the principal focus of the analysis
be on the total willingness to pay estimates for the complete containment
scenario only.
c) The Box-Cox econometric transformation as applied in this study is one of
several methods for dealing with large bids and outliers. The estimates
appear sensitive to the treatment of zero bids. Comparative evaluations
of other approaches for dealing with the skewness in the valuation
responses should be undertaken before accepting the Box-Cox results.
In the absence of such comparative evaluations and detailed analysis of
all the possible outcomes of the valuation questions, it is not possible to
judge whether the Box-Cox econometric estimates alone provide an
acceptable and defensible method for dealing with the scenarios and the
large bids associated with them.
d) We do not believe that the approaches for treating embedding, dealing
with scenario rejection, and reflecting the potential effects of non-bids can
be assessed for their reliability on the basis of the information provided in
the report. The procedures described in the McClelland etal. report are
research innovations that have not been evaluated in the literature and
were not evaluated as part of their study. Further, multivariate modelling
of determinants of the disembedding responses and of the factors
influencing whether individuals provided no bids, zero bids, or large bids
would be essential to answer the fourth question in the EPA charge.
Because this information was not provided as part of the report, it is
impossible for the Committee to make the judgment required to address
question four of the charge.
e) The valuation responses in the survey for non-use value have not been
demonstrated to be accurate and reliable enough for the EPA to estimate
the non-use values for groundwater contamination. It is not clear that any
of the three decomposition methods provides a basis for estimating non-
use values accurately and in a reproducible manner. The literature does
not provide specific documentation indicating that these methods are
capable of separating the total values, especially for a situation where the
commodity definition involves water rationing, risk, and groundwater
protection as part of the complete containment scenario. Disaggregating
the components of the total value on the basis of motives alone would not
necessarily assure that the values are exclusively associated with non-
24
-------
use values or that they could be transferred to another hypothetical
situation with distinctly different features.
Improving the estimates of non-use values will require a composite
scenario that includes the recommendations earlier provided by Mitchell-
Carson along with the insights provided in this study. That is, it will
require an evaluation that attempts to control the respondents' percep-
tions of uses for the groundwater by identifying other sources for their
water and localizing the effects of contamination to deposits that were not
now (or not thought to be) likely resources for future use. Because these
alternatives were part of what was recommended from the Mitchell-
Carson report and were identified as possible alternatives in the Boyle
(1993) summary of existing literature, it seems they should have been
part of the experimental design.
25
-------
4 CONCLUSIONS AND RECOMMENDA TIONS
4.1 Summary
It is possible to evaluate the report from a number of different perspectives.
From the perspective of contributions to methodology and the practice of contingent
valuation analysis, there is little doubt that this report represents a substantive
contribution, extending our understanding of the issues associated with contingent
valuation estimation of non-market values. Its progressive development of information
treatments, innovative experimental design, and attention to comparative analysis of
the values provided by alternative methods of separating use from non-use or passive
use values should be treated as new research insights to evolving literature on
contingent valuation. However, this is not the criterion that the Committee was asked
to use in evaluating the report. In many respects, the Committee feels that the
problems in using the study results to meet the needs of the RIA charge arise from
requirements imposed on the research - the need for a separate estimate of non-use
value and for a method that abstracted from the specific features of local conditions.
After reviewing the full record, the Committee feels that this strategic decision on EPA's
part is the cause of many of the study's problems as well as the difficulties of imple-
menting the research findings in the RIA context. The approaches taken to deal with
these requirements have no basis in the theory of non-market valuation, nor precedent
in practice.
Finally, because of our criticisms of the definition of the commodity being valued
and our concerns about whether respondents fully understood what was being valued,
we are unable to make a judgement as to whether the estimates of total value for
groundwater cleanup are too high or too low. Thus, the Committee concludes the study
does not offer evidence that the survey has in fact developed estimates of non-use (or
passive use) values. Moreover, there is no basis for believing that estimates of the
representative household's non-use values (or the total values) could be developed for
groundwater cleanup independent of the particular circumstances associated with the
groundwater that has been contaminated.
As noted below, this conclusion does not imply that further research with the
survey would not yield valuable insights into how households evaluate groundwater
cleanup.
26
-------
4.2 Recommended Research
The Committee would like to commend EPA for supporting innovative policy
research of this type. Environmental commodities have distinctive characteristics in
that they are often not the object of explicit or implicit market transactions. In this
instance, there are no readily available economic data to enable economists to
precisely assess the value of removing groundwater contamination to those who will
not be using the potentially contaminated water. To obtain such values, original
economic research is needed. Research frequently will not completely resolve all of
the uncertainties with respect to benefit assessment in these controversial areas.
However, beginning the process of exploring the appropriate value to attach to these
problematic benefit components should be a high priority for future EPA funding.
As the above comments indicated, the Committee believes that there are a
number of ways in which this specific research effort could be enhanced. For example,
providing respondents with a better understanding of the commodity and ensuring that
the responses isolate the non-use value of environmental damage are clearly of
paramount concern. Indeed, had such a program existed, prior to the McClelland et al.
report, it would have been possible to provide more specific answers to the
methodological questions posed in the Charge to this Committee.
The research issues associated with developing aggregate estimates of the total
value households place on groundwater cleanup are much too detailed to discuss
completely in this report. Five general areas that are identified from the McClelland et
al. report and seem especially relevant for EPA's activities associated with the
corrective action RIA will be discussed. It is, of course, also true that further
multivariate analysis with the survey data from the McClelland et al. survey along with
research investigating the survey mode, role of substitute groundwater resources, and
other features of the commodity described in their survey instrument would be very
desirable and seems likely to enhance our ability to interpret the findings from the
study.
The first of these has received very limited research attention. It concerns the
extent of the market for environmental commodities like groundwater. For the most
part, economic research has focused on developing estimates of the typical or
representative household values for specific commodities and not on the criteria for
determining how many households can be expected to hold these values. The
McClelland, Schulze, Lazo report (1993) includes some very preliminary and therefore
limited discussion of these issues. Addressing this question is fundamental to the task
27
-------
of measuring aggregate values for the cleanup of groundwater resources and may well
have a much greater quantitative impact on these estimates than would refinements in
estimates of the representative household's values for changes in some environmental
commodities.8
The second area of research would involve a detailed study of the sensitivity of
CV outcomes to the survey methods used - whether telephone, in-person, or mail
surveys. The evidence provided to the Committee by Professor Dillman (letter, March
31, 1993) when compared with the recommendations of the NOAA Panel suggested
that there is not a consensus in the profession about the superiority of in-person
interviews over mail surveys. More research on this topic is clearly warranted. Such
research would consider not only the implications of the method used to implement the
survey but also the elicitation procedure and the role of information in the design of the
survey questions. It would require parallel efforts conducting the same valuation
exercise with at least two, and preferably all three, of the methods available. It would
be important not to compromise the evaluation by requiring that it provide "definitive"
answers for an ongoing policy issue. Rather it would be important to have the
commodity offered be a "real one" that is of some significance to the households that
are interviewed, while the goal of the evaluation be clearly identified as generic
research, not estimates to be used in a specific subsequent RIA.
A third area of research would address issues associated with defining changes
in environmental commodities. This research is central to understanding whether the
contingent valuation responses are internally consistent. The recommended research
would involve developing methods and protocols for practice that could be used to
understand how environmental commodities are best measured and how those
measures relate to the descriptions offered in the framing of contingent valuation
questions. Consideration of the differences that might arise in circumstances when
total values are dominated by use-related components in comparison with those when
non-use related services dominate should be an important component of this design.
Equally important, to the extent that consideration of risk is an important element in the
ongoing activity of EPA policy evaluations, some component of perceived risk should
be incorporated as an additional factor in understanding commodity definition for CV
work.
Given the extremely limited nature of the available research on the value of groundwater cleanup, the Committee feels that in this
case both types of research are exceptionally important.
28
-------
The fourth area of research arises from an implicit strategy for benefit transfer
that influenced the design of the CV study. Based on the introduction to the study (as
well as earlier interim study reports and EPA staff briefings), it is clear that the study
was intended to be used as a basis for estimating national non-use values that would
be combined with separately computed estimates of components of the use values
resulting from groundwater cleanup. This strategy implicitly accepts the validity of
separating the use from non-use value. Before this approach can be adopted, there is
a clear need for research on whether respondents can understand this task in a
meaningful way.
The final high priority area for research concerns the desirability of evaluating
the potential of these methods developed in marketing research for calibration of CV
estimates and the development of methods for gauging the potential implications of
using a hypothetical setting to elicit valuations in relationship to real choices. Is
calibration a meaningful concept in the relationship to contingent valuation responses?
What would be the mechanism for developing calibration adjustments? These are
issues that are important to the ongoing development of the contingent valuation
method. Research evaluating whether or not contingent valuation results can be
transferred from one setting to others is also critical. The Committee believes this
should be a central component of the future research that EPA undertakes in support of
its policy evaluation.
This research should not be regarded as peripheral to policy evaluations, but
instead central to the ability of the EEAC to respond to the specific types of questions
posed by EPA staff about this study and the associated RIA analysis. The Committee
assumes that there will be a continuing need to evaluate analyses intended to appraise
household's willingness to pay for improvements in environmental commodities. There
is simply not sufficient information on the research shelf to address in a meaningful way
many of the questions that have been posed to the EEAC about contingent valuation
and its use. EPA should begin to sponsor research required to evaluate the methods
used in its policy evaluations and to develop a greater set of experience in valuing
changes to important economic resources.
29
-------
5 REFERENCES
Boyle, K. J. A Review of Contingent Valuation Studies of the Benefits of Groundwater
Protection (report to U.S. Environmental Protection Agency), Research Triangle
Institute, April 1993).
Dillman, D.A. Letter to Alan Carlin, EPA, March 31, 1993
Kahneman, D. and Knetsch, J. K. Valuing Public Goods: The Purchase of Moral
Satisfaction," Journal of Environmental Economics and Management, Vol 22,
January 1992, 57-70.
Kneese, A. and Smith, V.K. Memorandum to R.C. Mitchell, March 23, 1993.
McClelland, G.H., Schulze, W.D. Lazo, J.K., Waldman, D.M., Doyle, J.K., Elliott, S.R.
and Irwin, J.R. Methods for Measuring Non-Use Values: A Contingent Valuation
Study of Groundwater Cleanup (Draft report to the U.S. EPA), University of
Colorado, Center for Economic Analysis, October, 1992.
McClelland, G.H., Schulze, W.D. and Lazo, J.K. Additional Explication of Methods for
Measuring Non-Use Values: A Contingent Valuation Study of Groundwater
Cleanup, Memorandum to the SAB/Environmental Economics Advisory Commit-
tee. Center for Economic Analysis, University of Colorado, June 29, 1993.
Mitchell, R.C. and Carson, R.T. Existence Values for Groundwater Protection. Draft
Final Report to U.S. Environmental Protection Agency (Washington, D.C.:
Resources for the Future, May 1989).
National Oceanographic and Atmospheric Administration (NOAA), Report of the NOAA
Panel on Contingent Valuation, Federal Register, January 15, 1993, Vol. 58,
#10, pp. 4601-4614.-
R-1
------- |