UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON D.C. 20460
OFFICE OF THE ADMINISTRATOR
SCIENCE ADVISORY BOARD
June 8, 2007
EPA-COUNCIL-07-002
Honorable Stephen L. Johnson
Administrator
U.S. Environmental Protection Agency
1200 Pennsylvania Avenue, N.W.
Washington, DC 20460
Subject: Benefits and Costs of Clean Air Act - Direct Costs and Uncertainly Analysis
Dear Administrator Johnson:
EPA's Office of Air and Radiation (OAR) is developing the Agency's "Benefits and
Costs of the Clean Air Act 1990 - 2020: EPA's Second Prospective Analysis." This study is
designed to estimate the net health, welfare, ecological and economic benefits of all 1990
Clean Air Act Amendment programs. As part of this effort, OAR has recently submitted to
the Advisory Council on Clean Air Compliance Analysis (Council) its draft Direct Cost
Report as well as its draft Uncertainty Analysis Plan. In February 2007, OAR requested input
from the Council on these draft products. The Council met in a face-to-face meeting on
March 15-16, 2007, to discuss the OAR's charge questions related to these reports. The
Council generally endorses OAR's approach and results to date.
With respect to OAR's Direct Cost Report, the Council notes that two types of cost
estimates are required - partial equilibrium social costs that are frequently reported as the cost
of regulation, and private costs that are input to the computable general equilibrium (CGE)
model OAR will use to estimate general equilibrium costs and other effects of the Clean Air
Act Amendment programs. The Council encourages OAR to distinguish and report separately
the two types of costs (when they differ significantly).
With respect to OAR's Uncertainty Analysis Plan, the Council commends OAR for its
intention to use several types of graphical display to communicate the uncertainty of its
estimates. The Council recommends that OAR be vigilant in reporting what uncertainties are
incorporated in a probability distribution and on what other parameter values these
distributions are conditioned (e.g., uncertainties in health effects and valuation may be
represented as probability distributions conditional on an assumed economic growth
-------
scenario). The Council cautions OAR against presenting scenarios or sensitivity analysis
without some statement of the probabilities associated with the alternative scenarios or
parameter values. Readers will understandably conclude that the values presented encompass
a relevant range of possibilities, and communication should be improved if OAR provides
some guidance concerning where the presented values lie in an overall probability
distribution.
Detailed recommendations are included in the enclosed Advisory. On behalf of the
entire Council, we appreciate this opportunity to provide timely advice to the Agency. We
hope these comments are helpful to the Office of Air and Radiation as it proceeds with this
important work.
Sincerely,
/Signed/
James K. Hammitt, Chair
Advisory Council on Clean Air Compliance Analysis
Enclosure
11
-------
NOTICE
This report has been written as part of the activities of the U.S. Environmental
Protection Agency's Advisory Council on Clean Air Compliance Analysis (Council), a
federal advisory committee administratively located under the EPA Science Advisory Board
(SAB) Staff Office. The Council is chartered to provide extramural scientific information and
advice to the Administrator and other officials of the EPA. The Council is structured to
provide balanced, expert assessment of scientific matters related to issue and problems facing
the Agency. This report has not been reviewed for approval by the Agency and, hence, the
contents of this report do not necessarily represent the views and policies of the EPA, nor of
other agencies in the Executive Branch of the Federal government, nor does mention of trade
names or commercial products constitute a recommendation for use. Council reports are
posted on the SAB Web site at: http://www.epa.gov/sab.
in
-------
U.S. Environmental Protection Agency
Advisory Council on Clean Air Compliance Analysis
(COUNCIL)
CHAIR
Dr. James K. Hammitt, Professor of Economics and Decision Sciences, Harvard Center for
Risk Analysis, Harvard University, Boston, MA
COUNCIL MEMBERS
Dr. David T. Allen, Gertz Regents Professor in Chemical Engineering, Department of
Chemical Engineering, University of Texas, Austin, TX
Dr. Dallas Burtraw, Senior Fellow, Resources for the Future, Washington, DC
Dr. Trudy Ann Cameron, Raymond F. Mikesell Professor of Environmental and Resource
Economics, Department of Economics, University of Oregon, Eugene, OR
Dr. Charles T. Driscoll, Jr., Professor, Department of Civil and Environmental Engineering,
College of Engineering and Computer Science, Syracuse University, Syracuse, NY
Dr. Shelby Gerking, Professor of Economics, Department of Economics, College of
Business Administration, University of Central Florida, Orlando, FL
Dr. Wayne Gray, Professor, Department of Economics, Clark University, Worcester, MA
Dr. F. Reed Johnson, Senior Fellow and Principal Economist, RTI Health Solutions,
Research Triangle Institute, Research Triangle Park, NC
Dr. Katherine Kiel, Associate Professor, Department of Economics, College of the Holy
Cross, Worcester, MA
Dr. Nino Kuenzli, Associate Professor, Department of Preventive Medicine, Keck School of
Medicine, University of Southern California, Los Angeles, CA
Dr. Virginia McConnell, Senior Fellow and Professor of Economics, Resources for the
Future, Washington, DC
Dr. Bart Ostro, Chief, Air Pollution Epidemiology Unit, Office of Environmental Health
Hazard Assessment, California Environmental Protection Agency, Oakland, CA
IV
-------
Dr. David Popp, Associate Professor of Public Administration, Center for Policy Research,
The Maxwell School, Syracuse University, Syracuse, NY
Dr. Chris Walcek, Senior Research Scientist, Atmospheric Sciences Research Center, State
University of New York, Albany, NY
SCIENCE ADVISORY BOARD STAFF
Dr. Holly Stallworth, Designated Federal Officer, Environmental Protection Agency,
Science Advisory Board Staff Office, Washington, DC
-------
Charge Questions from the Office of Air and Radiation
for the Advisory Council on Clean Air Compliance Analysis
Meeting of March 15-16, 2007
1. Direct Cost Report - General Review. EPA requests that the Council review the draft
Section 812 Second Prospective Study direct cost report. Consistent with the statutory
language defining the role of the Council in reviewing the 812 studies, EPA respectfully
submits the following general charge questions:
a. Does the Council support the data choices made by the 812 Project Team for the
development of the draft direct cost report? If not, are there alternative data sets the
Council recommends should be applied instead?
b. Does the Council support the methodological choices made for analyzing those data
and developing the direct cost estimates for the relevant scenarios? If not, are there
alternative methodologies the Council recommends should be applied instead?
c. What advice does the Council have for the EPA Administrator with respect to the
validity, reliability, and utility of the draft direct cost report and the analytical results
presented therein? If the validity, reliability, and/or utility of the draft direct cost
report could be improved, what specific steps does the Council recommend the 812
Project Team pursue to accomplish these improvements?
2. Direct Cost Report - Specific Issues. The general charge question #1 covers any and all
aspects of the draft direct cost report which the Council might consider appropriate to
address in its review. In conducting this review, EPA also respectfully requests that the
Council consider the following particular issues during its review. These particular topic
areas represent specific analytical choices or outcomes for which the 812 Project Team is
especially interested in Council advice.
a. Unidentified Measures. Currently available air pollution control technologies do not
appear to be sufficient to achieve full attainment with the current PM and/or ozone
NAAQS. The first prospective 812 analysis adopted an approach where additional,
"unidentified measures" were built into the with-CAAA scenario to achieve the
future emissions reductions needed to progress toward full attainment. The costs of
those unidentified measures were capped at $10,000 per ton. The current draft cost
report again applies a fixed $10,000 per ton cap on precursor emissions. However,
other options have been identified by the 812 Project Team for estimating the costs
of unidentified measures. Three basic options (plus some variants) are described in a
white paper accompanying the draft direct cost report. The Agency requests that the
Council provide advice regarding the relative merits of these alternative approaches.
In addition, EPA respectfully requests that the Council consider the issue of reliance
on unidentified measures more broadly, and provide advice regarding the
appropriateness of providing qualitative discussion, caveats in the characterization of
results, and/or uncertainty analysis associated with the projection of future control
technologies and costs.
-------
b. Transferability of Learning Effects. The draft cost report applies learning effects
adjustments to a limited number of source categories and control technologies where
empirical data are available to specify a learning effect rate. This limited application
is consistent with prior Council advice to replace a generic 80% learning effect rate
with more source-specific rates based on available empirical data. Two of the key
applications of learning rates in the current draft cost report are for Selective
Catalytic Reduction (SCR) and Selective Non-Catalytic Reduction (SNCR) at
Electric Generating Units (EGUs). However, SCR and SNCR technologies are also
applied to some non-EGU sources, particularly industrial boilers. The current draft
cost report, however, does not apply learning effect adjustments to non-EGU
applications of SCR and SNCR. The Agency therefore requests that the Council
consider providing advice regarding the appropriateness of transferring learning
effects adjustments for a given control technology from one source category to
another.
c. Discount Rate Selection and Implementation. The current draft cost report uses a 5
percent social discount rate for the primary analysis, consistent with prior Council
advice to adopt a central discount rate value for the primary analysis, supplemented
by a sensitivity analysis using the 3 percent and 7 percent rates prescribed by
prevailing guidelines. The Agency requests that the Council consider providing
advice pertaining to the reasonableness of the choice of a 5 percent rate for the
primary analysis, as well as the appropriateness of the 812 Project Team's
application of this rate in the derivation of sector-specific direct cost estimates.
d. Treatment of Manufacturer Profit in Social Costing. Some of the direct compliance
cost estimates incorporated in the draft direct cost report are based on projected
changes in manufacturer cost (including production costs, distribution costs,
overhead, etc) and some estimates are based on projected changes in retail prices for
consumers. The essential difference between these two costing approaches is the
inclusion or exclusion of markups for producer profit. The Agency requests that the
Council review the white paper on the cost markup issue and consider providing
advice pertaining to the appropriate treatment of producer profit in estimates of the
social cost of compliance.
3. Uncertainty Analysis Plan. Consistent with advice from several 812 Council advisories
and the 2002 NAS report, the 812 Project Team has developed an expanded uncertainty
analysis plan, which the team has described in a white paper. This updated plan
represents a significant expansion of the original uncertainty analysis plans contained in
the May 2003 analytical blueprint. Consistent with the statutory language defining the
role of the Council in reviewing the 812 studies, EPA respectfully submits the following
charge questions:
a. Does the Council support the data choices made by the 812 Project Team for the
development of the 812 study uncertainty analysis, as described in the uncertainty
analysis plan white paper? If not, are there alternative data sets the Council
recommends should be applied instead?
-------
Does the Council support the methodological choices made for analyzing uncertainty
in the 812 analysis, as described in the uncertainty analysis plan white paper? If not,
are there alternative methodologies the Council recommends should be applied
instead?
###
-------
Council Advisory
on
OAR's Direct Cost Report and Uncertainty Analysis Plan
The following comments constitute the Council's response to the charge questions submitted
by the Office of Air and Radiation in February of 2007 concerning its draft "Direct Cost Estimates
for the Clean Air Act Second Section 812 Prospective Analysis" (Direct Cost Report) and its draft
paper entitled "Estimating Uncertainty in the Second Prospective Analysis of the Benefits and Costs
of the Clean Air Act" (Uncertainty Plan). Dated February 2007, both of these draft papers are
interim products developed for the "Benefits and Costs of the Clean Air Act 1990 - 2020: EPA's
Second Prospective Analysis" (Second Prospective) and are posted at:
http://www.epa.gov/oar/sect812/prospective2.htmltfmar07.
Direct Cost Report - General Comments
Overall, the Council endorses the methods and data incorporated in the Direct Cost report.
We note one significant conceptual issue - the difference between conventional partial-equilibrium
social costs of regulation and the private costs needed as input to the computable general
equilibrium (CGE) model. This issue is discussed in response I.e., which is addressed first.
Afterward, the Council offers some comments on data and methodological choices in responses l.a.
and l.b.
Validity, reliability and utility of the report: Partial- and general-equilibrium cost estimates (See
charge question I.e.)
The Council notes that two types of costs are relevant to the analysis for the Second
Prospective Report. First, private costs are needed as input to the computable general equilibrium
(CGE) model that will be used to estimate general-equilibrium effects of Clean Air Act Amendment
(CAAA) regulations. Second, conventional partial-equilibrium estimates of social costs of
regulations are of interest for comparison with the First Prospective Report and other estimates of
the costs of environmental regulation. The two types of costs differ in how transfer payments, such
as taxes and firm profits, are incorporated. When there are significant differences between them, the
Direct Cost report should report both. For social welfare analysis, the social costs are of primary
interest and so, to minimize reader confusion, it may be appropriate to highlight the social costs and
report the private costs used as input to the CGE in an appendix.
In addition, the Council identifies a potential inconsistency in the analysis if the general-
equilibrium effects captured by the CGE model are important. The Direct Cost Report starts with
output predictions from the Annual Energy Outlook (AEO) model, calculates the magnitude of
emission reductions needed to meet a variety of regulatory constraints, and concludes with the
increases in production cost resulting from emission reductions. These cost increases are calculated
by industry and region and are used as input to the CGE model. The CGE model simulates the
effects of these costs on industry and regional output, which influence regional emissions and may
affect regional control costs.
-------
To illustrate, suppose that the Direct Cost Report finds that paper production in the South
becomes much more costly due to local regulatory constraints. The CGE model will predict
reductions in demand for and production of Southern paper, offset in part by increased production of
Northern paper and non-paper goods. If these general-equilibrium shifts in production are important,
the resulting distribution of industry production will diverge from that assumed in the AEO, which
should in principle feed back into the emissions constraints: less Southern paper production means
fewer emissions in the South, reducing the cost of meeting regulatory constraints there (for paper
and other industries); increased Northern paper production raises abatement costs in the North, again
for paper and non-paper industries. Note that the full impact of increased costs on production levels
may not be captured for several years in the CGE model, given lags in capital investment. Although
resource constraints may preclude a complete recalculation of the direct cost model using the CGE
output, the Council recommends that the Project Team compare the CGE and AEO outputs to
identify any industry-region cases that are especially sensitive to general-equilibrium effects.
Data choices (See charge question l.a.)
The Council generally supports the data and methodological choices made by the 812
Project Team. However, some of the assumptions about vehicle fleets and vehicle costs might be
looked at more closely.
The assumed failure rate for inspection and maintenance (I&M) programs appears to be a
constant 14 percent. There is evidence that failure rates are lower for vehicles with on-board
diagnostic (OBD) systems, particularly those meeting recent standards (OBD-II), and remain lower
over the life of the vehicle (see National Research Council 2001, Evaluating Vehicle Emissions
Inspection and Maintenance Programs, http://books.nap.edu/catalog.php?record_id=10133). Other
I&M cost assumptions (e.g., time in line, value of time) also appear to be on the high end. On the
other hand, the emissions reductions may also be overestimated. There is evidence that the
emissions reductions assumed for I&M programs in the Mobile model may be too large (NRC,
2001). Cost offsets due to fuel savings are estimated using cost per gallon of $1.11 (p. 3-5) and
$1.14 (p. 3-16), which seem low. Given high excise taxes, the difference between private and social
fuel costs may be significant.
With respect to estimating the cost of meeting future motor-vehicle emission standards, the
Project Team may consider evidence about the ex ante and ex post cost estimates for meeting new
car standards in the recent past. The California Air Resources Board estimates for meeting the low
emissions vehicle (LEV) standard showed that the ex post costs appear to be slightly lower than ex
ante predicted costs for smaller vehicles; for larger vehicles, the ex post costs were slightly higher
(Presentation by Tom Cackette, 1998, cited in National Research Council 2006, State and Federal
Standards for Mobile Source Emissions, http://books.nap.edu/catalog.php?record_id=l 1586).
Similarly, it appears that California estimates of the costs of achieving zero emissions vehicles were
too optimistic (National Research Council, 2006).
Not all costs of compliance with CAAA regulations are incorporated in markets. For
example, the shift from two-cycle to four-cycle and fuel-injected gasoline engines for recreational
equipment (e.g., marine outboards, snowmobiles) may produce non-market costs in the form of
higher weight for these components, reducing ease of use and transport.
-------
Methodological choices (See charge question l.b.)
The application of the CGE modeling should reflect recent developments in the literature
that have moved beyond the general finding of a tax-interaction effect (and associated hidden costs)
associated with regulatory programs to the recognition that the magnitude of the effect differs by
sector and context. The recent finding that the tax-interaction effect is much reduced when there are
quasi-fixed factors of production (Bento and Jacobsen, "Ricardian rents, environmental policy and
the 'double-dividend' hypothesis," Journal of Environmental Economics and Management 53(1):
17-31, 2007) appears to be especially relevant to the electricity sector. There is even the possibility
of a double dividend in this case.
In comparing current and previous cost estimates (p. 1-8 of Direct Cost Report and
elsewhere) it is important to distinguish between regulations or regulatory goals considered in the
previous estimates and additional regulations not considered in the earlier estimates, such as new
regulations on hazardous air pollutants (HAPs).
To augment the Integrated Planning Model (IPM) estimates to account for sunk investments
(p. 2-6), the Project Team might consider comparisons with previous runs of IPM and its
predecessor. There have been numerous regulatory impact analyses (RIAs) and other reports that
include information about the cost of investments going forward. If such an estimate appears in a
1995 RIA, then when looking back from the next vantage point one could take the cost forecast of
the 1995 report as one measure of the capital costs incurred after 1995.
Specific Comments on Direct Cost Report
Unidentified Measures (See charge question 2.a.)
The Project Team has been unable to identify measures that yield sufficient emission
reductions to comply with the National Ambient Air Quality Standards (NAAQS) and relies on
unidentified pollution control measures to make up the difference. Emission reductions attributed to
unidentified measures appear to account for a large share of emission reductions required for a few
large metropolitan areas but a relatively small share of emission reductions in other locations and
nationwide.
The Council agrees with the Project Team that there is little credibility and hence limited
value to assigning costs to these unidentified measures. It suggests taking great care in reporting
cost estimates in cases where unidentified measures account for a significant share of emission
reductions. At a minimum, the components of the total cost associated with identified and
unidentified measures should be clearly distinguished. In some cases, it may be preferable to not
quantify the costs of unidentified measures and to simply report the quantity and share of emissions
reductions attributed to these measures.
When assigning costs to unidentified measures, the Council suggests that a simple,
transparent method that is sensitive to the degree of uncertainty about these costs is best. Of the
three approaches outlined, assuming a fixed cost/ton appears to be the simplest and most
-------
straightforward. Uncertainty might be represented using alternative fixed costs per ton of emissions
avoided.
Transfer ability of Learning Effects (See charge question 2.b.)
In previous guidance, the Council suggested that the Project Team use sector-specific
empirical data on learning effects whenever possible in its cost analysis. The Project Team has
followed this advice and the draft Direct Cost Report describes a number of sector-specific estimates
of learning effects. For a number of sectors, however, no empirical data are available. These data
gaps include sectors, such as non-EGU point source control and costs of producing Clean Air Act-
mandated fuels, where costs are substantial.
The Council commends the Project Team for its work in collecting and evaluating sector-
specific learning-curve data and suggests that the Project Team continue to apply empirical data in
sectors for which such data are available. In sectors for which no data are available, several options
are available, as summarized below:
1. Where data gaps exist, attempt to identify sectors, for which data are available, that have
similar anticipated learning patterns
2. Assume that a default learning effects apply to sectors for which no empirical data are
available
3. Assume no learning effects occur in sector for which no data are available.
The Council suggests that the Project Team assume that some learning occurs in all sectors,
even when direct estimates for a specific sector are not available (e.g. non EGU SCR, or Clean Air
Act mandated fuels). Based on the range of empirical data reported by the Project Team, the Council
suggests using a consistent rate of 90% or 95% (option 2 above). Assuming a single default learning
effect for sectors for which empirical data do not exist, rather than attempting to estimate rates for
each sector, improves the transparency of the methodology. Moreover, several of the cited learning
curve studies are old (and may not reflect current rates), and some of the observed variation in
empirical estimates by sector may reflect random variation rather than systematic sectoral
differences, and so using a common rate across sectors may be more accurate.
Because learning effects are uncertain, the Council suggests that the Project Team conduct
one or more sensitivity analyses assuming different default learning rates. One logical sensitivity
analysis to perform would be to assume that no learning effects occur (option 3 above).
The Council also suggests that the Project Team consider whether there are learning costs
that should be incorporated. "Learning" results not only from experience, but also from factors such
as economies of scale and research and development. If learning costs are not explicitly included in
the analysis, the Project Team may wish to assume conservative values for learning rates so as to
avoid overstating the net benefits of learning. The Council also suggests that the Project Team
consider how cost and quality are related in estimating learning effects. Learning curve studies
typically hold quality constant. More stringent CAAA regulations will require better technologies
(e.g. higher removal efficiencies). The Project Team may consider how costs in the model are
related to the quality of emissions abatement technology.
-------
Discount Rate Selection and Implementation (See charge question 2.c.)
As discussed above, two types of costs are relevant to the study, private costs that are used as
input to the CGE model and partial-equilibrium estimates of social costs. Private costs should be
calculated using private discount rates and social costs should be calculated using social discount
rates.
For a social discount rate, the Council agrees that 5 percent is reasonable given the many
uncertainties involved in choice of discount rate and the modest time horizon (a few decades) of
concern in this analysis. In a very simple economic model, there would be only a single interest rate
that would reflect the equilibrium rate of substitution between consumption in different periods. In
practice, rates of return to private investment and savings differ because of taxes and other factors.
The analytically preferred method for handling this difference is to determine the extent to which
regulatory costs displace investment and consumption, to convert the displaced investment to
forgone future consumption increments (i.e., by adjusting for the shadow price of investment), and
to discount displaced and forgone consumption at the consumption discount rate. However, this
analytically preferred method appears to be rarely practical because of difficulties in determining the
extent to which regulations displace investment and consumption and uncertainty about the shadow
price of capital. Moreover, a 5 percent rate is intermediate to typical estimates of the rate of return to
private investment and the rate of return to savings (e.g., OMB Circulars A-94 and A-4 suggest
values of 7 and 3 percent, respectively).
For private costs used as input to the CGE model, the relevant discount rate should be the
private cost of investment funds, which may be somewhat higher than 5 percent. This suggests that
there is no need to adjust IPM's annual capital costs to reflect the 5 percent social discount rate (see
p. 2-9 of the Direct Cost Report). The IPM interest rate and useful life assumptions in Exhibit 2-2
appear reasonable and are appropriate for the purpose of obtaining cost estimates to use as inputs to
the computable general equilibrium analysis.
Treatment of Manufacturer Profit in Social Costing (See charge question 2.d.)
Appropriate treatment of markups in the analysis is related to the distinction between social
and private/consumer costs. Markups reflecting pure economic profit are a transfer from consumers
to producers, and as such should not count towards social costs. However, for an industry to
generate sizable economic profits in the long run, the firms in that industry need considerable
market power, which does not seem to be true of the industries in question (e.g., motor vehicle
producers). Any observed markups probably reflect unobserved opportunity costs of production, not
economic profit, and should be included in social cost.
In some cases, other costs that could be considered a transfer (e.g., property taxes) may
alternatively reflect real resource costs to society (e.g., higher taxes may be incurred when a firm
adds capital equipment and requires more local government resources for police or fire protection).
Note that, if there are measurable differences between social and private costs (e.g., because
of taxes), the Direct Cost Report should calculate both types of costs. The social costs are useful for
-------
comparison to other partial equilibrium numbers, and the private costs are needed as inputs for the
CGE model. Taxes and other transfers may distort behavior and hence create welfare losses that
should be captured in a CGE analysis.
Comments on OAR's Uncertainty Analysis Plan
The current Uncertainty Plan includes analyses beyond those proposed in the May 2003
analytical blueprint. In addition to the primary forecasts of costs and benefits based on central
economic and population growth assumptions, scenarios corresponding to low and high economic
and population growth will also be developed. The other major addition to the Uncertainty Plan is
the inclusion of the results of the expert elicitation study on particulate matter (PM) mortality
concentration-response functions that can provide an alternative to current uncertainty analysis
through results from epidemiology studies. Several offline uncertainty analyses are also described.
The Council concludes that scenario analyses with high and low growth forecasts are a
reasonable set of results to display. It recommends clearly acknowledging that there are no
behavioral responses in the estimates to changes in these growth estimates. For example, if incomes
are lower, the size distribution of the vehicle fleet could shift (toward older and smaller cars) with
implications for overall emissions. This type of change would not be included in the analysis as
proposed.
There is a sense in the plan that uncertainty about benefits is much greater than uncertainly
about costs. While the Council concurs with this judgment, it fears that uncertainty about costs may
be underemphasized; as the Project Team recognizes, factors such as uncertainty about unidentified
control measures, learning effects, and technological change may contribute substantially to cost
uncertainty. There is direct evidence about historical expenditures and other costs, from which one
might be able to derive a range of costs for at least some current technologies. Future costs are
sensitive to learning, technology change, input substitution, and other factors. This uncertainty
might be best examined through some type of alternative assumptions in a scenario analysis.
The Council commends the Project Team for proposing to use a variety of visual displays
that can help to communicate uncertainty to readers (e.g., box-and-whisker plots, cumulative
distribution functions). As these displays make clear, characterizing uncertainty about study results
requires estimates of both the possible outcomes (e.g., benefits, costs) and their probabilities.
Similarly, the proposed break-even analysis requires estimates of the probability of scenarios
occurring under which net benefits of CAAA regulations are negative. Ideally, the Project Team
could develop probability distributions to characterize uncertainty about each of the important input
variables and choices (including, inter alia, model selection) and propagate these uncertainties
through the suite of simulation models to derive a multivariate probability distribution over output
variables.
Such an effort is unrealistic because resource constraints preclude a sufficient number of
model runs. A more fundamental limitation, however, may be the lack of widely-accepted
procedures for constructing probability distributions to characterize uncertainty about inputs (note
these must be multivariate distributions to account for dependence among input variables). In
response to these limitations, the Project Team reasonably proposes to conduct a mixture of scenario
-------
and sensitivity analyses (estimating outcomes conditional on specified input values with no
quantification of the likelihood that those input values are "correct") and uncertainty analyses in
which some uncertainties about input values are characterized as probability distributions,
propagated through appropriate simulation models, and reported as a probability distribution on
outputs.
A mixed approach to uncertainty analysis appears to be the best that can be achieved.
Implementation requires many choices about which uncertainties to represent as probabilities and
which to address through scenario or sensitivity analysis; these should be made carefully, with
attention to how important the uncertainty is to overall uncertainty about the outputs and to the
theoretical and evidentiary basis for estimating probabilities of alternative scenarios or input values.
Similarly, many choices are required about which uncertainties are sufficiently important to justify
including displays showing their effects on outcomes.
In reporting uncertainty, the Project Team should recognize that probabilities are always
conditional on some maintained assumptions (e.g., about the appropriate fate and transport model or
economic growth scenario) and be careful to make clear what uncertainties are incorporated in a
distribution and on what other parameter values or other assumptions these distributions are
conditioned. While it is easy to report multiple results based on alternative input assumptions and to
declare that one cannot say anything about the relative likelihoods of the results, this approach can
be misleading. The simple reporting of a possible outcome may lead readers to infer that the
outcome is likely enough that it merits attention (if not, why report it?). Readers may reasonably
infer that the range of outcomes that is presented spans some probabilistically relevant subset of all
possible outcomes. Moreover, a range gives no information about which parts are more or less likely
- implicitly it may suggest that all values between the ends of the range are equally likely, even
when values near the center are much more likely than values near the endpoints.
In choosing multiple scenarios and other parameter values, analysts must be wary of the
problem of compounding conservatism. This arises when constructing extreme outcomes by
combining uncertainties about multiple inputs. For example, if outcome z is the product of three
probabilistically independent factors w, x, and y, then multiplying the 0.01 fractiles for each of the
three inputs yields the one-in-a-million fractile for z.
Analyzing uncertainty and communicating the results to decision makers is inherently
complex. The Council commends the Project Team for its plan to rely on more informative
graphical displays and cautions the team to avoid simply snowing decision makers a range of
values based on sensitivity analysis. Communication may be improved if the Project Team provides
guidance about where the outcomes that are presented are likely to lie in an overall probability
distribution.
10
------- |