NCEE 0
NATIONAL CENTER FOR
ENVIRONMENTAL ECONOMICS
Do Regulators Overestimate the Costs of Regulation?
R. David Simpson
Working Paper Series
Working Paper # 11 -07
December, 2011
^e.0 sr^ U.S. Environmental Protection Agency
g ra National Center for Environmental Economics
s z 1200 Pennsylvania Avenue, NW (MC 1809)
^ Washington, DC 20460
sP http://www.epa.gov/economics
ProT^-
-------
Do Regulators Overestimate the Costs of Regulation?
R. David Simpson
NCEE Working Paper Series
Working Paper # 11-07
December, 2011
DISCLAIMER
The views expressed in this paper are those of the author(s) and do not necessarily represent those
of the U.S. Environmental Protection Agency. In addition, although the research described in this
paper may have been funded entirely or in part by the U.S. Environmental Protection Agency, it
has not been subjected to the Agency's required peer and policy review. No official Agency
endorsement should be inferred.
-------
Do Regulators Overestimate the Costs of Regulation?
R. David Simpson
National Center for Environmental Economics
United States Environmental Protection Agency
August 2011
I thank Jennifer Bowen, Cynthia Morgan, Carl Pasurka, Ronald Shadbegian, Nathalie Simon,
William Wheeler and Ann Wolverton for comments on related work, Winston Harrington for a
helpful discussion, and Robert Hahn for comments on an earlier draft. The author remains
responsible for any errors. The opinions expressed here are those of the author, and do not
necessarily reflect the views of the United States Environmental Protection Agency.
-------
Do Regulators Overestimate the Costs of Regulation?
Abstract
It has occasionally been asserted that regulators typically overestimate the costs of the
regulations they impose. A number of arguments have been proposed for why this might be the
case, with the most widely credited one being that regulators fail sufficiently to appreciate the
effects of innovation in reducing regulatory compliance costs. Most existing studies have found
that regulators are more likely to over- than to underestimate costs. Moreover, the ratio of ex
ante estimates of compliance costs to ex post estimates of the same costs is generally greater than
one. In this paper I argue that neither piece of evidence necessarily demonstrates that ex ante
estimates are biased. There are several reasons to suppose that the distribution of compliance
costs would be skewed, so that the median of the distribution would lie below the mean. It is not
surprising, then, that most estimates would prove to be too high. Moreover, we would expect
from a simple application of Jensen's inequality that the expected ratio of ex ante to ex post
compliance costs would be greater than one. In this paper I propose a regression-based test of
the bias of ex ante compliance cost estimates, and cannot reject the hypothesis that estimates are
unbiased. Despite the existence of a number of papers reporting ex ante and ex post compliance
cost estimates, it is surprisingly difficult to get a large sample of such comparisons. My most
salient finding does not concern the bias of ex ante cost estimates so much as their inaccuracy
and the continuing paucity of careful studies.
-------
1. Introduction
Since the Reagan Administration regulatory agencies in the United States have been
required to perform cost-benefit analyses of high-profile regulations.1 Many other nations have
also instituted similar requirements for regulatory impact analyses (Radaelli 2005). How
accurate have such ex ante estimates of the costs and benefits of environmental, health, product,
and other regulations proved to be?
With respect to the costs of regulatory compliance, available evidence seems to suggest
that the answer is "not very". The cost estimates offered by regulators are generally higher than
are ex post estimates of compliance costs. A review of ten surveys, each of which reviews the
results of a number of different case studies, finds that in each survey ex ante estimates of
compliance costs exceed ex post estimates in a majority of instances. It is, admittedly, a risky
venture to attempt to make comparisons across studies of different regulations from different
regulators for different industries at different times, and sometimes in different places. If,
however, one performs what may seem a natural test of the overall accuracy of regulatory cost
estimates - averaging the ratio of ex ante to ex post cost estimates - she finds that the average
ratio exceeds one, and often is considerably greater than one.
A number of hypotheses have been advanced to explain why ex ante cost estimates are
often too high. Some emphasize that regulators do not have an incentive to conduct careful cost
estimates: if it appears that a regulation will pass a cost-benefit test anyway, there is no real
motivation to prepare a careful study, or, perhaps more importantly, to inflame opposition from
the affected entities by venturing more controversial estimates. Other authors note that the "first
draft" regulations for which compliance costs are predicted are often more stringent than those
eventually passed and with which regulated entities must comply (or, to introduce another
closely related hypothesis, with which they may not comply in practice). The explanation that
has received the most attention and which seems to generate the most credence, however, is that
regulators fail to account for innovation. As Lisa Heinzerling, former Associate Administrator in
EPA's Office of Policy wrote (albeit in 2002, before coming to EPA), "Regulatory analysis is
notorious for failing to take into adequate account the technological innovations that ultimately
make many regulations cheaper to implement than regulators anticipate". An anonymous
commentator from the Center for Progressive Reform opined that "There is lots of good reason
[sic] to believe the ex-ante estimates ... are systematically biased," and went on to note that
reliance on industry data could account for the bias in regulators' estimates, as well as the
difficulty of predicting technological innovation (Inside EPA 2011).
So, it seems that there are good reasons to suppose that regulators will overestimate the
costs of compliance with environmental regulation, and compelling evidence that their cost
estimates are biased.
1 In practice, this means regulations having an effect on the economy of $100 million or more per year, or
designated as "significant" by the Office of Management and Budget (OMB).
2 We describe the studies in the following section. An exception in one respect is Hodges (1997), who focuses on
the cost estimates offered by affected industries, rather than those prepared by government agencies. Not
surprisingly, Hodges finds that such estimates are especially inflated.
-------
Or are they?
In this paper I suggest that the evidence is not as clear-cut as it has seemed to some
commentators. My main arguments are statistical. Neither of the procedures that have been
employed to evaluate the accuracy of ex ante cost estimates in the existing studies provides a
valid test to determine whether or not ex ante cost estimates are biased. The fact that most ex
ante cost estimates exceed ex post estimates would only indicate a bias if it were reasonable to
suppose that the distribution of ex ante estimates were symmetric. I offer reasons to suppose a
priori that they are not.
The fact that the average ratio of ex ante to ex post estimates exceeds one is also not
unexpected. Here the argument is very simple. Holding the numerator of a fraction fixed, a
fraction is a convex function of its denominator. Because ex post costs will sometimes be lower
than their unbiased expectation, we would always expect the ratio of ex ante to ex post estimates
to be greater than one if the ex ante estimates are unbiased- this is just an application of Jensen's
inequality. If there is some probability that ex post costs are very low, then some of the ratios of
ex ante to ex post estimates may explode.
It is worth repeating before suggesting a better procedure for evaluating the accuracy of
ex ante cost estimates that any procedure involving comparisons between ex ante/ex post
estimate pairs from different industries, regulations, time periods, countries, etc., must rest on
heroic assumptions. Moreover, it is worth remarking that I have been referring to ex ante and ex
post cost "estimates". We can rarely be confident that we have observed the actual realization of
ex post costs.3
For the sake of argument, however, let us make the heroic assumption that we can regard
different observations of ex ante and ex post compliance cost pairs as being in some sense
generated by comparable processes. The null hypothesis is that the law of iterated expectations
holds: the expectation of the best estimate that can be made now of the best estimate that can be
made later is the expectation of the best estimate that can be made later. Thus the ex ante
estimate should be equal to the ex post estimate plus an uncorrected prediction error term. This
hypothesis can be tested by regressing ex ante estimates on a constant term and the
corresponding ex post estimates. The null hypothesis is that the intercept term of this regression
will be zero and the slope one. I cannot reject this hypothesis in a sample of 18 ex ante! ex post
compliance cost estimates.
What should we conclude from this exercise? First and most obviously, that existing
studies do not establish that regulators generate biased estimates of costs. To be fair, it should be
3 An exception to this observation might be found in the case of regulations that establish tradable permit markets.
Then we might compare the anticipated cost of permits with their actual values. The unexpectedly low price of S02
permits under the Clean Air Act Amendments of 1990 has become something of a cause celebre, for example (see,
e. g., Harrington, el al. 2000). It certainly supports the narrative that regulators failed to anticipate innovations in
compliance. I have not included cost estimates that report only the prices of permits in the statistical analysis below,
as I have attempted to confine attention to studies that venture estimates of total compliance costs, which would
include a) investment expenditures; and b) some measure of the range of production units or units of output affected.
-------
noted that this is not really the claim of most of the studies themselves so much as that of some
second-hand summaries of their findings. The authors of most studies are appropriately
circumspect in presenting their results and noting their limitations.4 This conclusion has an
obvious policy implication. If policy makers were tempted to conclude that regulatory cost
estimates are biased and should be revised downward so as to provide a more liberal benefit-cost
test of proposed regulation, this would appear to be premature.
There is, however, perhaps a more important conclusion to be drawn from this exercise.
The problem with existing ex ante cost estimates may not be that they are biased so much as that
they are bad. While it would be nice to have estimates of the costs of regulation that were right
on average, it would be even more comforting to have estimates that were close on average.
Finally, while the conclusion that "more research is needed" is certainly hackneyed (and
in many instances self-serving), if ever it were justified, this would be an instance. One very
surprising finding that comes out of a careful review of existing studies is that existing studies
are very limited. This is not to say that previous authors have not been careful and diligent.
They certainly have. But such fundamental questions as "what constitute costs?" have been
answered in different ways by different authors. Greater methodological standardization would
facilitate comparisons and conclusions of the type I have attempted to draw, and would provide
better guidance for policy. Moreover, one finds on closer inspection that many existing studies
do not record the kind of quantitative information that facilitates comparison. The reader may
have been surprised to read above that, from among the scores of cases comparing ex ante with
ex post costs of compliance, I have assembled a sample of only 18 usable observations. Authors
of some studies have - often of necessity - confined themselves to qualitative assessments. One
also finds that several surveys (such as this one) merely recombine existing studies rather than
generating new data. More primary data collection and analysis would certainly be useful.
This paper is presented in five sections, including this introduction. The next section
reviews the literature on the accuracy of ex ante cost estimates. Following that, I offer
arguments for why the measures reported in the existing literature - the frequency with which ex
ante costs are overestimated and the ratio of ex ante to ex post cost estimates - do not necessarily
shed light on the question of whether ex ante cost estimates are biased. In the fourth section I
propose an alternative statistical test and report its results. A fifth section briefly presents
conclusions.
2. Previous studies
A number of researchers have studied the accuracy of ex ante estimates of the costs of
environmental and other forms of regulation in the light of ex post estimates of such costs (I will
devote some considerable attention later in this paper as to how and why the two might differ).
In the interest of brevity I will distinguish between studies of the disparity between ex ante and
ex post estimates of costs and surveys of studies of such disparities. I will focus on the latter.
4 Moreover, at least some reviewers of the literature more generally come to very even-handed conclusions. Hahn
and Tetlock (2008) write that while allegations of bias are often encountered, the evidence as to which way (if
either) the bias tends is less clear-cut.
-------
There are now quite a number of reports whose authors have taken as their data the results of
earlier studies of particular regulations in particular industries and tried to evaluate the accuracy
of such studies generally. As we will see below, one of the challenges of such undertakings is to
define what it means for ex ante cost estimates to be "accurate". Existing studies generally
report accuracy in terms either of the fraction of studies that overestimate costs, or in terms of the
ratio of ex ante to ex post cost estimates. Broadly speaking, existing studies find that
overestimates are more common than underestimates, while the ratio of ex ante to ex post
estimates tends to be greater than one.
The first study of which I am aware devoted specifically to the consideration of the
accuracy of ex ante projections of the costs of regulation was conducted for EPA by the
consulting firm of Putnam, Hayes, and Bartlett and completed in 1980 (hereinafter, "PHB
1980"). The study compared EPA and industry ex ante estimates of required capital
expenditures for five rules passed in the 1970's with actual capital expenditures. In four of five
cases industry overestimated capital costs, while in three of five cases EPA overestimated capital
costs for the period from 1974 - 1977. The PHB results are somewhat more ambiguous for a
sixth case study, in which EPA and industry estimates of the effects of environmental regulations
on new car prices were compared.
The next major study of the accuracy of cost projections was conducted in 1995 by the
Office of Technology Assessment (OTA). OTA did not consider environmental regulations, but
its study of the accuracy of cost projections of Occupational Safety and Health Administration
(OSHA) regulations may have implications for the accuracy of regulatory cost estimation more
generally. OTA considered eight regulations in chemical, manufacturing, and service industries
enacted between 1974 and 1989. In all cases in which numerical estimates were hazarded
estimated costs exceeded actual costs. In two industries the OTA report suggests that costs may
actually have been negative: in finding ways to reduce risks, producers may actually have
identified processes that operate more efficiently. Such claims would substantiate Michael
Porter's (1991) hypothesis, that firms that operate under tougher environmental regulation can
actually be more competitive in world markets.
In 1997 Hart Hodges published a study of twelve environmental and workplace safety
regulations initiated between the 1970's and 1990's (Hodges 1997; the results are also
summarized in Goodstein and Hodges 1997). In each instance ex ante estimates of costs were
greater than were costs recorded later; in eleven of twelve cases, ex ante cost estimates were
more than double costs realized ex post. Hodges focuses on industry's rather than regulators'
estimates of costs. Inasmuch as industry will, in general, have a powerful incentive to overstate
costs, the discrepancies Hodges identifies are not surprising.
A very thorough comparison of ex ante to ex post estimates of costs was conducted in
2000 by Winston Harrington, Richard Morgenstern, and Peter Nelson. The researchers
considered 28 regulations written by EPA, OSHA, and a handful of other regional and
international regulators. A number of different industries were covered. Ex ante cost estimates
were considered "accurate" if they were within ± 25% of ex post values, and either too high or
too low if they fell outside this range. By this standard total costs of regulation were
overestimated in 15 instances, underestimated in only three, and deemed reasonably accurate in
-------
the remaining 11. Harrington et al. distinguish between total and unit costs of regulation (the
numbers I have just reported are for "total" cost estimates). The latter refer to the costs per unit
of output or the cost per plant. Total cost is per unit cost times output or number of plants
affected. Harrington, et al., find that unit costs tend to be overestimated as often as they are
underestimated, in contrast to total cost estimates. I will discuss below some reasons for which
this might be the case.
The next major retrospective study of the costs of regulation was completed in 2005 by
the Office of Management and Budget (OMB 2005). OMB reviewed 47 regulations initiated
between 1976 and 1995. EPA issued 18 of the regulations in the OMB sample, the most of any
of the five federal agencies included in the study (the others were the National Occupational
Safety and Health Administration (13 regulations included), the National Highway Traffic Safety
Administration (8), the Department of Energy (6) and the Nuclear Regulatory Commission (2)).
As is generally the case with estimates of regulatory costs, the sample was determined by the
availability of data, not by any attempt to generate a random cross-section of regulatory activity.
The results of the OMB study are less striking than those of some other researchers. Of 40
regulations for which comparable ex ante and ex post data are available, 16 ex ante projections
overestimated cost, 12 underestimated them, and 12 were approximately accurate. The OMB
study was not completely independent of earlier work, however: for instance, nine of the studies
in its sample were adopted from Harrington, et al. 2000.
At least three studies have been conducted of the accuracy of ex ante cost measures in
other countries (in addition, Harrington et al. 2000 includes three examples drawn from
Singapore, Norway, and Canada among their 28 case studies). While such inquiries obviously
consider costs generated under different legal and regulatory structures than prevail in the U. S.,
they may still be useful in interpreting general approaches to regulatory cost estimation. It might
also be noted in passing that international standards for the analysis of regulatory impacts have
become more similar over time, with the United Kingdom (MacLeod, et al., 2006) and the
European Union adopting such requirements.5 A study conducted by the Stockholm
Environmental Institute considered the cost estimates presented by industry in regulatory
negotiations, and found them to be consistently higher than ex post realizations of actual costs
(Bailey, et al., 2002).
MacLeod, et al. (2006) performed a similar analysis of ex ante costs in UK rulemaking.
The authors of this study adopted the same ± 25% standard as used in Harrington, et al., 2000,
and found that by this standard the costs of five of eight regulations considered were
overestimated, those of two regulations were underestimated, and those of one were
approximately on target.
In 2006 Oosterhuis, et al. published their estimates of ex ante and ex post costs of
regulation with five EU environmental regulations. They report that in four instances ex ante
cost estimates exceeded ex post costs by a factor of two or more, while the ex ante and ex post
5 See Radaelli 2005, however, who notes that "regulatory impact assessments" may still differ significantly from one
jurisdiction to another
-------
estimates were roughly the same in the fifth case.6 Oosterhuis et al. also report on an earlier
study of costs of compliance with Dutch environmental regulations of the first Dutch National
Environmental Policy Plan of 1988, as predicted ex ante by Jantzen (1989) and later estimated ex
post by RIVM (2001). These Dutch studies were, by the standards of the field, unusually
accurate. While the costs of five of the eight regulations considered were overestimated, only
one ex ante estimate was as much as twice its ex post realization, and in aggregate the total ex
ante estimate of slightly over €12 billion was only 13% higher than the ex post realization.
Oosterhuis et al. (2006) credit this unusually accurate performance to the existence of relatively
good statistics and studies in the Netherlands.
We will conclude this section with summaries of two studies that considered the accuracy
of ex ante cost predictions for specific consumer products. Anderson and Sherwood (2002)
compare cost estimates for EPA mobile source rules. These include six fuel-quality regulations
and eleven vehicle emission standards. In most instances Anderson and Sherwood found that ex
ante estimates of price increases induced by regulation were greater than actual price changes
observed. They also found, however, that EPA estimates tended to be closer to actual price
changes than were industry estimates.
Dale, et al. (2009) considered the costs associated with the Department of Energy's
efficiency regulations on consumer appliances such as air conditioners, refrigerators, and
washing machines. This study illustrates the challenges inherent in developing estimates for the
costs of regulation for consumer goods. Dale, et al. derived their ex post cost estimates using
hedonic regressions to tease out the separate effects of scale, general technological progress, and
more competitive behavior from those of the energy efficiency regulations themselves. Having
isolated these effects, the authors found, as have the other studies, that ex ante cost estimates
generally exceed those developed ex post.
3. What does the literature show?
The studies we have reviewed uniformly find that regulators overestimate the costs of
regulatory compliance more often than they underestimate them, and that the ratio of ex ante to
ex post compliance cost estimates is, on average, considerably greater than one. While this
might seem at first blush to establish that regulators' ex ante estimates of the costs of regulatory
compliance are biased upward, this assertion does not actually withstand closer scrutiny. I
consider the two types of evidence in turn, and show that neither necessarily reveals a bias in
estimates.
Skewed distributions
One of the most robust findings in the existing literature comparing ex ante to ex post
estimates of costs is that the former generally exceed the latter. I am aware of no study in which
more ex ante cost estimates were lower than ex post estimates, as opposed to higher, and in many
6 Oosterhuis et al. actually consider six environmental directives, addressing large combustion plants, integrated
pollution prevention and control, ozone control, ozone depleting substances packaging, and nitrates, but are unable
to develop ex ante compliance cost estimation numbers for the packaging directive.
-------
a substantial majority of ex ante estimates were higher than the corresponding ex post estimates.
Can we then conclude that ex ante cost estimates are generally biased upward?
The answer would appear to be "No," at least not on the basis of this simple observation
alone. An estimate is "biased," by the statistical definition of the term, if its expected value
differs from the mean of the population from which it is drawn. It is entirely possible that a
majority of observations would be below the mean of the distribution of the population from
which they are drawn. The median of a distribution will be below the mean if the distribution is
skewed, i. e., if it is not symmetrical about its mean.
Do we have any reason to suppose that the distribution of costs would not be
symmetrical? Yes, there are several. First, total costs are often estimated by multiplying an
estimate of unit costs by the number of units affected by the regulation (Harrington, et al. 2000).
Both cost-per-unit and units affected are random variables: the researcher cannot observe the
former accurately, while absent perfect understanding of market conditions and drivers, the
researcher cannot know the latter with certainty. Suppose that the increase in unit costs and the
number of units affected are each distributed independently and symmetrically on nonnegative
supports. Then their product will be distributed asymmetrically. Heuristically, there is a small
probability that unit costs will be large, and a small probability that the number of units affected
will be large. Thus, there is a very small probability that the cost of regulation will be very large.
Under such conditions the distribution of total costs will have a long right tail, and hence, be
asymmetric.
A very simple example illustrates the point. Suppose that both the increase in unit costs
and the number of affected units are distributed independently and uniformly on the interval [0,
1] (we can always make the supports the same by choice of units of measurement). Then it is
easily demonstrated that their product is distributed logarithmically on the interval [0, 1] with
probability distribution function - In 0. This function has mean ]A and median of approximately
0.187, and about 59.2% of observations are less than the mean.
Another reason to suppose that the distribution of costs is asymmetric is because the
mathematical forms that give rise to such costs are often asymmetrically distributed. It can be
shown that if production in a regulated industry can be modeled by a constant returns to scale
production function of unpriced, but regulated, emissions and purchased inputs, then the
reduction in profits that would result if emissions are required to be reduced by an amount Ae
may be approximated as
CC « ;r-(Ae/e), (1)
where Ae/e is the proportional change in allowed emissions and n is industry profit per unit of
emissions, and n is independent of the level of emissions. This expression will be approximately
true for small changes in allowed emissions or if the affected industry is small enough as to have
negligible impact on input and output prices.
Suppose that we choose a simple form for the production function; consider, for example,
the Cobb-Douglas form
-------
f(x,e) = xV~'
(2)
where x is the quantity of purchased inputs employed in production and e the amount of effluent
discharged. If p is the price of output and w the price of the input, x (we treat x as a scalar for
simplicity; nothing of consequence would change from treating it as a vector), and we normalize
the quantity of emissions discharged in the status quo ante to one, it can be shown that
71
= (l -a)pl a \ —
a
w)
(3)
If we treat a,p, and w as unknown random variables with independent symmetric distributions,
the resulting function k is asymmetrically distributed. This is not surprising, as the central limit
theorem applied to the product, rather than the sum, of independent random variables, implies
that the product will be lognormally distributed. A histogram for one such distribution is
presented below:
450
400
350
300
250
200
150
100
50
0
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70
.~
dflnnnnnn
a,p, and w are distributed normally with means of 0.75, 1, and 1, respectively, and standard
deviations of 0.05, 0.10, and 0.10, respectively. The expectation of the resultant distribution is
0.120, the median 0.106, and 59.1% of observations are less than the mean.
A third reason for supposing that the distribution of costs might be skewed may arise
from the attributes of innovation. The story that is often told as to why costs tend to be
overestimated is that the people recording estimates tend to discount the possibility of
innovation. They do not reflect the high likelihood that much more cost-effective ways of
-------
complying with regulation will be identified. It could well be, however, that while very cost-
effective strategies are, in fact, identified most of the time, spectacularly costly exceptions could
draw the mean cost of regulation considerably higher than the median.
Cost ratios
The fact that most ex ante estimates of costs are higher than are the corresponding ex post
estimates does not necessarily imply that such estimates are biased. In order to make the
determination of whether or not estimates are biased, we would need to know whether there are
n
occasional spectacular exceptions in which ex ante costs severely wwderestimated actual costs.
There are certainly instances in which the costs of rules have been underestimated. The
problem, though, is that we never have repeated samples from the same distribution. There is, in
each case, one rule whose costs are estimated, yielding one ex ante estimate and one ex post
estimate. It would require some truly heroic assumptions to say that the fact that costs were
wwderestimated on a regulation affecting electric utilities, for example, somehow "offsets" the
fact that costs were overestimated on rules affecting automobiles and appliances.
Let us suppose, however, that one were prepared to make such heroic assumptions.
Many studies note that, for most rules, ex ante estimates are a multiple greater than the
corresponding ex post estimates. What might we infer from such observations?
Again, the frustrating answer may be "not much". To see why, consider a very simple
example. Suppose that a rule is being contemplated, and that with 50% probability costs will be
50, while with 50% probability costs will be 150. In expectation, then, costs would be 100. On
average, however, the ratio of predicted ex ante to ex post costs will be Vi ¦ 100/50 + V2 • 100/150
= 1 1/3. Note that the issue here is not the symmetry of the distribution of realized costs. Rather,
it is that if ex post costs are low, the ratio of estimated to realized costs explodes, even if the
estimation is unbiased.
The problem may be even worse if it is made more realistic. It is not unreasonable to
suppose that costs of regulation in some instances could be near zero, or possibly even negative.
While it is unreasonable to suppose that expected costs of compliance would be negative (if they
were, why wouldn't firms make the changes absent regulatory urging?), it is certainly not
inconceivable that, once having been induced to consider some innovations, firms would
occasionally hit on some solutions that would actually reduce their costs of production and
increase their profits (Oosterhuis, et al., 2006 raise the possibility that innovations could result in
compliance at negative net cost; OTA 1995 suggests that a couple of OSHA rules might have
generated negative compliance costs). It is possible, then, that the ratio of ex ante to ex post
estimates would be unbounded.
In short, then, while it may sound troubling to read that ex ante cost estimates exceed ex
post estimates by substantial multiples, such a finding would not establish that ex ante estimates
7 The 2005 OMB study raises another interesting point that is worth considering in passing. We can never observe
the accuracy of cost estimates for rules that were never issued.
-------
were unbiased. It could rather, and perhaps counterintuitively, suggest that no statistical
o
inferences could be drawn on the basis of such data.
4. Evaluating the accuracy of Ex ante cost estimates: An alternative approach
In practice, both ex ante and ex post estimates of costs are just that - estimates of random
variables whose true values remain unknown either because the rule under contemplation has not
yet been enacted, in the case of ex ante estimates, or because we cannot completely and
accurately observe all affected entities' costs of in the case of ex post estimates. Somewhat more
formally, we could say that any estimate of costs, 9, may be written as E{9 \ Q), where Q is an
information set available at the time the estimate is made. At a later time a revised estimate of 9
might be formed based on an updated information set Q\
A convenient way to represent the relationship between an ex ante expectation (i. e.,
conditioned on the initial information) and an ex post expectation (conditioned on updated
information) is that
£(#|Q') = £(0|n)+e, (4)
where s is a forecasting error.
The ex ante estimate will be unbiased if E{s) = 0. Furthermore, the forecasting error s
should be uncorrected with the expectation of costs conditioned on the information that is
available before the rule is enacted. If it were correlated, knowledge of the correlation could be
used to derive a better estimate.
If we had a large sample, all of whose values of costs were drawn from the same
distribution, and all of whose realizations of conditioning information were drawn from the same
distribution, it would be relatively easy to test the hypothesis that s has zero mean. We could
simply compare the averages of ex ante and ex post costs.
This procedure would be somewhat problematic, in our case, however, as each
observation on a pair of ex ante and ex post cost estimates is drawn from a unique experiment.
The procedure of adding up ex ante costs estimated from different studies and comparing them
with the sum over a corresponding set of ex post costs would mix "apples and oranges", and the
resultant comparison of differences in means would be unduly influenced by those observations
for which costs were highest.
A better procedure would be to specify an empirical version of expression (4):
8 The classic example here is the quotient of two normal distributions, which has a Cauchy distribution. It would
be impossible to tell from the ratio of ex ante to ex post estimates if the former were accurate, as the Cauchy
distribution has no moments. Of course, it seems unreasonable to suppose that the costs of many regulations could
reasonably be defined on a support of [-oo, oo].
-------
£(0,|£V) = a + PE(6, | £!,) + *,,
(5)
where a and /? are unknown parameters to be determined, the subscript i indexes observations on
different prospective regulations, and is a random disturbance term with mean zero and which
is uncorrected with the ex post estimate of costs. The null hypothesis to be tested is, then, that a
= 0 and /? = 1.
I have estimated equation (5) using as data ex ante and ex post cost estimates reported in
Harrington, et al. (2000; specific cost data were found in an earlier working paper, Harrington, el
al. 1999), MacLeod, et al. (2006), and Oosterhuis, et al. (2006; this study includes both original
case studies conducted by the authors and summaries of eight other case studies in which ex ante
estimates were developed by Jantzen, et al. (1989) and ex post estimates reported by RIVM
As detailed descriptions of the data from these studies is included with each, I will not
repeat such descriptions here. I might, however, note, in passing that I was unable to employ
nearly as many data points as might be inferred from the numbers of cases considered in the
studies. It is rather surprising when one consults the actual studies that clear, consistent,
quantitative statements concerning both ex ante and ex post costs are more the exception than the
rule. Harrington, et al., for example, cite 28 cases. I use only seven. The others were eliminated
for want of quantitative data (either in Harrington, et al. 2000, or the working paper on which it
was based, Harrington et al., 1999)9, or because the authors reported only unit-cost estimates
which may not be comparable with aggregate estimates (this, incidentally, is why I have not
included any cost estimates from Anderson and Sherwood 2002, or Dale, et al., 2009).
Similarly, it was possible to derive comparable numbers for ex ante and ex post costs for only
three instances in the MacLeod et al. (2006) report, and Oosterhuis, et al. (2006) proved useful
only inasmuch as we adopted figures that it reported from Jantzen, et al. (1989) and RIVM
(2001). I did not consider studies such as OMB (2005), which compiles estimates from other
sources (relying heavily, for example, on Harrington, et al. 2000), or Hodges 1999, which reports
industry, rather than regulators', estimates of costs.
I decided on a sample of 18 regulations (see Table 1). Six are from the United States, one
from Canada, eight from the Netherlands, and three from the United Kingdom. Regrettably -
and surprisingly - only one US EPA regulation has clear quantitative estimates of both ex ante
and ex post costs corresponding to total (as opposed to unit) effects. Performing the regression
indicated in (5) yields the following results:
9 For example, the authors write of the phase-out of lead from gasoline that "There has not been a retrospective
analysis of the rule's costs but evidence indicates that EPA's analysts correctly forecast the costs or even
overestimated them." While this judgment allowed Harrington et al. to classify this rule among those for which ex
ante costs were estimated with reasonable accuracy, it does not allow me to employ the observation in my
quantitative procedures.
(2001)).
E(ei I n,.') = -0.092 + 0.948E(ei I Q,)
(0.132) (0.084)
(6)
-------
2
R = 0.889 (standard errors in parentheses).
In neither specification can I reject the hypothesis that the intercept is zero and slope one, i. e.,
that ex ante estimates of the costs of regulation are unbiased.10
It would be foolish to try to make too much of these results. Among other potential problems, it
is more reasonable to regard the eighteen before-and-after estimates of costs I have used as a
convenience sample than as any sort of random draw from the entire universe of cost estimates.
In fact, one might suggest that the fact that I do have good before-and-after estimates for these
eighteen rules is evidence that they were more carefully analyzed than were the many other rules
that have been mentioned in studies comparing ex ante to ex post estimates of costs. 1
Moreover, the observation that I cannot reject the hypothesis that ex ante cost estimates are
biased does not imply that such estimates are "good". There is still considerable variation in the
sample, as evidence by the fact that the ratio of ex ante to ex post estimates ranges from 0.207 to
11.2. If nothing else, it would appear that cost estimates can be a long way off, in either
direction.
5. Conclusions
Conclusions were foreshadowed in the introduction, so I will only briefly recapitulate
them here. The first is that any presumption that regulatory costs are overestimated, and hence
that a more liberal interpretation of cost-benefit tests is warranted, is premature. While I am not
aware of any commentator who has made this point explicitly, at least with regard to regulator's,
as opposed to industry's estimates of costs, it is certainly the subtext of some comments. Again,
such evidence as exists does not support such a procedure.
"Such evidence as exists" is, however, sparse. While the authors of existing studies have
labored diligently to gather evidence, the evidence remains limited. Moreover, different studies
have assembled different data in different ways. While I have tried to compare studies that
report similar measures of costs, discrepancies remain between studies as to, e. g., how to include
capital investments and variable costs, time periods, discounting, etc. My results can only be
considered suggestive at best.
Moreover, as other authors have suggested, conducting retrospective studies of the
accuracy of ex ante cost estimates remains something of an orphan activity (see, e. g., Hahn and
Tetlock 2008). It is understandable that regulators would put a higher priority on predicting the
10 It might reasonably be suggested that the regression reported in equation (6) will be very inefficient, as we might
reasonably expect considerable heteroskedasticity: the costs of very costly rules are likely estimated with
considerably higher errors than are less costly ones. I also transformed (6) by weighting by ex ante estimates and
found again that I could not reject the hypothesis that ex ante estimates were unbiased.
11 Some authors have noted that estimates reported in such studies might not have been chosen at random reasons
(see, e. g., Hahn and Tetlock 2008). High-profile regulations, rules for which ex ante predictions were spectacularly
inaccurate, or instances illustrating economists' favorite hobby horses (e. g., those allowing allowance trading)
might all be more likely to be considered.
-------
effects of prospective regulations than they would on evaluating the accuracy of their predictions
of regulations that have already been promulgated. It is also understandable that those who have
ventured predictions in the past would be reluctant to revisit them: the best possible outcome
would be that they would be shown to have done their job competently, while the alternative is
that their best efforts would be found lacking. Be that as it may, however, it would certainly be
useful to high-level decision makers to know how reliable the information they are receiving is -
or at least, how reliable it has been in the past. Ultimately, this information might show why
different studies have over- or underestimated costs, and whether the prospect for technological
innovation is, in fact, underappreciated.
-------
References
Anderson, J.F., and Sherwood, T. 2002. Comparison of EPA and Other Estimates of Mobile
Source Rule Costs to Actual Price Changes. Paper presented at the SAE Government
Industry Meeting, Washington, DC, May 14, 2002.
Anonymous. 2011. EPA Launches Study To Improve Cost Estimates For New Regulations.
Inside EPA. 31 August. Available online at insideepa.com.
Bailey, Peter D., Gary Haq, and Andy Goudson. 2002. Mind the gap! Comparing ex ante and
ex post assessments of the costs of complying with environmental regulation. European
Environment 12 (5): 245 - 256.
Dale, Larry, Camille Antinori, Michael McNeil, James E. McMahon and K. Sydny Fujita. 2009.
Retrospective evaluation of appliance price trends. Energy Policy 31,2 (February). 597-605.
Goodstein, Eban, and Hart Hodges. 1997. Polluted data: Overestimating environmental costs.
American Prospect 8 (35): 64-69.
Hahn, Robert, and Rohit Malik. 2004. Is regulation good for you? Harvard Journal of Law and
Public Policy 27, 3 (October). 893 - 916.
Harrington, Winston, Richard D. Morgenstern, and Peter Nelson. 2000. On the accuracy of
regulatory cost estimates. Journal of Policy Analysis and Management 19 (2). 297 - 322.
Heinzerling, Lisa. 2002. Markets for arsenic. Georgetown Law Review 90: 2311 -2339/
Hodges, Hart. 1997. Falling prices: Cost of complying with environmental regulations almost
always less than advertised. Economic Policy Institute Briefing Paper. Washington, DC:
Economic Policy Institute.
Jantzen, J. (1989), Costs of Environmental Management, 1988-2010, 3 policy scenarios. Report
for the Ministry of VROM, The Hague, 25 May 1989.
MacLeod, Michael, Dominic Moran, Manuel Lago Aresti, Winston Harrington, and Richard
Morgenstern. 2006. Comparing the ex ante and ex post costs of complying with regulatory
change. Final report to DEFRA. London: Department for Environment, Farms, and Rural
Affairs.
Office of Management and Budget, Office of Information and Regulatory Affairs (OMB). 2005.
Validating Regulatory Analysis: 2005 Report to Congress on the Costs and Benefits of Federal
Regulations and Unfunded Mandates on State, Local, and Tribal Entities. Washington DC:
Government Printing Office.
-------
Office of Technology Assessment (OTA). 1995. Gauging Control Technology and Regulatory
Impacts in Occupational Safety and Health: An Appraisal of OSHA's Analytic Approach. OTA-
ENV-635. Washington DC: Government Printing Office.
Porter, Michael. 1991. America's green strategy. Scientific American 264(4). 96.
Radaelli, Claudio. 2005. What Does Regulatory Impact Assessment Mean in Europe ?
Washington: AEI-Brookings Joint Center for Regulatory Studies.
RIVM (2000), Techno 2000; Modellering van de daling van eenheidskosten van technologieen
in de tijd. Rapportnummer 773008003, Bilthoven, April 2000.
-------
Ex ante cost
Ex ante cost
Nation (for US,
estimates
estimates
Agency in
(millions of US
(millions of US
parentheses)
Rule
dollars)
dollars)
Source
Ontario
Ontario water
58
51
Harrington, etal. 2000
US (OSHA)
Vinyl Chloride
1000
253
Harrington, etal. 2000
US (OSHA)
Cotton Dust
280
83
Harrington, etal. 2000
US (OSHA)
Occupational
Lead
224
20
Harrington, etal. 2000
US (OSHA)
Formaldehyde
11
6
Harrington, etal. 2000
US (EPA)
S02 Phase 1
764
779
Harrington, etal. 2000
US (OSHA)
Ethylene oxide
24
25
Harrington, etal. 2000
UK
Control of Major
Accidents
Hazards
155
416
MacLeod, etal. 2006
UK
Food Safety
(General Food
Hygiene/Butchers'
Shops)
5
25
MacLeod, etal. 2006
England
The Welfare of
Farmed Animals
3
3
MacLeod, etal. 2006
Netherlands
Acidification
2620
1248
Jantzen 1989
Netherlands
Climate change
617
839
Jantzen 1989
Netherlands
Eutrophication
1471
814
Jantzen 1989
Netherlands
Hazardous
Jantzen 1989
Substances
3465
2738
Netherlands
Waste
Jantzen 1989
Management
4848
5443
Netherlands
Soil sanitation
914
881
Jantzen 1989
Netherlands
Disturbance
923
763
Jantzen 1989
Netherlands
Other
1939
2140
Jantzen 1989
------- |