NCEE 0 NATIONAL CENTER FOR ENVIRONMENTAL ECONOMICS Do Regulators Overestimate the Costs of Regulation? R. David Simpson Working Paper Series Working Paper # 11 -07 December, 2011 ^e.0 sr^ U.S. Environmental Protection Agency g ra National Center for Environmental Economics s z 1200 Pennsylvania Avenue, NW (MC 1809) ^ Washington, DC 20460 sP http://www.epa.gov/economics ProT^- ------- Do Regulators Overestimate the Costs of Regulation? R. David Simpson NCEE Working Paper Series Working Paper # 11-07 December, 2011 DISCLAIMER The views expressed in this paper are those of the author(s) and do not necessarily represent those of the U.S. Environmental Protection Agency. In addition, although the research described in this paper may have been funded entirely or in part by the U.S. Environmental Protection Agency, it has not been subjected to the Agency's required peer and policy review. No official Agency endorsement should be inferred. ------- Do Regulators Overestimate the Costs of Regulation? R. David Simpson National Center for Environmental Economics United States Environmental Protection Agency August 2011 I thank Jennifer Bowen, Cynthia Morgan, Carl Pasurka, Ronald Shadbegian, Nathalie Simon, William Wheeler and Ann Wolverton for comments on related work, Winston Harrington for a helpful discussion, and Robert Hahn for comments on an earlier draft. The author remains responsible for any errors. The opinions expressed here are those of the author, and do not necessarily reflect the views of the United States Environmental Protection Agency. ------- Do Regulators Overestimate the Costs of Regulation? Abstract It has occasionally been asserted that regulators typically overestimate the costs of the regulations they impose. A number of arguments have been proposed for why this might be the case, with the most widely credited one being that regulators fail sufficiently to appreciate the effects of innovation in reducing regulatory compliance costs. Most existing studies have found that regulators are more likely to over- than to underestimate costs. Moreover, the ratio of ex ante estimates of compliance costs to ex post estimates of the same costs is generally greater than one. In this paper I argue that neither piece of evidence necessarily demonstrates that ex ante estimates are biased. There are several reasons to suppose that the distribution of compliance costs would be skewed, so that the median of the distribution would lie below the mean. It is not surprising, then, that most estimates would prove to be too high. Moreover, we would expect from a simple application of Jensen's inequality that the expected ratio of ex ante to ex post compliance costs would be greater than one. In this paper I propose a regression-based test of the bias of ex ante compliance cost estimates, and cannot reject the hypothesis that estimates are unbiased. Despite the existence of a number of papers reporting ex ante and ex post compliance cost estimates, it is surprisingly difficult to get a large sample of such comparisons. My most salient finding does not concern the bias of ex ante cost estimates so much as their inaccuracy and the continuing paucity of careful studies. ------- 1. Introduction Since the Reagan Administration regulatory agencies in the United States have been required to perform cost-benefit analyses of high-profile regulations.1 Many other nations have also instituted similar requirements for regulatory impact analyses (Radaelli 2005). How accurate have such ex ante estimates of the costs and benefits of environmental, health, product, and other regulations proved to be? With respect to the costs of regulatory compliance, available evidence seems to suggest that the answer is "not very". The cost estimates offered by regulators are generally higher than are ex post estimates of compliance costs. A review of ten surveys, each of which reviews the results of a number of different case studies, finds that in each survey ex ante estimates of compliance costs exceed ex post estimates in a majority of instances. It is, admittedly, a risky venture to attempt to make comparisons across studies of different regulations from different regulators for different industries at different times, and sometimes in different places. If, however, one performs what may seem a natural test of the overall accuracy of regulatory cost estimates - averaging the ratio of ex ante to ex post cost estimates - she finds that the average ratio exceeds one, and often is considerably greater than one. A number of hypotheses have been advanced to explain why ex ante cost estimates are often too high. Some emphasize that regulators do not have an incentive to conduct careful cost estimates: if it appears that a regulation will pass a cost-benefit test anyway, there is no real motivation to prepare a careful study, or, perhaps more importantly, to inflame opposition from the affected entities by venturing more controversial estimates. Other authors note that the "first draft" regulations for which compliance costs are predicted are often more stringent than those eventually passed and with which regulated entities must comply (or, to introduce another closely related hypothesis, with which they may not comply in practice). The explanation that has received the most attention and which seems to generate the most credence, however, is that regulators fail to account for innovation. As Lisa Heinzerling, former Associate Administrator in EPA's Office of Policy wrote (albeit in 2002, before coming to EPA), "Regulatory analysis is notorious for failing to take into adequate account the technological innovations that ultimately make many regulations cheaper to implement than regulators anticipate". An anonymous commentator from the Center for Progressive Reform opined that "There is lots of good reason [sic] to believe the ex-ante estimates ... are systematically biased," and went on to note that reliance on industry data could account for the bias in regulators' estimates, as well as the difficulty of predicting technological innovation (Inside EPA 2011). So, it seems that there are good reasons to suppose that regulators will overestimate the costs of compliance with environmental regulation, and compelling evidence that their cost estimates are biased. 1 In practice, this means regulations having an effect on the economy of $100 million or more per year, or designated as "significant" by the Office of Management and Budget (OMB). 2 We describe the studies in the following section. An exception in one respect is Hodges (1997), who focuses on the cost estimates offered by affected industries, rather than those prepared by government agencies. Not surprisingly, Hodges finds that such estimates are especially inflated. ------- Or are they? In this paper I suggest that the evidence is not as clear-cut as it has seemed to some commentators. My main arguments are statistical. Neither of the procedures that have been employed to evaluate the accuracy of ex ante cost estimates in the existing studies provides a valid test to determine whether or not ex ante cost estimates are biased. The fact that most ex ante cost estimates exceed ex post estimates would only indicate a bias if it were reasonable to suppose that the distribution of ex ante estimates were symmetric. I offer reasons to suppose a priori that they are not. The fact that the average ratio of ex ante to ex post estimates exceeds one is also not unexpected. Here the argument is very simple. Holding the numerator of a fraction fixed, a fraction is a convex function of its denominator. Because ex post costs will sometimes be lower than their unbiased expectation, we would always expect the ratio of ex ante to ex post estimates to be greater than one if the ex ante estimates are unbiased- this is just an application of Jensen's inequality. If there is some probability that ex post costs are very low, then some of the ratios of ex ante to ex post estimates may explode. It is worth repeating before suggesting a better procedure for evaluating the accuracy of ex ante cost estimates that any procedure involving comparisons between ex ante/ex post estimate pairs from different industries, regulations, time periods, countries, etc., must rest on heroic assumptions. Moreover, it is worth remarking that I have been referring to ex ante and ex post cost "estimates". We can rarely be confident that we have observed the actual realization of ex post costs.3 For the sake of argument, however, let us make the heroic assumption that we can regard different observations of ex ante and ex post compliance cost pairs as being in some sense generated by comparable processes. The null hypothesis is that the law of iterated expectations holds: the expectation of the best estimate that can be made now of the best estimate that can be made later is the expectation of the best estimate that can be made later. Thus the ex ante estimate should be equal to the ex post estimate plus an uncorrected prediction error term. This hypothesis can be tested by regressing ex ante estimates on a constant term and the corresponding ex post estimates. The null hypothesis is that the intercept term of this regression will be zero and the slope one. I cannot reject this hypothesis in a sample of 18 ex ante! ex post compliance cost estimates. What should we conclude from this exercise? First and most obviously, that existing studies do not establish that regulators generate biased estimates of costs. To be fair, it should be 3 An exception to this observation might be found in the case of regulations that establish tradable permit markets. Then we might compare the anticipated cost of permits with their actual values. The unexpectedly low price of S02 permits under the Clean Air Act Amendments of 1990 has become something of a cause celebre, for example (see, e. g., Harrington, el al. 2000). It certainly supports the narrative that regulators failed to anticipate innovations in compliance. I have not included cost estimates that report only the prices of permits in the statistical analysis below, as I have attempted to confine attention to studies that venture estimates of total compliance costs, which would include a) investment expenditures; and b) some measure of the range of production units or units of output affected. ------- noted that this is not really the claim of most of the studies themselves so much as that of some second-hand summaries of their findings. The authors of most studies are appropriately circumspect in presenting their results and noting their limitations.4 This conclusion has an obvious policy implication. If policy makers were tempted to conclude that regulatory cost estimates are biased and should be revised downward so as to provide a more liberal benefit-cost test of proposed regulation, this would appear to be premature. There is, however, perhaps a more important conclusion to be drawn from this exercise. The problem with existing ex ante cost estimates may not be that they are biased so much as that they are bad. While it would be nice to have estimates of the costs of regulation that were right on average, it would be even more comforting to have estimates that were close on average. Finally, while the conclusion that "more research is needed" is certainly hackneyed (and in many instances self-serving), if ever it were justified, this would be an instance. One very surprising finding that comes out of a careful review of existing studies is that existing studies are very limited. This is not to say that previous authors have not been careful and diligent. They certainly have. But such fundamental questions as "what constitute costs?" have been answered in different ways by different authors. Greater methodological standardization would facilitate comparisons and conclusions of the type I have attempted to draw, and would provide better guidance for policy. Moreover, one finds on closer inspection that many existing studies do not record the kind of quantitative information that facilitates comparison. The reader may have been surprised to read above that, from among the scores of cases comparing ex ante with ex post costs of compliance, I have assembled a sample of only 18 usable observations. Authors of some studies have - often of necessity - confined themselves to qualitative assessments. One also finds that several surveys (such as this one) merely recombine existing studies rather than generating new data. More primary data collection and analysis would certainly be useful. This paper is presented in five sections, including this introduction. The next section reviews the literature on the accuracy of ex ante cost estimates. Following that, I offer arguments for why the measures reported in the existing literature - the frequency with which ex ante costs are overestimated and the ratio of ex ante to ex post cost estimates - do not necessarily shed light on the question of whether ex ante cost estimates are biased. In the fourth section I propose an alternative statistical test and report its results. A fifth section briefly presents conclusions. 2. Previous studies A number of researchers have studied the accuracy of ex ante estimates of the costs of environmental and other forms of regulation in the light of ex post estimates of such costs (I will devote some considerable attention later in this paper as to how and why the two might differ). In the interest of brevity I will distinguish between studies of the disparity between ex ante and ex post estimates of costs and surveys of studies of such disparities. I will focus on the latter. 4 Moreover, at least some reviewers of the literature more generally come to very even-handed conclusions. Hahn and Tetlock (2008) write that while allegations of bias are often encountered, the evidence as to which way (if either) the bias tends is less clear-cut. ------- There are now quite a number of reports whose authors have taken as their data the results of earlier studies of particular regulations in particular industries and tried to evaluate the accuracy of such studies generally. As we will see below, one of the challenges of such undertakings is to define what it means for ex ante cost estimates to be "accurate". Existing studies generally report accuracy in terms either of the fraction of studies that overestimate costs, or in terms of the ratio of ex ante to ex post cost estimates. Broadly speaking, existing studies find that overestimates are more common than underestimates, while the ratio of ex ante to ex post estimates tends to be greater than one. The first study of which I am aware devoted specifically to the consideration of the accuracy of ex ante projections of the costs of regulation was conducted for EPA by the consulting firm of Putnam, Hayes, and Bartlett and completed in 1980 (hereinafter, "PHB 1980"). The study compared EPA and industry ex ante estimates of required capital expenditures for five rules passed in the 1970's with actual capital expenditures. In four of five cases industry overestimated capital costs, while in three of five cases EPA overestimated capital costs for the period from 1974 - 1977. The PHB results are somewhat more ambiguous for a sixth case study, in which EPA and industry estimates of the effects of environmental regulations on new car prices were compared. The next major study of the accuracy of cost projections was conducted in 1995 by the Office of Technology Assessment (OTA). OTA did not consider environmental regulations, but its study of the accuracy of cost projections of Occupational Safety and Health Administration (OSHA) regulations may have implications for the accuracy of regulatory cost estimation more generally. OTA considered eight regulations in chemical, manufacturing, and service industries enacted between 1974 and 1989. In all cases in which numerical estimates were hazarded estimated costs exceeded actual costs. In two industries the OTA report suggests that costs may actually have been negative: in finding ways to reduce risks, producers may actually have identified processes that operate more efficiently. Such claims would substantiate Michael Porter's (1991) hypothesis, that firms that operate under tougher environmental regulation can actually be more competitive in world markets. In 1997 Hart Hodges published a study of twelve environmental and workplace safety regulations initiated between the 1970's and 1990's (Hodges 1997; the results are also summarized in Goodstein and Hodges 1997). In each instance ex ante estimates of costs were greater than were costs recorded later; in eleven of twelve cases, ex ante cost estimates were more than double costs realized ex post. Hodges focuses on industry's rather than regulators' estimates of costs. Inasmuch as industry will, in general, have a powerful incentive to overstate costs, the discrepancies Hodges identifies are not surprising. A very thorough comparison of ex ante to ex post estimates of costs was conducted in 2000 by Winston Harrington, Richard Morgenstern, and Peter Nelson. The researchers considered 28 regulations written by EPA, OSHA, and a handful of other regional and international regulators. A number of different industries were covered. Ex ante cost estimates were considered "accurate" if they were within ± 25% of ex post values, and either too high or too low if they fell outside this range. By this standard total costs of regulation were overestimated in 15 instances, underestimated in only three, and deemed reasonably accurate in ------- the remaining 11. Harrington et al. distinguish between total and unit costs of regulation (the numbers I have just reported are for "total" cost estimates). The latter refer to the costs per unit of output or the cost per plant. Total cost is per unit cost times output or number of plants affected. Harrington, et al., find that unit costs tend to be overestimated as often as they are underestimated, in contrast to total cost estimates. I will discuss below some reasons for which this might be the case. The next major retrospective study of the costs of regulation was completed in 2005 by the Office of Management and Budget (OMB 2005). OMB reviewed 47 regulations initiated between 1976 and 1995. EPA issued 18 of the regulations in the OMB sample, the most of any of the five federal agencies included in the study (the others were the National Occupational Safety and Health Administration (13 regulations included), the National Highway Traffic Safety Administration (8), the Department of Energy (6) and the Nuclear Regulatory Commission (2)). As is generally the case with estimates of regulatory costs, the sample was determined by the availability of data, not by any attempt to generate a random cross-section of regulatory activity. The results of the OMB study are less striking than those of some other researchers. Of 40 regulations for which comparable ex ante and ex post data are available, 16 ex ante projections overestimated cost, 12 underestimated them, and 12 were approximately accurate. The OMB study was not completely independent of earlier work, however: for instance, nine of the studies in its sample were adopted from Harrington, et al. 2000. At least three studies have been conducted of the accuracy of ex ante cost measures in other countries (in addition, Harrington et al. 2000 includes three examples drawn from Singapore, Norway, and Canada among their 28 case studies). While such inquiries obviously consider costs generated under different legal and regulatory structures than prevail in the U. S., they may still be useful in interpreting general approaches to regulatory cost estimation. It might also be noted in passing that international standards for the analysis of regulatory impacts have become more similar over time, with the United Kingdom (MacLeod, et al., 2006) and the European Union adopting such requirements.5 A study conducted by the Stockholm Environmental Institute considered the cost estimates presented by industry in regulatory negotiations, and found them to be consistently higher than ex post realizations of actual costs (Bailey, et al., 2002). MacLeod, et al. (2006) performed a similar analysis of ex ante costs in UK rulemaking. The authors of this study adopted the same ± 25% standard as used in Harrington, et al., 2000, and found that by this standard the costs of five of eight regulations considered were overestimated, those of two regulations were underestimated, and those of one were approximately on target. In 2006 Oosterhuis, et al. published their estimates of ex ante and ex post costs of regulation with five EU environmental regulations. They report that in four instances ex ante cost estimates exceeded ex post costs by a factor of two or more, while the ex ante and ex post 5 See Radaelli 2005, however, who notes that "regulatory impact assessments" may still differ significantly from one jurisdiction to another ------- estimates were roughly the same in the fifth case.6 Oosterhuis et al. also report on an earlier study of costs of compliance with Dutch environmental regulations of the first Dutch National Environmental Policy Plan of 1988, as predicted ex ante by Jantzen (1989) and later estimated ex post by RIVM (2001). These Dutch studies were, by the standards of the field, unusually accurate. While the costs of five of the eight regulations considered were overestimated, only one ex ante estimate was as much as twice its ex post realization, and in aggregate the total ex ante estimate of slightly over €12 billion was only 13% higher than the ex post realization. Oosterhuis et al. (2006) credit this unusually accurate performance to the existence of relatively good statistics and studies in the Netherlands. We will conclude this section with summaries of two studies that considered the accuracy of ex ante cost predictions for specific consumer products. Anderson and Sherwood (2002) compare cost estimates for EPA mobile source rules. These include six fuel-quality regulations and eleven vehicle emission standards. In most instances Anderson and Sherwood found that ex ante estimates of price increases induced by regulation were greater than actual price changes observed. They also found, however, that EPA estimates tended to be closer to actual price changes than were industry estimates. Dale, et al. (2009) considered the costs associated with the Department of Energy's efficiency regulations on consumer appliances such as air conditioners, refrigerators, and washing machines. This study illustrates the challenges inherent in developing estimates for the costs of regulation for consumer goods. Dale, et al. derived their ex post cost estimates using hedonic regressions to tease out the separate effects of scale, general technological progress, and more competitive behavior from those of the energy efficiency regulations themselves. Having isolated these effects, the authors found, as have the other studies, that ex ante cost estimates generally exceed those developed ex post. 3. What does the literature show? The studies we have reviewed uniformly find that regulators overestimate the costs of regulatory compliance more often than they underestimate them, and that the ratio of ex ante to ex post compliance cost estimates is, on average, considerably greater than one. While this might seem at first blush to establish that regulators' ex ante estimates of the costs of regulatory compliance are biased upward, this assertion does not actually withstand closer scrutiny. I consider the two types of evidence in turn, and show that neither necessarily reveals a bias in estimates. Skewed distributions One of the most robust findings in the existing literature comparing ex ante to ex post estimates of costs is that the former generally exceed the latter. I am aware of no study in which more ex ante cost estimates were lower than ex post estimates, as opposed to higher, and in many 6 Oosterhuis et al. actually consider six environmental directives, addressing large combustion plants, integrated pollution prevention and control, ozone control, ozone depleting substances packaging, and nitrates, but are unable to develop ex ante compliance cost estimation numbers for the packaging directive. ------- a substantial majority of ex ante estimates were higher than the corresponding ex post estimates. Can we then conclude that ex ante cost estimates are generally biased upward? The answer would appear to be "No," at least not on the basis of this simple observation alone. An estimate is "biased," by the statistical definition of the term, if its expected value differs from the mean of the population from which it is drawn. It is entirely possible that a majority of observations would be below the mean of the distribution of the population from which they are drawn. The median of a distribution will be below the mean if the distribution is skewed, i. e., if it is not symmetrical about its mean. Do we have any reason to suppose that the distribution of costs would not be symmetrical? Yes, there are several. First, total costs are often estimated by multiplying an estimate of unit costs by the number of units affected by the regulation (Harrington, et al. 2000). Both cost-per-unit and units affected are random variables: the researcher cannot observe the former accurately, while absent perfect understanding of market conditions and drivers, the researcher cannot know the latter with certainty. Suppose that the increase in unit costs and the number of units affected are each distributed independently and symmetrically on nonnegative supports. Then their product will be distributed asymmetrically. Heuristically, there is a small probability that unit costs will be large, and a small probability that the number of units affected will be large. Thus, there is a very small probability that the cost of regulation will be very large. Under such conditions the distribution of total costs will have a long right tail, and hence, be asymmetric. A very simple example illustrates the point. Suppose that both the increase in unit costs and the number of affected units are distributed independently and uniformly on the interval [0, 1] (we can always make the supports the same by choice of units of measurement). Then it is easily demonstrated that their product is distributed logarithmically on the interval [0, 1] with probability distribution function - In 0. This function has mean ]A and median of approximately 0.187, and about 59.2% of observations are less than the mean. Another reason to suppose that the distribution of costs is asymmetric is because the mathematical forms that give rise to such costs are often asymmetrically distributed. It can be shown that if production in a regulated industry can be modeled by a constant returns to scale production function of unpriced, but regulated, emissions and purchased inputs, then the reduction in profits that would result if emissions are required to be reduced by an amount Ae may be approximated as CC « ;r-(Ae/e), (1) where Ae/e is the proportional change in allowed emissions and n is industry profit per unit of emissions, and n is independent of the level of emissions. This expression will be approximately true for small changes in allowed emissions or if the affected industry is small enough as to have negligible impact on input and output prices. Suppose that we choose a simple form for the production function; consider, for example, the Cobb-Douglas form ------- f(x,e) = xV~' (2) where x is the quantity of purchased inputs employed in production and e the amount of effluent discharged. If p is the price of output and w the price of the input, x (we treat x as a scalar for simplicity; nothing of consequence would change from treating it as a vector), and we normalize the quantity of emissions discharged in the status quo ante to one, it can be shown that 71 = (l -a)pl a \ — a w) (3) If we treat a,p, and w as unknown random variables with independent symmetric distributions, the resulting function k is asymmetrically distributed. This is not surprising, as the central limit theorem applied to the product, rather than the sum, of independent random variables, implies that the product will be lognormally distributed. A histogram for one such distribution is presented below: 450 400 350 300 250 200 150 100 50 0 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 .~ dflnnnnnn a,p, and w are distributed normally with means of 0.75, 1, and 1, respectively, and standard deviations of 0.05, 0.10, and 0.10, respectively. The expectation of the resultant distribution is 0.120, the median 0.106, and 59.1% of observations are less than the mean. A third reason for supposing that the distribution of costs might be skewed may arise from the attributes of innovation. The story that is often told as to why costs tend to be overestimated is that the people recording estimates tend to discount the possibility of innovation. They do not reflect the high likelihood that much more cost-effective ways of ------- complying with regulation will be identified. It could well be, however, that while very cost- effective strategies are, in fact, identified most of the time, spectacularly costly exceptions could draw the mean cost of regulation considerably higher than the median. Cost ratios The fact that most ex ante estimates of costs are higher than are the corresponding ex post estimates does not necessarily imply that such estimates are biased. In order to make the determination of whether or not estimates are biased, we would need to know whether there are n occasional spectacular exceptions in which ex ante costs severely wwderestimated actual costs. There are certainly instances in which the costs of rules have been underestimated. The problem, though, is that we never have repeated samples from the same distribution. There is, in each case, one rule whose costs are estimated, yielding one ex ante estimate and one ex post estimate. It would require some truly heroic assumptions to say that the fact that costs were wwderestimated on a regulation affecting electric utilities, for example, somehow "offsets" the fact that costs were overestimated on rules affecting automobiles and appliances. Let us suppose, however, that one were prepared to make such heroic assumptions. Many studies note that, for most rules, ex ante estimates are a multiple greater than the corresponding ex post estimates. What might we infer from such observations? Again, the frustrating answer may be "not much". To see why, consider a very simple example. Suppose that a rule is being contemplated, and that with 50% probability costs will be 50, while with 50% probability costs will be 150. In expectation, then, costs would be 100. On average, however, the ratio of predicted ex ante to ex post costs will be Vi ¦ 100/50 + V2 • 100/150 = 1 1/3. Note that the issue here is not the symmetry of the distribution of realized costs. Rather, it is that if ex post costs are low, the ratio of estimated to realized costs explodes, even if the estimation is unbiased. The problem may be even worse if it is made more realistic. It is not unreasonable to suppose that costs of regulation in some instances could be near zero, or possibly even negative. While it is unreasonable to suppose that expected costs of compliance would be negative (if they were, why wouldn't firms make the changes absent regulatory urging?), it is certainly not inconceivable that, once having been induced to consider some innovations, firms would occasionally hit on some solutions that would actually reduce their costs of production and increase their profits (Oosterhuis, et al., 2006 raise the possibility that innovations could result in compliance at negative net cost; OTA 1995 suggests that a couple of OSHA rules might have generated negative compliance costs). It is possible, then, that the ratio of ex ante to ex post estimates would be unbounded. In short, then, while it may sound troubling to read that ex ante cost estimates exceed ex post estimates by substantial multiples, such a finding would not establish that ex ante estimates 7 The 2005 OMB study raises another interesting point that is worth considering in passing. We can never observe the accuracy of cost estimates for rules that were never issued. ------- were unbiased. It could rather, and perhaps counterintuitively, suggest that no statistical o inferences could be drawn on the basis of such data. 4. Evaluating the accuracy of Ex ante cost estimates: An alternative approach In practice, both ex ante and ex post estimates of costs are just that - estimates of random variables whose true values remain unknown either because the rule under contemplation has not yet been enacted, in the case of ex ante estimates, or because we cannot completely and accurately observe all affected entities' costs of in the case of ex post estimates. Somewhat more formally, we could say that any estimate of costs, 9, may be written as E{9 \ Q), where Q is an information set available at the time the estimate is made. At a later time a revised estimate of 9 might be formed based on an updated information set Q\ A convenient way to represent the relationship between an ex ante expectation (i. e., conditioned on the initial information) and an ex post expectation (conditioned on updated information) is that £(#|Q') = £(0|n)+e, (4) where s is a forecasting error. The ex ante estimate will be unbiased if E{s) = 0. Furthermore, the forecasting error s should be uncorrected with the expectation of costs conditioned on the information that is available before the rule is enacted. If it were correlated, knowledge of the correlation could be used to derive a better estimate. If we had a large sample, all of whose values of costs were drawn from the same distribution, and all of whose realizations of conditioning information were drawn from the same distribution, it would be relatively easy to test the hypothesis that s has zero mean. We could simply compare the averages of ex ante and ex post costs. This procedure would be somewhat problematic, in our case, however, as each observation on a pair of ex ante and ex post cost estimates is drawn from a unique experiment. The procedure of adding up ex ante costs estimated from different studies and comparing them with the sum over a corresponding set of ex post costs would mix "apples and oranges", and the resultant comparison of differences in means would be unduly influenced by those observations for which costs were highest. A better procedure would be to specify an empirical version of expression (4): 8 The classic example here is the quotient of two normal distributions, which has a Cauchy distribution. It would be impossible to tell from the ratio of ex ante to ex post estimates if the former were accurate, as the Cauchy distribution has no moments. Of course, it seems unreasonable to suppose that the costs of many regulations could reasonably be defined on a support of [-oo, oo]. ------- £(0,|£V) = a + PE(6, | £!,) + *,, (5) where a and /? are unknown parameters to be determined, the subscript i indexes observations on different prospective regulations, and is a random disturbance term with mean zero and which is uncorrected with the ex post estimate of costs. The null hypothesis to be tested is, then, that a = 0 and /? = 1. I have estimated equation (5) using as data ex ante and ex post cost estimates reported in Harrington, et al. (2000; specific cost data were found in an earlier working paper, Harrington, el al. 1999), MacLeod, et al. (2006), and Oosterhuis, et al. (2006; this study includes both original case studies conducted by the authors and summaries of eight other case studies in which ex ante estimates were developed by Jantzen, et al. (1989) and ex post estimates reported by RIVM As detailed descriptions of the data from these studies is included with each, I will not repeat such descriptions here. I might, however, note, in passing that I was unable to employ nearly as many data points as might be inferred from the numbers of cases considered in the studies. It is rather surprising when one consults the actual studies that clear, consistent, quantitative statements concerning both ex ante and ex post costs are more the exception than the rule. Harrington, et al., for example, cite 28 cases. I use only seven. The others were eliminated for want of quantitative data (either in Harrington, et al. 2000, or the working paper on which it was based, Harrington et al., 1999)9, or because the authors reported only unit-cost estimates which may not be comparable with aggregate estimates (this, incidentally, is why I have not included any cost estimates from Anderson and Sherwood 2002, or Dale, et al., 2009). Similarly, it was possible to derive comparable numbers for ex ante and ex post costs for only three instances in the MacLeod et al. (2006) report, and Oosterhuis, et al. (2006) proved useful only inasmuch as we adopted figures that it reported from Jantzen, et al. (1989) and RIVM (2001). I did not consider studies such as OMB (2005), which compiles estimates from other sources (relying heavily, for example, on Harrington, et al. 2000), or Hodges 1999, which reports industry, rather than regulators', estimates of costs. I decided on a sample of 18 regulations (see Table 1). Six are from the United States, one from Canada, eight from the Netherlands, and three from the United Kingdom. Regrettably - and surprisingly - only one US EPA regulation has clear quantitative estimates of both ex ante and ex post costs corresponding to total (as opposed to unit) effects. Performing the regression indicated in (5) yields the following results: 9 For example, the authors write of the phase-out of lead from gasoline that "There has not been a retrospective analysis of the rule's costs but evidence indicates that EPA's analysts correctly forecast the costs or even overestimated them." While this judgment allowed Harrington et al. to classify this rule among those for which ex ante costs were estimated with reasonable accuracy, it does not allow me to employ the observation in my quantitative procedures. (2001)). E(ei I n,.') = -0.092 + 0.948E(ei I Q,) (0.132) (0.084) (6) ------- 2 R = 0.889 (standard errors in parentheses). In neither specification can I reject the hypothesis that the intercept is zero and slope one, i. e., that ex ante estimates of the costs of regulation are unbiased.10 It would be foolish to try to make too much of these results. Among other potential problems, it is more reasonable to regard the eighteen before-and-after estimates of costs I have used as a convenience sample than as any sort of random draw from the entire universe of cost estimates. In fact, one might suggest that the fact that I do have good before-and-after estimates for these eighteen rules is evidence that they were more carefully analyzed than were the many other rules that have been mentioned in studies comparing ex ante to ex post estimates of costs. 1 Moreover, the observation that I cannot reject the hypothesis that ex ante cost estimates are biased does not imply that such estimates are "good". There is still considerable variation in the sample, as evidence by the fact that the ratio of ex ante to ex post estimates ranges from 0.207 to 11.2. If nothing else, it would appear that cost estimates can be a long way off, in either direction. 5. Conclusions Conclusions were foreshadowed in the introduction, so I will only briefly recapitulate them here. The first is that any presumption that regulatory costs are overestimated, and hence that a more liberal interpretation of cost-benefit tests is warranted, is premature. While I am not aware of any commentator who has made this point explicitly, at least with regard to regulator's, as opposed to industry's estimates of costs, it is certainly the subtext of some comments. Again, such evidence as exists does not support such a procedure. "Such evidence as exists" is, however, sparse. While the authors of existing studies have labored diligently to gather evidence, the evidence remains limited. Moreover, different studies have assembled different data in different ways. While I have tried to compare studies that report similar measures of costs, discrepancies remain between studies as to, e. g., how to include capital investments and variable costs, time periods, discounting, etc. My results can only be considered suggestive at best. Moreover, as other authors have suggested, conducting retrospective studies of the accuracy of ex ante cost estimates remains something of an orphan activity (see, e. g., Hahn and Tetlock 2008). It is understandable that regulators would put a higher priority on predicting the 10 It might reasonably be suggested that the regression reported in equation (6) will be very inefficient, as we might reasonably expect considerable heteroskedasticity: the costs of very costly rules are likely estimated with considerably higher errors than are less costly ones. I also transformed (6) by weighting by ex ante estimates and found again that I could not reject the hypothesis that ex ante estimates were unbiased. 11 Some authors have noted that estimates reported in such studies might not have been chosen at random reasons (see, e. g., Hahn and Tetlock 2008). High-profile regulations, rules for which ex ante predictions were spectacularly inaccurate, or instances illustrating economists' favorite hobby horses (e. g., those allowing allowance trading) might all be more likely to be considered. ------- effects of prospective regulations than they would on evaluating the accuracy of their predictions of regulations that have already been promulgated. It is also understandable that those who have ventured predictions in the past would be reluctant to revisit them: the best possible outcome would be that they would be shown to have done their job competently, while the alternative is that their best efforts would be found lacking. Be that as it may, however, it would certainly be useful to high-level decision makers to know how reliable the information they are receiving is - or at least, how reliable it has been in the past. Ultimately, this information might show why different studies have over- or underestimated costs, and whether the prospect for technological innovation is, in fact, underappreciated. ------- References Anderson, J.F., and Sherwood, T. 2002. Comparison of EPA and Other Estimates of Mobile Source Rule Costs to Actual Price Changes. Paper presented at the SAE Government Industry Meeting, Washington, DC, May 14, 2002. Anonymous. 2011. EPA Launches Study To Improve Cost Estimates For New Regulations. Inside EPA. 31 August. Available online at insideepa.com. Bailey, Peter D., Gary Haq, and Andy Goudson. 2002. Mind the gap! Comparing ex ante and ex post assessments of the costs of complying with environmental regulation. European Environment 12 (5): 245 - 256. Dale, Larry, Camille Antinori, Michael McNeil, James E. McMahon and K. Sydny Fujita. 2009. Retrospective evaluation of appliance price trends. Energy Policy 31,2 (February). 597-605. Goodstein, Eban, and Hart Hodges. 1997. Polluted data: Overestimating environmental costs. American Prospect 8 (35): 64-69. Hahn, Robert, and Rohit Malik. 2004. Is regulation good for you? Harvard Journal of Law and Public Policy 27, 3 (October). 893 - 916. Harrington, Winston, Richard D. Morgenstern, and Peter Nelson. 2000. On the accuracy of regulatory cost estimates. Journal of Policy Analysis and Management 19 (2). 297 - 322. Heinzerling, Lisa. 2002. Markets for arsenic. Georgetown Law Review 90: 2311 -2339/ Hodges, Hart. 1997. Falling prices: Cost of complying with environmental regulations almost always less than advertised. Economic Policy Institute Briefing Paper. Washington, DC: Economic Policy Institute. Jantzen, J. (1989), Costs of Environmental Management, 1988-2010, 3 policy scenarios. Report for the Ministry of VROM, The Hague, 25 May 1989. MacLeod, Michael, Dominic Moran, Manuel Lago Aresti, Winston Harrington, and Richard Morgenstern. 2006. Comparing the ex ante and ex post costs of complying with regulatory change. Final report to DEFRA. London: Department for Environment, Farms, and Rural Affairs. Office of Management and Budget, Office of Information and Regulatory Affairs (OMB). 2005. Validating Regulatory Analysis: 2005 Report to Congress on the Costs and Benefits of Federal Regulations and Unfunded Mandates on State, Local, and Tribal Entities. Washington DC: Government Printing Office. ------- Office of Technology Assessment (OTA). 1995. Gauging Control Technology and Regulatory Impacts in Occupational Safety and Health: An Appraisal of OSHA's Analytic Approach. OTA- ENV-635. Washington DC: Government Printing Office. Porter, Michael. 1991. America's green strategy. Scientific American 264(4). 96. Radaelli, Claudio. 2005. What Does Regulatory Impact Assessment Mean in Europe ? Washington: AEI-Brookings Joint Center for Regulatory Studies. RIVM (2000), Techno 2000; Modellering van de daling van eenheidskosten van technologieen in de tijd. Rapportnummer 773008003, Bilthoven, April 2000. ------- Ex ante cost Ex ante cost Nation (for US, estimates estimates Agency in (millions of US (millions of US parentheses) Rule dollars) dollars) Source Ontario Ontario water 58 51 Harrington, etal. 2000 US (OSHA) Vinyl Chloride 1000 253 Harrington, etal. 2000 US (OSHA) Cotton Dust 280 83 Harrington, etal. 2000 US (OSHA) Occupational Lead 224 20 Harrington, etal. 2000 US (OSHA) Formaldehyde 11 6 Harrington, etal. 2000 US (EPA) S02 Phase 1 764 779 Harrington, etal. 2000 US (OSHA) Ethylene oxide 24 25 Harrington, etal. 2000 UK Control of Major Accidents Hazards 155 416 MacLeod, etal. 2006 UK Food Safety (General Food Hygiene/Butchers' Shops) 5 25 MacLeod, etal. 2006 England The Welfare of Farmed Animals 3 3 MacLeod, etal. 2006 Netherlands Acidification 2620 1248 Jantzen 1989 Netherlands Climate change 617 839 Jantzen 1989 Netherlands Eutrophication 1471 814 Jantzen 1989 Netherlands Hazardous Jantzen 1989 Substances 3465 2738 Netherlands Waste Jantzen 1989 Management 4848 5443 Netherlands Soil sanitation 914 881 Jantzen 1989 Netherlands Disturbance 923 763 Jantzen 1989 Netherlands Other 1939 2140 Jantzen 1989 ------- |