UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON D.C. 20460
OFFICE OF THE ADMINISTRATOR
SCIENCE ADVISORY BOARD
April 11,2013
EPA-SAB-13-002
The Honorable Bob Perciasepe
Acting Administrator
U.S. Environmental Protection Agency
1200 Pennsylvania Avenue, N.W.
Washington, D.C. 20460
Subj ect: SAB Review of EPA's Retrospective Cost Study of the Costs of EPA Regulations:
An Interim Report of Five Case Studies (March 2012)
Dear Acting Administrator Perciasepe:
EPA's Science Advisory Board (SAB) was asked to review and comment on the Retrospective Cost
Study of the Costs of EPA Regulations: An Interim Report of Five Case Studies (March 2012, RCS). The
RCS is composed of five case studies developed by EPA's National Center for Environmental
Economics (NCEE) to investigate how well EPA has predicted the costs of regulatory compliance. The
RCS sought to compare cost estimates ex post (after the regulation) to EPA's ex ante (before the
regulation) predictions. EPA asked the SAB a series of questions on the case study approach applied in
the RCS and specifically on appropriate methodologies to be applied in these and future case studies.
The SAB applauds EPA's efforts to evaluate the ex post costs of regulations and to examine systematic
differences between the ex post and ex ante estimates. Ensuring that costs of regulations are accurately
predicted and assessed is critical for designing cost-effective regulations. The RCS provided some
much-needed attention to the topic of cost of regulatory compliance.
The SAB finds that the case studies in the RCS are useful in showing that there are often significant
divergences in assumptions underlying the ex ante analysis and ex post outcomes regarding methods of
compliance, timing of compliance, and impact of compliance on production and market parameters. A
comprehensive assessment of the ex post costs of regulations is, however, limited by available data. In
addition, there is significant heterogeneity across regulations in terms of the pollution medium, industry
structure and size, timelines, variety of compliance methods and rates of technological change. EPA's
case studies generally showed more evidence of overestimation of ex ante costs than underestimation
although the results cannot be generalized given the small sample size of cases considered and paucity
of data. Based on these case studies, the SAB agrees with the Agency's conclusion that it is not possible
to draw meaningful conclusions regarding general tendencies to under- or over-estimate costs in ex ante
evaluations.
The SAB would like to see EPA broaden its goal beyond the narrow question of whether ex ante costs
are higher or lower than ex post costs. Rather than simply focusing on the question of whether EPA
generally overestimates or underestimates costs, the SAB recommends a focus on the drivers of the
-------
differences in ex post and ex ante costs so that insights can be gained that would improve ex-ante cost
estimation in the future. To do this, a more systematic approach needs to be applied across case studies.
This approach should be based on an overarching conceptual framework that provides a foundation for
the RCS and explains the various reasons that could lead to differences between ex post and ex ante
costs. This framework can be used to develop hypotheses whose validity can be examined in specific
cases. For example, one hypothesis could be that EPA regulations induce innovation in industry and
drive down costs of compliance over time. Case studies can examine the extent to which this hypothesis
is supported for various regulations. EPA can then use this insight to change their assumptions about
technological change in ex-ante cost of compliance estimation.
More case studies are needed. Reliance on a small number of case studies is not adequate to support the
task. EPA's sample size will need to extend far beyond the 5 case studies completed and 5 case studies
planned. These case studies should be selected randomly rather than be ones where differences are
expected to be large or more data is available in order to avoid selection bias and to allow for more
generalizable conclusions of the causes of differences between ex post and ex ante costs. The SAB
recommends that EPA do more, but shorter qualitative analyses of regulation, focusing more on
qualitative factors with the goal of presenting well-formed and documented 'stories' that identify the
large drivers of differences between ex ante and ex post results.
More data are needed. The Agency does not typically collect information on costs of compliance ex
post. As a result, EPA was limited to publicly available data and collaboration with industry experts. If
the Agency is to draw meaningful lessons for future ex ante cost estimates as well as for the direction
and extent to which ex post cost estimates diverge from ex ante estimates, it must go beyond the existing
data sources and pursue a primary data collection approach using surveys, focus groups and broad based
expert opinions as well as consult with the outside research community. The SAB recommends that
EPA consider ways of building in a routine ex post data collection effort either within the Agency or in
collaboration with other agencies and strongly urges the Agency to invest the resources needed to do a
proper evaluation. This data would be useful to improve ex ante cost analysis and increase the cost-
effectiveness of future regulations with potential savings in the costs of compliance.
The enclosed SAB report offers specific suggestions for how EPA's methodology might be improved.
We appreciate the opportunity to provide advice on this important subject and look forward to the
Agency's response.
Sincerely,
"r\
Dr. David T. Allen, Chair Dr. Madhu Khanna, Chair
EPA Science Advisory Board SAB Environmental Economics Advisory
Committee
Enclosure
-------
U.S. Environmental Protection Agency
Science Advisory Board
Environmental Economics Advisory Committee
CHAIR
Dr. Madhu Khanna, Professor, Department of Agricultural and Consumer Economics, University of
Illinois at Urbana-Champaign, Urbana, IL
MEMBERS
Dr. Nicholas Flores, Professor, Department of Economics, Arts and Sciences, University of Colorado,
Boulder, Boulder, CO
Dr. Wayne Gray, Professor, Department of Economics, Clark University, Worcester, MA
Dr. John List, Professor, Department of Economics, University of Chicago, Chicago, IL (did not
participate in this review)
Dr. Karen Palmer, Senior Fellow, Resources for the Future, Washington, DC
Dr. George Parsons, Professor, Department of Economics, College of Marine Studies, University of
Delaware, Newark, DE
Dr. James Shortle, Professor, Agricultural Economics and Rural Sociology, Pennsylvania State
University, University Park, PA
Dr. Laura Taylor, Professor, Department of Agricultural and Resource Economics, North Carolina
State University, Raleigh, NC
Dr. Peter J. Wilcoxen, Associate Professor, Economics and Public Administration, Syracuse
University, Syracuse, NY
Dr. JunJie Wu, Emery N. Castle Professor of Resource and Rural Economics, Department of
Agricultural and Resource Economics, Oregon State University, Corvallis, OR
Dr. Jinhua Zhao, Professor, Department of Economics, Department of Agricultural, Food and
Resource Economics, Michigan State University, East Lansing, MI
Dr. David Zilberman, Professor, Agriculture Resource Economics Department, College of Natural
Resources, University of California - Berkeley, Berkeley, CA
SCIENCE ADVISORY BOARD STAFF
Dr. Holly Stallworth, Designated Federal Officer, U.S. Environmental Protection Agency, Science
Advisory Board, Washington, DC
-------
U.S. Environmental Protection Agency
Science Advisory Board
CHAIR
Dr. David T. Allen, Gertz Regents Professor of Chemical Engineering and the Director of the Center
for Energy and Environmental Resources, The University of Texas, Austin, TX
MEMBERS
Dr. George Alexeeff, Director, Office of Environmental Health Hazard Assessment, California
Environmental Protection Agency, Oakland, CA
Dr. Pedro Alvarez, Department Chair and George R. Brown Professor of Engineering, Department of
Civil & Environmental Engineering, Rice University, Houston, TX
Dr. Joseph Arvai, Svare Chair in Applied Decision Research, Institute for Sustainable Energy,
Environment, & Economy, Haskayne School of Business, University of Calgary, Calgary, Alberta,
Canada
Dr. Thomas Burbacher, Professor, Department of Environmental and Occupational Health Sciences,
School of Public Health, University of Washington , Seattle, WA
Dr. Ingrid Burke, Director, Haub School and Ruckelshaus Institute of Environment and Natural
Resources, University of Wyoming, Laramie, WY
Dr. Thomas Burke, Professor and Jacob I. and Irene B. Fabrikant Chair in Health, Risk and Society
Associate Dean for Public Health Practice, Johns Hopkins Bloomberg School of Public Health, Johns
Hopkins University, Baltimore, MD
Dr. Edward T. Carney, Departmental Senior Science Leader and Director of Predictive Toxicology
Center, Toxicology & Environmental Research and Consulting, The Dow Chemical Company, Midland,
MI
Dr. Terry Daniel, Professor of Psychology and Natural Resources, Department of Psychology, School
of Natural Resources, University of Arizona, Tucson, AZ
Dr. George Daston, Victor Mills Society Research Fellow, Global Product Stewardship, The Procter &
Gamble Company, Mason, OH
Dr. Costel Denson, Managing Member, Costech Technologies, LLC, Newark, DE
Dr. Otto C. Doering III, Professor, Department of Agricultural Economics, Purdue University, W.
Lafayette, IN
Dr. Michael Dourson, President, Toxicology Excellence for Risk Assessment, Cincinnati, OH
11
-------
Dr. Joel Ducoste, Professor, Department of Civil, Construction, and Environmental Engineering,
College of Engineering, North Carolina State University, Raleigh, NC
Dr. David A. Dzombak, Walter J. Blenko, Sr. University Professor of Environmental Engineering,
Department of Civil and Environmental Engineering, College of Engineering, Carnegie Mellon
University, Pittsburgh, PA
Dr. T. Taylor Eighmy, Vice Chancellor for Research and Engagement, University of Tennessee,
Knoxville, TN
Dr. Elaine Faustman, Professor and Director, Environmental and Occupational Health Sciences,
University of Washington, Seattle, WA
Dr. R. William Field, Professor, Department of Occupational and Environmental Health, and
Department of Epidemiology, College of Public Health, University of Iowa, Iowa City, IA
Dr. H. Christopher Frey, Distinguished University Professor, Department of Civil, Construction and
Environmental Engineering, College of Engineering, North Carolina State University, Raleigh, NC
Dr. John P. Giesy, Professor and Canada Research Chair, Veterinary Biomedical Sciences and
Toxicology Centre, University of Saskatchewan, Saskatoon, Saskatchewan, Canada
Dr. Barbara L. Harper, Risk Assessor and Environmental-Public Health Toxicologist, and Division
Leader, Hanford Projects, and Program Manager, Environmental Health, Department of Science and
Engineering, Confederated Tribes of the Umatilla Indian Reservation (CTUIR), West Richland, WA
Dr. Cynthia M. Harris, Director and Professor, Institute of Public Health, Florida A&M University,
Tallahassee, FL
Dr. Robert Johnston, Director of the George Perkins Marsh Institute and Professor, Economics, Clark
University, Worcester, MA
Dr. Kimberly L. Jones, Professor and Chair, Department of Civil Engineering, Howard University,
Washington, DC
Dr. Bernd Kahn, Professor Emeritus and Associate Director, Environmental Radiation Center, Georgia
Institute of Technology, Atlanta, GA
Catherine Karr, Associate Professor - Pediatrics and Environmental and Occupational Health Sciences
and Director - NW Pediatric Environmental Health Specialty Unit, University of Washington, Seattle,
WA
Dr. Madhu Khanna, Professor, Department of Agricultural and Consumer Economics, University of
Illinois at Urbana-Champaign, Urbana, IL
Dr. Nancy K. Kim, Senior Executive, Health Research, Inc., Albany, NY
in
-------
Dr. Francine Laden, Mark and Catherine Winkler Associate Professor of Environmental
Epidemiology, Harvard School of Public Health, and Channing Division of Network Medicine, Brigham
and Women's Hospital and Harvard Medical School, Boston, MA
Dr. Cecil Lue-Hing, President, Cecil Lue-Hing & Assoc. Inc., Burr Ridge, IL
Dr. Elizabeth Matsui, Associate Professor, Pediatrics, School of Medicine, Johns Hopkins University,
Baltimore, MD
Dr. Surabi Menon, Director of Research, ClimateWorks Foundation, San Francisco, CA
Dr. James R. Mihelcic, Professor, Civil and Environmental Engineering, University of South Florida,
Tampa, FL
Dr. Christine Moe, Eugene J. Gangarosa Professor, Hubert Department of Global Health, Rollins
School of Public Health, Emory University, Atlanta, GA
Dr. Horace Moo-Young, Dean and Professor, College of Engineering, Computer Science, and
Technology, California State University, Los Angeles, CA
Dr. Eileen Murphy, Director of Research and Grants, Ernest Mario School of Pharmacy, Rutgers
University, Piscataway, NJ
Dr. James Opaluch, Professor and Chair, Department of Environmental and Natural Resource
Economics, College of the Environment and Life Sciences, University of Rhode Island, Kingston, RI
Dr. Duncan Patten, Director, Montana Water Center, and Research Professor, Hydroecology Research
Program, Department of Land Resources and Environmental Sciences, Montana State University,
Bozeman, MT
Dr. Martin Philbert, Dean and Professor, Environmental Health Sciences, School of Public Health,
University of Michigan, Ann Arbor, MI
Dr. Stephen Polasky, Fesler-Lampert Professor of Ecological/Environmental Economics, Department
of Applied Economics, University of Minnesota, St. Paul, MN
Dr. C. Arden Pope, III, Professor, Department of Economics, Brigham Young University, Provo, UT
Dr. Stephen M. Roberts, Professor, Center for Environmental and Human Toxicology, University of
Florida, Gainesville, FL
Dr. Amanda Rodewald, Professor of Wildlife Ecology, School of Environment and Natural Resources,
The Ohio State University, Columbus, OH
Dr. James Sanders, Director and Professor, Skidaway Institute of Oceanography, Savannah, GA
Dr. William Schlesinger, President, Gary Institute of Ecosystem Studies, Millbrook, NY
iv
-------
Dr. Gina Solomon, Deputy Secretary for Science and Health, Office of the Secretary, California
Environmental Protection Agency, Sacramento, CA
Dr. Daniel O. Stram, Professor, Department of Preventive Medicine, Division of Biostatistics,
University of Southern California, Los Angeles, CA
Dr. Peter S. Thorne, Director, Environmental Health Sciences Research Center and Professor and
Head, Departmment of Occupational and Environmental Health, College of Public Health, University of
Iowa, Iowa City, IA
Dr. Paige Tolbert, Professor and Chair, Department of Environmental Health, Rollins School of Public
Health, Emory University, Atlanta, GA
Dr. Jeanne VanBriesen, Professor, Department of Civil and Environmental Engineering, Carnegie
Mellon University, Pittsburgh, PA
Dr. John Vena, University of Georgia Foundation Professor in Public Health and
Head, Department of Epidemiology and Biostatistics, Georgia Cancer Coalition Distinguished Scholar,
College of Public Health , University of Georgia, Athens, GA
Dr. R. Thomas Zoeller, Professor, Department of Biology, University of Massachusetts, Amherst, MA
SCIENCE ADVISORY BOARD STAFF
Dr. Angela Nugent, Designated Federal Officer, U.S. Environmental Protection Agency, Science
Advisory Board, Washington, DC
-------
NOTICE
This report has been written as part of the activities of the EPA Science Advisory Board (SAB), a
public advisory group providing extramural scientific information and advice to the Administrator and
other officials of the Environmental Protection Agency. The SAB is structured to provide balanced,
expert assessment of scientific matters related to problems facing the agency. This report has not been
reviewed for approval by the agency and, hence, the contents of this report do not necessarily represent
the views and policies of the Environmental Protection Agency, nor of other agencies in the Executive
Branch of the Federal government, nor does mention of trade names of commercial products constitute a
recommendation for use. Reports of the SAB are posted on the EPA website at http://www.epa.gov/sab.
VI
-------
TABLE OF CONTENTS
Acronyms and Abbreviations viii
1. EXECUTIVE SUMMARY 1
2. INTRODUCTION 5
3. RESPONSES TO EPA's CHARGE QUESTIONS 6
3.1. EVALUATING THE METHODOLOGY 6
3.1.1. Literature Review 6
3.1.2. Potential Reasons Ex Ante and Ex Post Cost Estimates Differ 7
3.1.3. Case Study Methodology 8
3.2. EVALUATING THE CASE STUDY APPROACH 12
3.2.1. Data Challenges 12
3.2.2. Analytic Challenges 13
3.2.3. Weight of Evidence Conclusions 14
3.3. MOVING FORWARD 15
3.3.1. Using Ex Post Cost Comparisons to Inform the Agency 15
3.3.2. General Statements on the Bias of Ex Ante Estimates 17
3.3.3. Rules for Evaluation 18
3.3.4. Building a Database 20
REFERENCES 22
APPENDIX A: CHARGE QUESTIONS A-l
vn
-------
Acronyms and Abbreviations
AF&PA - American Forests and Paper Association
CAAA - Clean Air Act Amendments
CUE - Critical Use Exemption
MACT - Maximum Available Control Technology
MBr - Methyl Bromide
MBTOC - Methyl Bromide Technical Options Committee
NCASI - National Council for Air and Stream Improvement
NCEE - National Center for Environmental Economics
PACE - Pollution Abatement Costs and Expenditures
PM - Paniculate Matter
RCS - Retrospective Cost Study
RIA - Regulatory Impact Analysis
SAB - Science Advisory Board
SEC - Securities and Exchange Commission
Vlll
-------
1. EXECUTIVE SUMMARY
The EPA routinely performs benefit-cost analysis on major environmental regulations. Historically, the
benefits portion of these analyses has received the most attention given the need to put monetary values
on unpriced goods and services such as clean air or clean water. There is a large literature in
environmental economics devoted to "nonmarket valuation" or the monetization of unpriced
environmental goods. There is also a large literature examining the cost-effectiveness and environmental
effects of market-based regulations relative to command-and-control or technology-based regulations.
While these are clearly important, it is also critical to have the most accurate possible estimates of the
costs of specific regulations in order to comply with the Presidential Executive Orders to carry out cost-
benefit analysis, assess the impact of regulations on the regulated community and design cost-effective
regulations in the future. However, there are relatively few such studies that estimate the realized costs
of existing regulations, largely due to lack of easily accessible data. Toward that end, the EPA's
National Center for Environmental Economics (NCEE) has developed a White Paper comparing its own
ex ante (before the regulation) cost analyses to an ex post (after the regulation) evaluation. In its
Retrospective Cost Study of the Costs of EPA Regulations: An Interim Report of Five Case Studies
(March 2012, RCS), the EPA used a case study approach to address the question of whether ex ante and
ex post estimates differ by a substantial degree. When a substantial difference exists, the agency sought
to identify any systematic reasons for the differences in order to improve future ex ante cost modeling.
The SAB was asked to review this interim EPA report, provide technical advice and make
recommendations on how to improve the methodologies for cost analysis. In conducting this review the
SAB received a briefing from the NCEE staff and heard public comments from various industry and
environmental interest groups.
Evaluating the Retrospective Cost Study Methodology (Charge Questions 1-3)
The RCS begins with a review of existing studies that examine the accuracy of ex ante cost estimates.
As described in the RCS, these studies find that costs are more often overestimated than underestimated
and identify some of the major reasons for the divergence of ex post and ex ante costs. The potential
reasons ex ante and ex post estimates differ are described in the RCS but these factors are not analyzed
in a systematic taxonomy that could provide a consistent structure throughout the study. Some factors
will affect ex ante estimates differently from others. Strategic misreporting by firms, for example, would
tend to raise ex ante estimates whereas induced innovation tends to lower actual (ex post) costs.
The SAB recommends that EPA develop a conceptual framework that can be used consistently
throughout the case studies. This conceptual framework would set out the various reasons that could
lead to differences between ex post and ex ante costs, and would incorporate the various drivers of costs,
providing a foundation for the literature review and the case studies that follow. The framework could
be used to identify the key components of compliance costs relevant to a regulation and the factors
likely to lead to each of those components being larger or smaller in the ex post assessment. For each
regulation, the EPA should categorize the ex ante expectations about compliance costs (compliance
methods, key factors affecting compliance costs for each method, and distribution of methods across the
industry), and identify any ex post deviations from those expectations. This approach can then be used to
undertake a systematic assessment of whether each of the components turned out to be larger or smaller
than the ex ante estimate and the characteristics of the regulation that could influence the divergence
1
-------
between ex ante and ex post costs. Having a similar framework for all case studies would enhance
comparability of outcomes across different cases and lead to generalizable conclusions that could be
used to improve ex-ante cost estimation.
Interestingly, the RCS recognized possible incentives for regulators to understate costs, but there may
also be an incentive to overstate costs when benefits are extremely high. In fact, as acknowledged in one
case study, EPA overestimated costs "in order to avoid preparing estimates that were too low and could
be challenged."1 Rather than tailoring a cost estimate to satisfy other objectives, cost estimates should
be based on the best possible science with assumptions and uncertainties clearly stated. If EPA has the
discretion to choose a regulation based on net benefits, less accurate cost estimates will lead to sub-
optimal regulations. Ex ante cost estimates that are too low can lead to regulations that are too stringent.
Ex ante estimates that are too large can lead to regulations that are insufficiently protective.
Evaluating the Case Study Approach (Charge Questions 4-6)
EPA selected ten rules for inclusion in the RCS, the first five of which are profiled in "Phase 1" of the
RCS. EPA limited its study to publicly available information; however for four of the five case studies
in Phase 1, there was insufficient information in the public domain to estimate ex post costs. Thus EPA
had to rely on industry experts, some of whom participated in the ex ante cost analysis. For the next five
case studies, EPA plans to go beyond publicly available data, conducting site visits and exploring the
possibility of comprehensive surveys of the affected industries. These efforts can provide better
quantitative measures of ex post costs, but to improve comparability of results across rules, the studies
should also include a coordinated qualitative approach with a systematic description of the ex ante
expected compliance techniques and costs along with any deviations observed ex post.
In the long run, developing an extensive and authoritative set of studies comparing ex ante and ex post
analysis will require additional time and resources and involvement with the outside research
community. One major contribution towards conducting better ex post analysis would be the resumption
of the Pollution Abatement Costs and Expenditures (PACE) survey conducted by the Census Bureau,
which can contribute to both qualitative and quantitative analysis. When conducting future ex ante
cost analyses, the SAB recommends that the EPA consider ways to build a routine effort to
organize ex post data collection, either within the agency or in collaboration with other agencies.
One source of ex post information could be regulatory staff (from both EPA and state agencies) involved
in monitoring and enforcing compliance with specific regulations. In addition to the PACE survey, the
agency should explore the possibility of working with data-collecting agencies to add questions related
to costs of compliance to existing surveys (e.g., the surveys of costs of agricultural enterprises taken by
the National Agricultural Statistics Service). The SAB strongly urges the EPA to allocate the additional
resources needed to support ex post analyses.
The SAB generally agrees with the conclusions offered in the five case studies. The five case studies
illustrate enormous heterogeneity in regulations — in terms of the pollution medium, industry structure
and size, timelines, variety and availability of compliance methods, rates of technological change and
the role of prices. This, of course, made it difficult to render a meaningful judgment about whether ex
ante costs are generally higher or lower than ex post costs. All case studies were bedeviled by a paucity
of data on ex post costs, thus the EPA utilized a variety of sources to construct ex post costs and made
reasonable judgments in the face of severe data limitations.
1 RCS, p. 53.
-------
The EPA's five case studies yielded disparate conclusions. In the arsenic case study of the effects of
EPA's Maximum Contaminant Level (MCL) for arsenic in drinking water, the agency concluded that it
is not possible to render a judgment about how the EPA's ex ante cost estimates compare to ex post
realizations. After examining the effects of the agency's restrictions on the use of Methyl Bromide
(MBr) in agriculture, the EPA concluded that its ex ante estimate of the cost of the MBr phase-out
exceeded ex post estimates by more than 25 percent for the 2006 to 2010 growing seasons. For the
Maximum Available Control Technology (MACT) II rule enacted to control toxic air emissions from
pulp and paper mills, EPA found that it overestimated the cost of capital investment by roughly 25
percent and overestimated annual costs by nearly five times. For its 1998 rule mandating emissions
reductions in locomotives, the EPA found that the total costs of bringing switch locomotives into
compliance was likely lower than anticipated while its ex post assessment of the costs of bringing line-
haul locomotives into compliance was inconclusive. In each of these case studies, the EPA needed more
data, time and resources.
For the Cluster rule designed to reduce chlorine emissions to water bodies from pulp and paper mills, the
EPA found that it over-estimated the capital cost by 30 to 100 percent depending on the choice of
baseline year. One analytic issue that arose with this rule is the choice of a timeline for estimating the
cost of compliance. The EPA chose not to include industry's voluntary spending that occurred prior to
the promulgation of the regulation; however, there is evidence of large expenditures by the pulp and
paper industry to eliminate dioxin from air and water emissions prior to the official promulgation of the
Cluster rule. The SAB concludes that this pre-regulatory, voluntary spending should be counted in the
agency's cost estimates, both ex ante and ex post, if there is clear evidence that it was undertaken in
response to impending regulations.
Moving Forward (Charge Questions 7-10)
The SAB did not consider it possible to establish a weight of evidence determination on the direction of
ex post costs compared to ex ante costs based on the lack of data and small number of case studies
examined. Focusing only on regulations for which data are available instead of a randomly selected
representative sample of regulations will make it problematic to reach generalizable conclusions about
the extent to which ex ante costs are over-estimates or under-estimates of ex post costs. To advance the
RCS and draw lessons learned for future ex ante analyses, the EPA needs a larger sample size to identify
patterns in the data. The SAB recommends that EPA do more, but shorter qualitative analyses of
randomly sampled regulations, focusing more on qualitative factors as well as the drivers of
differences in ex ante and ex post cost estimates. Note that the SAB is suggesting "qualitative"
analyses in the sense that they do not yield bottom-line total cost estimates, completeness across all cost
categories, or flesh out details on each cost component. Instead, they leave the analyst with the
flexibility to discover and present well-formed and documented 'stories' of the methods used by
industry to achieve compliance, the time line of these compliance efforts and any accompanying
technological change. This approach would help to identify the differences between ex ante assumptions
and ex post behavior and the large drivers of differences between ex ante projections of costs and ex
post realizations. After doing a substantial number of these qualitative studies, the deviations from ex
ante expectations could be examined to look for patterns in the types of deviations observed across
different settings (by pollution medium, by industry size, by the time lag until implementation, by the
key factors affecting costs, by the amount of resources devoted to the analysis). If certain types of
deviations are very common (e.g., always underestimating the rate of technological change in
regulations with a long time lag), future ex ante analyses could take that into account. Trying to rely on
3
-------
the relatively detailed quantitative analyses conducted in Phase 1 and anticipated for Phase 2 seems less
likely to yield generalizable results. The SAB recommends that the goal of this study should be to
improve the accuracy of ex ante cost estimation rather than to quantify the magnitude of the
differences between ex ante and ex post costs per se. This can be achieved by focusing on the
drivers or factors that influence the accuracy of ex ante cost estimates and identifying and
categorizing the reasons for any deviations in ex post costs from the expected components of
compliance costs. EPA should seek to understand which factors could have been anticipated and what,
if any, adjustments to ex ante cost approaches should be made to provide better predictions of actual
costs of compliance.
To summarize, the SAB offers these recommendations for the RCS:
• Develop a conceptual framework that includes the taxonomy of key factors that result in a
divergence of ex ante and ex post cost estimates and provides a consistent structure throughout
the study.
• Use a large randomly sampled set of regulations for the ex post cost estimation and conduct a
qualitative/semi-quantitative analysis of the key factors that can explain the difference in ex ante
and ex post costs
• Consider ways to collect ex post cost information routinely, through surveys, focus groups,
expert assessments and consultation with the Research Community while avoiding sample
selection bias.
• Focus on the drivers or factors that influence the accuracy of ex ante cost estimates rather than
on ascertaining whether the weight of evidence supports a hard line on the magnitude of the
difference between ex ante and ex post costs of compliance.
-------
2. INTRODUCTION
The U.S. Environmental Protection Agency's National Center for Environmental Economics (NCEE)
has launched an effort to evaluate the accuracy of the EPA's ex ante estimates of the costs of
environmental regulation as part of its mission to provide the agency with guidance on analyzing the
benefits, costs, and economic impacts of regulations and policies. In its draft Retrospective Study of the
Costs of EPA Regulations: An Interim Report of Five Case Studies (RCS, March 2012), the EPA sought
to determine whether a "weight of evidence" determination could be made about whether the agency's
ex ante costs are higher or lower than ex post costs. The EPA used a case study approach to identify
differences between ex post and ex ante compliance cost estimates. The RCS summarizes the EPA's
initial findings from a small set of pilot case studies that attempt to evaluate the costs of EPA's
regulations. The initial set of case studies rely on a variety of methods for collecting ex post information
with the aim of examining key drivers of compliance costs to see if informed judgments (weight-of-
evidence) can be made about whether ex post costs are generally higher or lower than the estimates of ex
ante costs. The RCS focused on examining aggregate estimates of the costs of regulations (ex post and
ex ante) rather than a fine-grained comparison of the causes of differences between ex ante and ex post
estimates. The case studies covered air and water regulations on the pulp and paper industry; a pesticide
regulation that affected strawberry farms; arsenic limits for drinking water; and emissions standards for
locomotives.
The SAB was asked to review this interim report and provide advice on the methodology applied in the
case studies and offer potential improvements for future case studies. The agency sought the SAB's
advice on potential improvements in the way in which cost analyses are done. The SAB's
Environmental Economics Advisory Committee (EEAC), a standing committee of the SAB composed
of economists, discussed the RCS in a series of teleconferences (April 19 - 20, 2012; July 12, 2012; and
September 7, 2012). The EEAC received written and oral comments from the American Forests & Paper
Association, American Petroleum Institute, American Water Works Association, International Paper and
the National Council for Air and Stream Improvement as well as presentations from EPA's NCEE. On a
public teleconference on March 7, 2013, the chartered SAB deliberated on and approved the EEAC draft
report subject to clarifying edits.
This Advisory responds to the EPA's charge questions found in Appendix A. Highlights of the SAB
advice are provided in the Executive Summary and detailed responses to the individual charge questions
are provided in the body of the report.
-------
3. RESPONSES TO EPA's CHARGE QUESTIONS
3.1. Evaluating the Methodology
3.1.1. Literature Review
Charge Question 1: Section 2 of the report summarizes existing retrospective cost studies. Has EPA
adequately summarized the existing literature or mischaracterized it in some way? Is the SAB aware of
any additional studies not captured in the interim report? Have we captured key take away messages
offered by this literature? How can this discussion be improved?
Section 2 of the RCS summarizes the existing literature and main results showing that regulatory
compliance costs are more often overestimated than underestimated. Section 2 also presents studies that
actually show that the ex ante costs are higher than the ex post costs. The emphasis on studies of Title
IV of the 1990 Clean Air Act Amendments (CAAA) was especially appropriate because it identified the
main forces that affect the adjustment to regulations and the causes of deviation from ex ante to ex post.
The study by Popp (2003) is especially insightful and an excellent reference.
The RCS covered most of the relevant literature and summarized its findings about whether ex ante
costs exceeded or were lower than ex post costs. It also described the key reasons for these differences.
However, the literature review was not based on a conceptual framework that would have set the stage
for a more systematic analysis of why ex post costs could differ from ex ante costs. An improved
literature search could be used to identify hypotheses for the key drivers of these differences.
The SAB recommends that the EPA take a broader perspective on this question and begin with a
conceptual framework. In principle, ex ante studies estimate something that will happen in the future
and as such, an ex ante estimate is the sum of biases reflecting uncertainty about the future (timing bias)
plus a random measurement error term. If the ex post estimate is the sum of the true cost plus a random
measurement error, then we can disaggregate the difference between the ex ante and ex post estimates
into a bias term plus a random error term. Then, the literature review can be used to understand any
systematic biases and random effects that may be causing the difference. Such an understanding could
lead to improvements in future cost studies.
A richer analysis would be achieved by first forming a hypothesis regarding the differences between ex
ante and ex post, then reviewing the literature in this light. As discussed in the RCS and in the next
section, there are several hypotheses about the causes of differences between ex ante and ex post
estimates. One such hypothesis is strategic behavior by regulated agents. Since regulated agents provide
information that is used to estimate the ex ante costs, the regulated agents may report higher than
expected costs in order to influence the regulator's decision. The other hypothesized cause is
technological change, a phenomenon that is frequently underestimated, especially in response to
regulation. Other issues that may result in bias are related to issues of enforcement and compliance and
better understanding of the structure of the problem. Exploring these hypotheses before surveying the
literature can guide the analysis.
One study that is quite insightful from Europe that was not mentioned is by Bailey et al. (2002). This
study raises the issue of strategic behavior as well as adjustment in terms of technology, innovation and
implementation that can drive a wedge between ex ante and ex post costs. Another study that should be
scrutinized is the National Research Council's Review of the EPA 's Economic Analysis of Final Water
6
-------
Quality Standards for Lakes and Flowing Waters in Florida (NRC 2012) which provided a very explicit
framework for comparing the future time paths of costs at each stage of implementation of the water
quality standard.
Other studies that were overlooked include Boardman et al. (1994), Dayton (1998), Grosse et al. (2005),
LaFrance and de Goiter (1985), Lindner and Jarrett (1978), Rideout and Omi (1995) and Sappington and
Stiglitz (1987).
3.1.2. Potential Reasons Ex Ante and Ex Post Cost Estimates Differ
Charge Question 2: Section 3 of the report briefly describes potential reasons ex ante and ex post
estimates might differ. Has EPA accurately described the various hypotheses? If not, how can this
discussion be improved? Are there other hypotheses that should be included or considered?
The RCS does a good job amassing and explaining the variety of reasons for discrepancies between ex
ante and ex post cost estimates. It lists such factors as industry incentives to overstate their costs in order
to influence policy in their favor, induced innovation after implementation of regulations, changing
market conditions over time such as the reduction in costs of transporting low sulfur coal in SC>2
regulation, etc. The RCS describes possible regulator incentives to understate costs but it should also
acknowledge that regulators may face incentives to overstate costs when benefits are extremely high. A
high ratio of benefits to costs can reduce the regulators' incentive to precisely estimate compliance costs
given that the economic feasibility of the rule is ensured. As the EPA moves forward in its next phases
of the RCS, it will be important to uncover and state incentives faced by the regulator. Interestingly, the
EPA acknowledges its own data manipulation in one case study: "EPA was "conservative in its cost
estimates, in order to avoid preparing estimates that were too low and could be challenged." Economic
analysis should not be manipulated to satisfy other objectives.
While the RCS describes the factors that can cause wedges between ex ante and ex post estimates, it did
not attempt to group or classify these factors, nor did it suggest how these factors can bias the estimates.
For some factors, the direction of the ex ante and ex post differences can go both ways, but for others,
there is a strong theoretical (and sometimes empirical) argument for the differences to go one direction
only. It might be easier to "correct for" some differences but not for others. An alternative classification
scheme to highlight these features is suggested below.
• Ex ante and ex post differences due to inherent uncertainties. Fundamentally, ex post costs are
dependent on realizations of particular values of random variables, while ex ante estimates
represent the best effort to estimate the distribution of these random variables. As such,
differences, even large differences, are genuine features of imperfect information and are hard to
correct. This kind of difference can go both ways, but if the ex ante distribution is skewed, the ex
post realizations are asymmetrically distributed around the ex ante mean, leading to a higher
probability of the ex post realization above or below the mean depending on the direction of the
skewness.
• Differences arising from asymmetric information and strategic (mis-)reporting. As
acknowledged in the RCS3, regulated industries have incentive to strategically misreport their
2/RCS. p. 53.
3/RCS, p. 25.
-------
private information, specifically to over-report compliance costs rather than under-report. Over-
reporting is more likely and of a higher magnitude when (i) more is at stake for the industries,
(ii) the firms are more effectively organized and are more homogeneous, (iii) the degree of
information asymmetry is high (e.g., lack of correlated information from objective third parties),
and (iv) opposing groups (such as consumer groups) are more pronounced in voicing their
opinions. (Note that the four kinds of factors provide, at least theoretically, guidelines about the
drivers for misreporting and possible ways of correction such as utilizing results from the
mechanism design literature.)
• Differences due to firm responses to regulation. This includes induced innovation as well as
other gradual responses such as substitution toward different inputs. The endogenous firm
responses will make ex post costs lower than ex ante estimates.
• Differences due to exogenous shocks. They include exogenous changes in market conditions, in
(related but separate) regulations, and in natural conditions after the implementation of the
regulation. Ex ante and ex post differences can go either way. Although efforts can be made to
better assess future changes, such shocks are difficult to capture fully in ex ante cost estimation.
• Differences due to inconsistent assumptions between the ex ante and ex post estimates about the
timeline for implementation of a regulation, about the rate of compliance to regulations by the
regulated parties, and the form and rate of technological innovation and adoption of compliance
technologies.
• Differences due to the inability to perfectly forecast all types of costs that may be incurred. This
could lead to an incomplete inclusion of relevant costs or the inclusion of irrelevant costs in the ex
ante cost estimation.
3.1.3. Case Study Methodology
Charge Question 3: One goal of the EPA study is to demonstrate the use of different methodologies for
obtaining ex post information on key drivers of compliance cost. While the level of coverage and detail
does not match what is typically used in an ex ante analysis in support of a rule, the purpose is to gather
enough evidence on key drivers to establish a weight of evidence determination on the direction of ex
post costs compared to ex ante costs. These approaches are briefly described in section 4 of the report.
Are some approaches more defensible than others? If so, which ones and why? Which of the
methodologies are more likely to yield reliable results and why? Or are the merits of each method
dependent upon characteristics of the case study to which they are being applied? If some
methodologies are more applicable under specific contexts or rule settings, please identify and describe.
Are there other methodologies that should be considered for ex post cost analyses?
In some cases, it may be possible to find a dataset containing sufficient information for an ex post
statistical analysis that would identify the costs of a new regulation, but this usually will not be possible.
Two major obstacles are the reluctance of firms to provide detailed data on production costs and the
likelihood that other factors besides the regulation have changed. In the locomotive case study, firms
were reluctant to cooperate by providing data on costs, and there were major changes in industry
demand and diesel fuel prices.
It is important for an ex post analysis of regulatory costs to take a systematic approach to the process.
Start with a "story" about the ways in which the industry was expected to achieve compliance (based on
information from the ex ante regulatory analysis) and connect this story to the various components that
went into the ex ante cost calculations. This provides a "bottom up" calculation for total industry costs.
Then identify ways in which the actual industry experience was similar to or different from the ex ante
-------
expectations, including an assessment of how much of an impact (positive or negative) each difference
would have had on compliance costs. This could include a discussion of the heterogeneity of responses
among the firms in the industry and factors other than regulation that affected the industry's level of
activity. The end result would be at least a qualitative idea of whether actual compliance costs were
significantly lower than expected, higher than expected, or similar - and why. Perhaps more importantly,
it provides information that could be used to modify the methodology for future ex ante estimates - or at
least a healthy degree of skepticism about how much can be known in advance.
The methodology section of the RCS includes an overview of how the rules to be analyzed were
selected, a description of the decision rule used to identify substantial differences between ex ante and
ex post cost estimates and a discussion of different potential sources of data for the ex post cost
estimates and ways to gather pertinent information about compliance activities as well as what would
have happened in the absence of regulation. The discussion in this section largely is focused on the last
issue regarding identifying sources of cost data and approaches to gathering other information necessary
to understand compliance activities. We discuss each of these sections in turn.
Selection of rules for study: The process used to identify the rules has the virtue of starting from a fairly
comprehensive list of regulations issued in the past 17 years. The rules for eliminating regulations from
the list (small regulations, etc) seem reasonable. The EPA economists used EPA's Rules and Policy
Information Development System (RAPIDS) to identify candidate rules for analysis. They focused on
economically significant rules (those with costs over $100 million) and recent regulations (issued since
1995) that were deemed easier to evaluate. Older rules were deemed less appropriate because they pre-
dated recent advances in benefit cost analysis. This process yielded a list of 111 entries from which
various rules were discarded, e.g., rules that were not yet implemented, remanded by the courts,
represented minor amendments to existing rules or had costs below the $100 million cost threshold, or
rules deemed too difficult to analyze. This winnowing of the set resulted in a list of 42 regulations that
was vetted internally at the EPA to make sure no major rules were missing. Notably, over half of the
rules on the list of 42 are air regulations. The RAPIDS database does not include regulations applied to
chemicals, including pesticides, so one rule in this category was selected through a separate process. It
would be helpful to know the process by which that rule was selected and how many pesticide (and
chemical) regulations might be included if these rules had been treated similarly to the waste, effluent
and air emissions rules included in the larger sample.
Ultimately the EPA selected 10 rules for the RCS study. The five rules included in Phase 1 were chosen
to illustrate the use of different data-gathering methodologies and a range of regulations (different media
and regulatory forms). For Phase 2 the regulations were selected using a randomized stratified approach
(three air regulations and two other). Randomization is important for, as Simpson (2011) points out,
most of the rules that have been included in prior studies comparing ex ante and ex post analyses have
not been randomly selected, but selected because there is data or, in some cases, to make a point about
particular forms of regulation (such as incentive-based regulations, or regulations of energy efficiency
standards for appliances). As a result, it may be difficult to generalize the findings of the prior studies.
The main goal in rule selection should be to identify a random sample of rules as well as to cover a
range of different EPA regulations - not just focus on air quality regulations where estimated health
benefits can be very high, providing little incentive to estimate costs precisely.
The small number of rules that the EPA has chosen for study in the RCS also limits the generalizability
of its findings. One way to supplement the set of case studies currently planned would be to do more
qualitative studies of a larger sample of randomly selected rules to identify the most important factors
9
-------
that affect differences between ex post and ex ante cost estimates across the different rules. Such studies
could provide important lessons regarding the relative importance of different cost drivers and how
those drivers differ across regulations.
Discerning Real Cost Differences: In the draft RCS study, EPA uses a ±25% cost difference threshold
to define discernible differences between ex ante and ex post costs. There is precedent in the literature
for using this threshold (Harrington et al. 2000; OMB 2005); however focusing too much on this cutoff
point risks overlooking cases where insight could be gained about key drivers. If strategic mis-reporting
caused an ex ante estimate to be too high while asymmetric information caused the estimate to be too
low, these two factors could cancel out each other and leave the misimpression that neither factor was
important. It is important to understand the reasons for cost differences even if these differences fall
below the 25% threshold.
Sources of Cost Data and Related Information: Ideally, to facilitate replicability and independence, the
RCS would use only publicly available data and not rely on sources of ex post cost data from sources
that had a vested interest in the findings of the ex ante analysis of compliance costs (the Regulatory
Impact Analyses or RIAs). In the absence of existing data, the EPA could use an industry wide survey to
gather relevant information from all regulated firms, but such surveys are costly to conduct, difficult to
implement and subject to strategic misreporting and a biased sample. Data would need to be collected
from a randomly selected sample of firms using survey techniques that ensure that the findings are
representative of the population of regulated firms.
Historically, data on environmental compliance costs were collected in the Pollution Abatement Costs
and Expenditures (PACE) survey formerly conducted annually by the Census Bureau (and in some years
funded by the EPA). Regulated establishments were required by law to answer this survey and this
limited the number of non-responding establishments and led to a fairly complete dataset. However,
these data have not been collected since 2005 and not on an annual basis for almost 20 years. Also, the
PACE survey historically did not break out regulatory costs by particular regulations (only by media and
category of cost), so discerning the costs of complying with particular regulations would likely require
additional information to be collected should the survey be reinitiated in the future. Nonetheless, this
data is better than no data and is preferred to regulation-specific data provided by a small sample of
affected parties that might be subject to selection bias; thus it would be very useful to resume annual
administration of the PACE Survey and data collection based on the survey. Conducted annually by the
Census Bureau from 1973-1994, these data have been the basis for many studies of regulatory costs over
the years. A redesigned version was launched in 2005, but was only collected for one year.
The PACE survey, if collected annually, would provide baseline measures of pollution abatement
expenditures from years before a regulation is adopted (or even proposed), and would permit
comparison with similarly measured abatement costs after the regulation is implemented. By contrast,
acquiring the same data with regulation-specific surveys is likely to be difficult or impossible, given the
time lags inherent in survey approval and the uncertainties in developing comparable measures of pre-
regulation costs. In addition, the 2005 PACE included questions about specific water and solid waste
pollution abatement techniques in use at the facility, along with usage of a detailed list of air pollution
control devices, distinguishing between newly installed devices and those previously in use. Thus the
PACE data could assist with both quantitative statistical analysis of changes in pollution abatement costs
following the implementation of a new regulation and a detailed qualitative analysis about the
techniques being used to comply with that regulation. The major limitation of the PACE survey is that it
10
-------
covers only manufacturing facilities, so alternative data sources would be needed for regulations
covering other sectors.
For four of the five rules included in Phase 1 of the study, there was insufficient information in the
public domain to estimate ex post costs. Thus for those rules (Cluster, MACT II, Arsenic and
Locomotive rules) the EPA had to rely on industry experts, many of whom provided information and
expertise to the development of ex ante costs used in the RIAs. An extensive process was used to
identify experts and develop a questionnaire relevant to each case study. Given the reliance on experts
for information, it would be useful for the EPA staff to consult the literature on expert elicitation to see
if there are any particular statistical issues or techniques that they should be aware of when using this
approach.
The RCS indicates that more detailed information gathering (including a site visit to a regulated facility)
will be incorporated in Phase 2. An important part of that process will be the selection of the rules under
which the site visit will be conducted and identification of the site to be visited. The timing of the site
visit could also be important to the extent that it provides lessons that might be useful in conducting the
ex post cost evaluations of other rules.
Methodology for Comparing Rules: An important omission from the methodology section is the
specification of a framework to make the studies of the different regulations more comparable to one
another and provide consistent documentation of the different factors that contribute to differences (or
similarities) between ex ante and ex post cost estimates. Understanding the underlying factors that
contribute to differences in cost is important to drawing conclusions about which factors could have
been anticipated and to the ultimate goal of this study, which is to understand what, if any, adjustments
to ex ante cost estimation approaches should be made to provide better predictions of actual costs of
compliance. Understanding these factors does require consultation with industry experts including
experts within the EPA or consultants about what firms and farms did to comply with regulation and
comparing those outcomes to what was expected in the RIAs. Publicly available sources of data will
generally not be sufficient for getting this type of information (unless a prior ex post study has been
conducted). Public data sources may be particularly deficient when there are multiple avenues for
compliance and considerable heterogeneity among the regulated firms.
One approach to enhancing comparability across the case studies that might be generalizable is
illustrated in the locomotive rule case study where each of the underlying components of the ex ante cost
calculation is identified and the difference between each ex ante assumption and post regulatory
realizations for each component is evaluated to understand its contribution to differences in ultimate cost
estimates. This approach helps to clarify the sources of differences and similarities and could help to
illustrate what factors might be common across regulations and what factors are idiosyncratic to
particular regulations.
While the stated focus of the RCS is to do a retrospective cost calculation in order to understand why ex
post estimates differ from ex ante, it would be useful to know more about the underlying assumptions
about technological change, input and output prices and other key inputs in the ex ante study. Instead of
comparing a single ex ante estimate with a single ex post estimate it might be better to generate a range
of estimates under alternative assumptions for these key inputs and see if the resulting range or
distribution of ex ante and ex post estimates overlap. By starting with the ex ante distribution of cost
estimates and the underlying assumptions, one could examine which assumptions might explain the gap
between ex ante and ex post estimates.
11
-------
3.2. Evaluating the Case Study Approach
3.2.1. Data Challenges
Charge Question 4: Each case study encountered a number of data-related challenges. Comprehensive,
detailed data on compliance costs simply were not available for any of the case studies. In some cases,
the EPA obtained detailed data on compliance costs for a small segment of the affected industry. In
others, the Agency obtained aggregate level data for a larger proportion of the industry.
a. Under what conditions are different sources of ex post data useful/not useful for assessing ex
post costs? In responding, please consider each of the following sources of data: aggregate
information from states, data from demonstration projects, detailed data for a small portion of a
heterogeneous industry, detailed data on what technologies have been adopted but unit costs for
a "typical" entity, expert opinion on costs without external validation, cost estimates from
contractors that worked on the original rule.
By design, the RCS used only publicly available data, most of which is highly aggregated. Survey data
would have strengthened all the case studies, but failing that, it is appropriate to use aggregate
information from states, data from demonstration projects, detailed data from small portions of industry,
unit costs for "typical" entities, expert opinions and cost estimates from contractors. All of these data
can be intelligently interpreted with proper qualifications to broaden the scope of the ex post analysis.
In particular, if available quantitative and qualitative data suggest the same conclusion, then it is
appropriate to note that. Aggregate information does not reveal what really caused the differences in ex
ante and ex post costs but supporting stories on what caused the differences underlying the data can help
improve understanding. The approach of using unit costs for a "typical" entity is useful. Like a "price
and quantity" approach, prices (unit cost) can be multiplied by quantity (number of units) to get an
estimate but the more heterogeneous the industry, the more sampling would be needed to get unit costs
for each technology. When using expert opinions, it is helpful to have corroborating information.
b. Is there anything more we could do with the data we have for these case studies that would yield
meaningful conclusions?
In general, improvements to the case studies depend upon getting more data but a systematic approach
across case studies would at least improve the readability of the case studies and it may improve the
analysis. The locomotive case study did a nice job in following a systematic approach as exemplified by
their presentation slide which outlined 11 distinct components of compliance costs, the available sources
of ex post data for each component, and an assessment of whether that component had turned out to be
larger or smaller than the ex ante estimate. One suggestion for the locomotive case study would be to
identify cases where the industry behaved differently than expected and use that information to look
back at the Agency's original assumptions about costs. The locomotive case study assumed a fairly high
fixed cost to develop and certify a new engine model, leading to the conclusion that few new models
would be developed. In fact, many new models were developed, which could have raised the question
of whether the fixed costs of model development may have been lower than expected.
12
-------
c. Do you have ideas on ways we can collect better ex post information for these case studies given
limited resources? Are there data sources we have not considered that we could use?
In the longer run, for future ex ante cost analyses, the EPA might consider ways of building in a routine
ex post data collection effort either within the agency or in collaboration with other agencies. One
source of ex post information could be regulatory staff (from both the EPA and state agencies) involved
in monitoring and enforcing compliance with specific regulations.The agency might also explore the
possibility of working with statistical agencies to insert questions related to costs of compliance into
regularly conducted surveys (e.g., the surveys of costs of agricultural enterprises taken by the National
Agricultural Statistics Service).
3.2.2. Analytic Challenges
Charge Question 5: When conducting these ex post cost analyses, EPA also struggled with a number of
analytic challenges including establishing a clear counter/actual and disentangling costs incurred in
response to a regulation from costs associated with other activities pursued simultaneously (e.g., system
upgrades, product redesigns).
a. Do you have suggestions on ways to better meet these challenges?
The EPA was appropriately cautious about data that might have included other kinds of capital
spending, such as upgrades to expand water treatment facilities or paper mills. A related analytic issue is
the construction of a baseline. For the Cluster Rule case study, the construction of industry's baseline
spending should be revisited. The public comments provided by the National Council for Air and Stream
Improvement (NCASI) and American Forest and Paper Association (AF&PA) provided strong evidence
of voluntary, pre-emptive spending before the promulgation of the Cluster Rule in 1998. It is worth
revisiting the choice of a "starting point" for the construction of a baseline. For the Cluster Rule, it
appears that voluntary capital expenditures were clearly made to remove dioxin from air and water
emissions in anticipation of the regulation that was ultimately promulgated in 1998.The SAB believes
this pre-regulatory, voluntary spending should be counted in the EPA's cost estimates, both ex ante and
ex post, if there is clear evidence that it was directly driven by impending regulations.
b. Do you have any thoughts or recommendations on how EPA can estimate or better apportion
costs across activities pursued by industry and differentiate between the regulatory driven and
non-regulatory driven changes?
More detailed survey data on regulation-specific abatement actions and costs of compliance from a
randomly drawn sample of regulated parties are needed across the board. One opportunity to separate
regulatory driven and non-regulatory changes is presented in the MBr case study in which it might be
possible to take advantage of heterogeneity in regulation at the state and local level. As noted in the
study, the rules on fumigant use vary considerably between states and even between townships within a
state. Survey data collected from farmers could take advantage of that variation to separate regulatory
effects from other changes in the industry.
13
-------
c. What should the EPA do in cases where there is a real paucity of data that limits the analytic
options? Are some methods of estimation more or less useful in these cases (e.g., expert opinion,
maintaining consistency with ex ante; a rough estimate based on assumptions and publicly
available data)?
The ex post analysis used available data in reasonable and creative ways, combining expert opinion,
engineering estimates, publicly available data, aggregate data from states, etc. Given that the
appropriate method of estimation is extremely context-dependent, it does not seem wise to generalize
about particular methods.
3.2.3. Weight of Evidence Conclusions
Charge Question 6: Given the various data and analytic challenges, is it appropriate to draw "weight
of evidence " conclusions on compliance costs based on the examination of key drivers of cost?
It may be reasonable to reach conclusions about the key drivers of costs and the likely direction of the
divergence between ex ante and ex post costs but, to do this, the SAB has recommended a conceptual
framework coupled with a more systematic documentation of the different factors that contribute to
differences between ex ante and ex post estimates. More details are provided in Section 3.1.3.
a. Are there cases where drawing conclusions is less or more defensible?
If the EPA had good quantitative data on ex ante and ex post costs from a large number of randomly
selected cases, then it would be defensible to draw a "weight of evidence" conclusion.
b. Some previous retrospective exercises have used hard metrics for evaluating whether costs are
over or underestimated (e.g., Harrington et al. used +/- 25percent). This same metric was
applied in the case studies presented in the Interim Report. Should the EPA continue to use this
metric? What are the drawbacks, if any, to applying a consistent metric across the case studies
given their differences in data quality?
Applying a hard metric like 25% as the cutoff for determining a "significant" difference between the ex
ante and the ex post estimate runs the risk of oversimplifying and generalizing across heterogeneous
case studies in which ex post cost estimates could differ for disparate reasons. The SAB is less in favor
of drawing a hard line than reporting actual numbers, e.g., 15% underestimate or 90% overestimate or
even a range.
14
-------
3.3. Moving Forward
3.3.1. Using Ex Post Cost Comparisons to Inform the Agency
Charge Question 7: It is difficult to make general statements about the accuracy of ex ante estimates of
the costs of a regulation because the promulgation of every regulation is a unique event. Different
considerations of timing, technology, industry structure, and a host of other factors go into the
estimation of ex ante costs and determine the accuracy of those ex ante estimates relative to ex post
experience. With this in mind, how can the EPA meaningfully make generalizations concerning ex ante
cost estimates based on ex post comparisons? How can ex post cost comparisons be used to inform how
the Agency estimates costs ex ante for future rules?
In order to make meaningful generalizations, one needs a sufficiently large sample size to identify
patterns in the data. Larger samples are a necessary but not sufficient condition to ensure generalizable
findings. The proposed studies would have to be designed in a systematic way, with guidance from an
underlying conceptual framework, to ensure strict comparability and the possibility of data synthesis.
The SAB suggests modifying the research strategy to do more, but shorter, more qualitative analyses of
regulations in Phase 2. These analyses would examine selected ex ante studies to identify three
components of prediction: (1) the expected compliance method(s); (2) the key factors expected to affect
the compliance cost for firms using each method; and (3) the expected distribution of the compliance
methods and costs across the regulated entities. This approach provides a "bottom-up" analysis,
breaking down total compliance costs along these three components. The goal would be to improve ex
ante analyses by identifying and categorizing any deviations observed ex post from the expected
components of compliance costs. Was there an unanticipated compliance method? Did surprisingly
rapid technological change or unexpected shifts in factor prices affect compliance costs for a given
compliance method? Did market forces change overall industry size or composition, affecting the mix
of compliance methods used?
Note that these deviations wouldn't necessarily cause prediction errors in total compliance costs,
because there could be multiple deviations, for example, a failure to foresee a cheaper compliance
method could be offset by an underestimate of the industry size. Getting "the right answer for the wrong
reasons" is not a good result. On the other hand, a well-conducted ex ante study could identify the
correct compliance method and understand the drivers of compliance costs perfectly, but underestimate
actual costs because of an unexpected rise in fuel prices.
At least some of the ex ante studies chosen for analysis should be selected in some predetermined
probabilistic manner and the number of studies should be large enough (perhaps 25 to 30 in all,
including the 10 already identified in Phase 1 and 2) for the sample to be representative.
Randomly selected studies will allow for generalization of results. The shorter nature of the analyses
should allow for a larger sample than currently planned. There should also be fewer worries about data
availability, since the qualitative approach is less demanding. The SAB also suggest that the
probabilistic-based selection process be a stratified random sample where regulation size, pollution type,
and even political salience of the regulations is considered in sampling, but that is an issue the EPA will
need to sort out given its particular needs. The studies selected for the qualitative analysis could also
include some or all of the Phase 1 and 2 studies already planned to see how informative a less intense
qualitative analysis is compared to a more intensive quantitative analysis and to design a method for
undertaking the qualitative analysis more effectively.
15
-------
After doing a substantial number of these qualitative studies, the deviations from ex ante expectations
could be examined to look for patterns in the types of deviations observed across different settings (by
pollution medium, by industry size, by the time lag until implementation, by the key factors affecting
costs, by the amount of resources devoted to the analysis). If certain types of deviations are very
common (e.g., always underestimating the rate of technological change in regulations with a long time
lag), future ex ante analyses could take that into account in their calculations. Perhaps more important,
this could encourage ex ante analyses to more explicitly deal with uncertainty along the lines where
errors are common in the ex ante studies. If fuel prices play a major role in compliance costs, the ex
ante analysis could highlight that assumption, e.g., "we assume that diesel fuel will cost $2 per gallon
and predict $150 million in total compliance costs, but if the price of diesel rises to $4, we predict $200
million in total compliance costs." Ex post cost analysis might suggest areas where greater sensitivity
analysis might be beneficial and a wider range of ex ante cost estimates could be provided to cover a
range of eventualities.
Note that the SAB is suggesting "qualitative or semi-quantitative" analyses in the sense that they do not
yield bottom-line total cost estimates, completeness across all cost categories, or flesh out details on
each cost component. Instead, they leave the analyst with the flexibility to discover and present well-
formed and documented 'stories' that identify the large drivers of differences between ex ante and ex
post results. Some regulations will require more depth than others. Most analyses would likely have a
quantitative aspect but with much less detail than one would find in full replication of the ex ante study
or what has been done thus far in Phase 1. For example, ex post unit cost estimates for different
compliance strategies would be obvious quantitative data points of interest.
The relatively detailed quantitative analyses conducted in Phase 1 and anticipated for Phase 2 seem
much less likely to yield generalizable results. First, the sample size is too small - only 10 regulations
studied, spread across a variety of pollution media and other differences (as noted in the charge question
itself). Second, the Phase 1 studies show that it is rare to find sufficient ex post data to construct
complete quantitative measures of compliance costs, which raises concerns about selectivity. If the
Agency only looks at regulations that happen "under the lamppost" of available data, its conclusions will
not reflect the full spectrum of regulations. Third, as Phase 1 demonstrates, the differences in data
quality across the regulations, and the differences in resources available for doing ex ante and ex post
studies, make it problematic to treat a set of "ex ante - ex post" quantitative calculations as
"homogenous" observations for the purposes of statistical (or regression) analysis.
Instead, the SAB suggests concentrating primarily on qualitative /semi-quantitative ex post analysis.
The existing Phase 1 and Phase 2 studies provide a set of 10 regulations. For each regulation, categorize
the ex ante expectations about compliance costs (compliance methods, key factors affecting compliance
costs for each method, and distribution of methods across the industry), and identify any ex post
deviations from those expectations. Develop a systematic approach to categorizing the expectations and
deviations, as well as the characteristics of the regulations. This will make it easier to analyze the results
of the qualitative studies for patterns, and to present those results in a way that facilitates comparisons
across regulations. The necessary qualitative information about deviations from expectations may be
available from industry experts within or outside EPA; some information (such as the actual compliance
methods being used) may also be available from federal or state inspectors responsible for enforcing the
regulation.
More complete quantitative analyses could be carried out for one or two of the Phase 2 regulations
where data are available. The broader (and more randomly selected) set of additional qualitative
16
-------
analyses, in addition to providing more generalizable results, will also provide a framework for
interpreting the detailed compliance cost comparisons in the quantitative analyses.
3.3.2. General Statements on the Bias of Ex Ante Estimates
Charge Question 8: Previous studies in the literature have focused on the relative proportions ofover-
and underestimates of costs or the average ratio of ex ante to ex post cost estimates. In section 3, EPA
suggested reasons for which these indicators might not reveal a bias in ex ante cost estimates, and have
suggested a regression-based procedure for making such a determination.
a. Is it possible to make general statements as to the accuracy of ex ante cost estimates? If so, what
might be the best way to evaluate their accuracy?
Without good data on ex ante and ex post costs from a large number of randomly selected cases, it is not
possible to make general statements as to the accuracy of ex ante cost estimates. The best way to
evaluate their accuracy is to identify systematic errors in procedures, methods and data used in previous
ex ante analyses and to uncover the key factors or driving forces that cause the differences between ex
ante and ex post cost estimates. However, it may be difficult to determine if there are procedural or
methodological errors that generate bias because there is only one analysis per regulation. Given the
uncertainty that accompanies both ex ante and ex post estimates and the variability in uncertainty over
time, the probability that ex ante and ex post estimates are identical should be small, and a finding that
ex ante generally over or under-estimates ex post costs is not a finding of procedural error. An
over/under estimate can occur every time, especially with few repeating analyses, without any
underlying procedural bias.
Given the above, quantitative assessments of ex ante cost estimate accuracy are not likely to be very
informative as supported by the analysis of Simpson (2011). However, qualitative assessments of
process and choice factors that appear to have been important in a random sample of cases may help
refine best practices. The goal should be a consistent set of best-practice rules that are followed, and
those rules should be updated frequently as lessons are learned from ex post analyses.
b. Is it appropriate to concentrate on the bias of ex ante cost estimates, or might other statistical
measures be more revealing?
Ex ante estimates of the effects of an environmental regulation, by definition, are based on information
available before the policy is undertaken. Because of uncertainties about technological progress and
other factors affecting the compliance costs, it is likely that ex ante estimates will differ from the ex post
costs. The larger the uncertainties, the smaller the probability that the ex ante estimates will equal the ex
post estimates. Under such circumstance, an appropriate ex ante analysis would provide an estimate of
the range of the expected compliance costs. Let /(C) denote the probability distribution function of the
ex ante estimate of the compliance cost of an environmental regulation, with C e [Cmin,Cmax]. The best
ex ante estimate of the compliance cost based on the information available equals the mean of the
distribution, denoted by c. The probability that the difference between the best ex ante estimate and the
ex post cost is greater than AC equals
C-AC
Prob(difference > AC) = J f(C)dC+ J f(C)dC
C+AC
17
-------
This suggests that the larger the uncertainties about technological progress and other factors affecting
the compliance costs, the larger the probability that the ex ante estimate will be different from the actual
ex post compliance cost by a large amount. A large and representative sample of ex post compliance
costs could be used to estimate a distribution of ex post costs and examine if there is a statistically
significant gap between ex ante and average ex post costs. However, SAB suggests that it is useful to
concentrate on the "bias" of ex ante cost estimates only if the objective is to uncover the systematic
errors in procedures and methods used to collect and analyze data used in previous ex ante analyses
and/or in reporting of results. In the absence of a large and representative sample of ex post cost
estimates to quantify the magnitude of the difference beween ex ante and ex post costs, the SAB
suggests undertaking a more qualitative/semi-quantitative comparison of these costs.
Such examinations are useful for future ex ante analyses as well as for policy reforms. If the ex ante
estimates of compliance costs are too small, the regulation might be too stringent; if the ex ante
estimates are too high, the regulation might be too lenient.
3.3.3. Rules for Evaluation
Charge Question 9: The rules addressed in the Interim Report were selected so as to cover a range of
media. Rules identified for Phase 2 of the project were randomly selected using stratified sampling. For
both Phase land Phase 2 rules, EPA has encountered (and continues to encounter) data challenges that
make it difficult to draw conclusions regarding realized compliance costs. If EPA continues with this
project, is there a way to credibly identify rules that would lead to informative ex post cost studies from
which conclusions can be drawn?
As noted above in the overall comments, discussion of ex ante versus ex post costs has emphasized
whether a "weight of evidence" determination can be made on whether ex ante costs are higher or lower
than ex post costs. Too narrow a focus on this goal naturally leads to a focus on information-rich rules.
Too much focus on information-rich rules may lead to bias that undermines the credibility of the effort.
Of particular concern is the possibility that more complete cost information may be confounded with
analyst or stakeholder concern that ex ante estimates are more likely to be off the mark for those rules.
Perhaps less cost information is associated with those rules for which ex ante estimates are more trusted.
This would imply data rich rules are not representative of the population of relevant rules.
The SAB recommends an alternative approach to randomly select rules. Determine what classes/sets of
compliance costs associated with the rule are reasonably measurable, and then conduct analysis based on
available information. This may lead to partial analyses that do not provide insight into a "weight of
evidence" determination on the relationship between ex ante and ex post costs. However, these partial
analyses may be quite useful in helping "identify systematic differences between ex post and ex ante
cost compliance estimation, and ultimately allow for improvements in the way in which ex ante analyses
are done."4 Qualitative information should be used to identify the important drivers of potential
differences between ex ante and ex post cost estimates. For some rules, cost information is so scarce that
no insights are possible. However there should be at least partial information for most rules that could
provide insights regarding this second goal. Using random rule selection and examining those
compliance cost drivers turns the unit of observation into randomly selected drivers. The focus on the
drivers or factors that influence the accuracy of ex ante cost estimates should also help to improve the
4/RCSp. 7
18
-------
usefulness of this analysis for future ex ante compliance cost assessments in regulatory impact analyses
conducted by the EPA.
a. What should our priorities be with regard to how we select the sample of rules for evaluation?
Should we compromise the random selection of rules in favor of selecting rules with more
readily available data? If so, on what types of rules should we focus? What types of rules could
we defensibly leave out? What kind of selection biases would we introduce under different
sampling methods? For instance, if we exclude rules where the industry is particularly
heterogeneous or data are difficult to identify, can we still draw conclusions that would be
generally applicable?
A significant portion of the rules selected for analysis should be drawn randomly from the candidate
rules. In addition, EPA should try to identify a few of the most important rules. Note the
recommendation is not to select rules according to how much information is available, but rather to
make sure some consideration is given to the most high profile rules. Since the initial screening process
eliminated rules with very low costs, candidate rules may have already been selected on account of their
significance. For most of the rules selected for study, random selection should not be compromised.
At the rule selection stage, there is no scientifically plausible defense for a priori leaving out rules
simply because they do not have complete information.
It is difficult to predict the direction of bias from studying only information-rich rules or rules that affect
homogeneous firms. Suppose that information is generated in response to a belief that ex ante and ex
post costs are more likely to diverge. In that situation, concentrating all analytical effort on information-
rich rules would likely provide a biased view on the precision of ex ante estimates.
b. How do we balance pragmatism vs. the purity of our sampling method, given our experiences
(e.g., the challenges/limitations we have faced due to lack of participation by industry, an
inability to identify industry compliance experts, potential contractor bias, and data limitations)?
Should our choice of methodology inform the way we select the sample of rules or vice versa?
With respect to the first question, the balance between pragmatism and purity of the sampling method is
fundamentally dependent on the analytical methods used. If, as discussed in Section 3 of the RCS, a
regression-based procedure is adopted to analyze the differences between the ex ante and ex post cost
estimates, then the statistical properties of the data are crucial to the validity of the inferences that are
drawn. In this case, the "purity" of the sampling method is very important.
However, the SAB agrees with the EPA that there is limited value in using econometric tools in the
absence of sufficient ex post cost data. Incomplete data, variability in the quality of the available data,
and incomplete specification of possible drivers and their measurement preclude specification of an
appropriate statistical methodology and sampling strategy. Even if all rules in Table 1.1 were included,
the sample is still too small (n=42) to permit robust inferences about multiple drivers, even disregarding
the obvious data issues (sample selection bias, measurement errors, missing variables, etc.). As
described elsewhere, the SAB favors a pragmatic qualitative review of a larger number of rules followed
by detailed quantitative analysis of a smaller number of rules, focused on analyzing drivers for
systematic differences between ex ante and ex post estimates.
19
-------
For the second question, the methodology should drive the sampling method. In following the
recommended "qualitative followed by quantitative" approach, the Agency might consider stratifying
rules. Proper stratification along the dimensions of pollutants (air, water, etc.), nature of regulated
entities (public vs. private), timing of regulation, and heterogeneity of responses, among others, enables
identification and understanding of the drivers and their impacts on the ex ante and ex post differences.
The analysis thus focuses on the drivers rather than cost differences themselves. In following this
approach, there is no need to balance pragmatism and purity because there is no tradeoff.
3.3.4. Building a Database
Charge Question 10: What additional suggestions does the SAB have on how best to build a database of
ex ante versus ex post cost comparisons of regulation given the difficulties the Agency has faced thus
far?
Although the RCS is focused specifically on EPA regulations, many of the issues of concern are not
unique to the United States, or even to environmental policy. As a result, it may be possible to expand
the database of ex post studies by surveying environmental regulatory agencies in Europe and elsewhere
that may have carried out similar exercises. In the long run, that would help the EPA establish
guidelines for ex post analysis that would draw on international best practices. Similarly, it may be
possible to expand the database through inter-agency collaboration as other federal or state agencies may
have carried out ex post analyses that could be included in the database.
In the long run, building an extensive and authoritative literature comparing ex ante and ex post analysis
will require deep and continuing involvement with the outside research community. To facilitate the
development of that literature, it would be very valuable to establish a long-term program on ex post
analysis that could provide research grants and funding for conferences and workshops.
Building a good base of knowledge for ex ante vs. ex post assessment of regulatory impacts is a major
challenge that will take time and resources. The EPA and the outside research community need to
establish a base of evidence that will be sufficiently large to allow statistical inference of differences
between ex ante and ex cost estimates under alternative circumstances. The time required to improve
this understanding can be shortened if, as noted above, the agency takes advantage of economies of scale
and increases the evidence considered to include cases related to regulations outside the United States
and in individual states. But increasing the evidence base may lead to a tradeoff between quantity and
quality and incorporation of extra evidence will require additional care. The understanding of the causes
of ex ante vs. ex post differences will be enhanced if it will become a major subject of academic
research, and that can be affected by financial and other incentives.
Differences between ex ante and ex post estimates of regulatory costs are not limited to environmental
regulation and occur in other fields. The SAB recommends that the EPA make a concerted and ongoing
effort to learn from experiences in other regulatory fields. The agency may be able gain additional data
for ex ante studies if it works with statistical agencies to insert questions related to costs of compliance
for regulatory requirements in various surveys and studies taken by these agencies (for example the
surveys of costs of agricultural enterprises taken by the National Agricultural Statistical Survey (NASS).
Care should be taken in the design of questions to account for strategic behavior by respondents.
In sum: The SAB supports the EPA's effort to undertake a RCS and compare ex post and ex ante
estimates of the costs of compliance with regulation with the primary goal of using the insights obtained
20
-------
to improve the methods and accuracy of ex ante cost estimation. The SAB offers the following
recommendations for the RCS:
• Develop a conceptual framework that includes the taxonomy of key factors that result in a
divergence of ex ante and ex post cost estimates and provides a consistent structure throughout
the study.
• Use a large randomly sampled set of regulations for the ex post cost estimation and conduct a
qualitative/semi-quantitative analysis of the key factors that can explain the difference in ex ante
and ex post costs
• Consider ways to collect ex post cost information routinely, through surveys, focus groups,
expert assessments and consultation with the Research Community while avoiding sample
selection bias.
• Focus on the drivers or factors that influence the accuracy of ex ante cost estimates rather than
on ascertaining whether the weight of evidence supports a hard line on the magnitude of the
difference between ex ante and ex post costs of compliance.
21
-------
REFERENCES
Bailey, P.D., Haq, G. and A. Gouldson. 2002. Mind the gap! Comparing ex ante and ex post assessments
of the costs of complying with environmental regulation. European Environment., 12 (5): 245-
256.
Boardman, Anthony E., Wendy L. Mallery and Aidan R. Vining. 1994. Learning from ex ante/ex post
cost-benefit comparisons: the coquihalla highway example. Socio-Economic Planning Sciences.,
28(2):69-84.
Dayton, Julia. 1998. World Bank HIV/ADDS Interventions: Ex Ante and Ex Post Evaluations. World
Bank Publication 389.
Grosse, Scott D., Norman J. Waitzman, Patrick S. Romano, and Joseph Mulinare. 2005. Reevaluating
the Benefits of Folic Acid Fortification in the United States: Economic Analysis, Regulation, and
Public Health. American Journal of Public Health. 95(11): 1917-1922.
Harrington, W., Morgenstern, R.D. and P. Nelson. 2000. On the accuracy of regulatory cost estimates.
Journal of Policy Analysis and Management, 19 (2): 297-322.
LaFrance, Jeffrey T. and Harry de Goiter. 1985. Regulation in a Dynamic Market: The U.S. Dairy
Industry. American Journal of Agricultural Economics, 67(4):821-832.
Lindner, R. K. and F. G. Jarrett. 1978. Supply Shifts and the Size of Research Benefits. American
Journal of Agricultural Economics., 60(l):48-58.
NRC (National Research Council). 2012. Review of the EPA 's Economic Analysis of Final Water
Quality Standards for Lakes and Flowing Waters in Florida. Washington, DC: The National
Academies Press.
OMB (Office of Management and Budget). 2005. Validating Regulatory Analysis: 2005 Report to Congress
on the Costs and Benefits of Federal Regulations and Unfunded Mandates on State, Local, and
Tribal Entities. Washington DC: Government Printing Office.
Popp, D. 2003. Pollution Control Innovations and the Clean Air Act of 1990. Journal of Policy Analysis
and Management, 22(4): 641-660.
Rideout, Douglas B. and Philip N. Omi. 1995. Estimating the Cost of Fuels Treatment. Forest Science,
41(4): 664-674(11).
Sappington, David E. M. and Joseph E. Stiglitz. 1987. Privatization, information and incentives.
Journal of Policy Analysis and Management, 6(4): 567-585.
Simpson, R. D. 2011. Do Regulators Overestimate the Cost of Regulation? NCEE Working Paper 11-
07, U.S. Environmental Protection Agency.
22
-------
APPENDIX A: CHARGE QUESTIONS
MEMORANDUM
To: Holly Stallworth, Designated Federal Officer
Science Advisory Board Staff Office
From: Nathalie B. Simon, Associate Director
National Center for Environmental Economics
Date: March 30, 2012
Subject: Charge Questions for SAB-EEAC Advisory on the "Retrospective Study of the Costs of EPA
Regulations: An Interim Report of Five Case Studies," April 19-20, 2012
The purpose of this memorandum is to transmit charge questions for consideration by the Science
Advisory Board's Environmental Economics Advisory Committee (SAB-EEAC) during the upcoming
Advisory meeting scheduled for April 19 and 20* on the "Retrospective Study of the Costs of EPA
Regulations: An Interim Report of Five Case Studies."
Benefit-cost analyses are often conducted to inform decision-making at the Environmental Protection
Agency (EPA). While the EPA strives to use the best available science and engineering when
conducting its economic analyses, they are by their very nature uncertain, relying on forecasted
information. While new science and the need to quantify more, previously unquantified benefits has
driven benefits analysis, comparatively less work has been done retroactively examining how well EPA
estimates the costs (or benefits) of regulation. The ex post cost studies that are available in the literature
are often based on limited data and overlap in coverage - many of the same regulations appear in
multiple publications. And, while the literature posits a number of hypotheses for why one might expect
ex ante and ex post cost estimates to differ, ex post analyses are too few in number to be able to credibly
accept or reject these hypotheses.
The National Center for Environmental Economics (NCEE) has launched an effort to evaluate the
feasibility of augmenting the existing literature with additional ex post evaluations of costs. Using a
case study approach, we attempt to determine if sufficient information can be gathered on individual
rules to make a "weight of evidence" determination about whether ex ante costs are higher or lower than
ex post costs. If the case study approach is successful, there is much that can be learned from this
effort. A careful assessment of ex post costs could help identify systematic differences between ex post
and ex ante compliance cost estimation and, ultimately, allow for improvements in the way in which ex
ante analyses are done. For instance, if unanticipated changes in market conditions, energy prices, or
available technologies regularly result in an over or underestimate of costs, the EPA can invest in
improving methods that better capture these effects on costs ex ante.
"Retrospective Study of the Costs of EPA Regulations: An Interim Report of Five Case Studies"
(March 2012) summarizes the initial findings from a small set of pilot case studies that attempt to
evaluate the costs of EPA regulatory and other policy actions ex post. The initial set of case studies rely
on a variety of methods for collecting ex post information - some mainly rely on publically available
A-l
-------
data and literature and are conducted internally, while others rely on industry experts or third-party data
collected by a contractor.
The five case studies presented in the interim report are:
• Integrated NESHAP and Effluent Guidelines for Pulp and Paper (1998)
• NESHAP: Chemical Recovery Combustion Sources at Kraft, Soda, Sulfite and Stand-
Alone Semichemical Pulp Mills (2001)
• Methyl Bromide Critical Use Nomination for Preplant Soil Use for Strawberries Grown
for Fruit in Open Fields on Plastic Tarps (2004-2008)
• National Primary Drinking Water Regulation for Arsenic (2001)
• Locomotive Emission Standards (1998)
To be clear, the case studies in this report do not aim to estimate ex post costs of these EPA actions.
Rather, they examine key drivers of compliance costs to see if informed judgments (weighing the
evidence) can be made about whether ex post costs are higher or lower than the estimates of ex ante
costs.
While a number of these case studies are suggestive of overestimation of costs ex ante, EPA could not
cost-effectively gather sufficient information in other case studies to form judgments. As a group they
expand our understanding of how and why ex ante costs may differ, but they are not conclusive. First,
they only represent a small subset of regulatory and other policy actions taken by the EPA. Second,
conducting ex post analysis has proven more challenging than anticipated. With regard to data, these
challenges have included the inability to identify qualified industry experts that did not also work on the
rule and limited access to industry data. Analytic challenges have included how to evaluate a highly
heterogeneous industry with a limited set of information, how to form a reasonable counterfactual, and
disentangling the costs of compliance from other factors, to name a few.
Before proceeding with additional work in this area, NCEE is seeking advice from the SAB-EEAC on
the case study approach applied in the paper and specifically on appropriate methodologies to be applied
in these and future case studies. As such, the findings presented in the report should be considered
preliminary and are subject to change.
Please contact me if you have any questions about the attached charge.
A-2
-------
Charge Questions for SAB-EEAC Advisory on the "Retrospective Study of the Costs of EPA
Regulations: An Interim Report of Five Case Studies," April 19-20, 2012
Part I: Evaluating the Retrospective Cost Study Methodology
(1) Section 2 of the report summarizes existing retrospective cost studies. Have we adequately
summarized the existing literature or have we mischaracterized it in some way? Are you aware of
any studies we have missed? Have we captured key take away messages offered by this
literature? How can this discussion be improved?
(2) Section 3 of the report briefly describes potential reasons ex ante and ex post estimates might
differ. Have we accurately described the various hypotheses? If not, how can this discussion be
improved? Are there other hypotheses that should be included or considered?
(3) One goal of this study is to demonstrate the use of different methodologies for obtaining ex post
information on key drivers of compliance cost. While the level of coverage and detail does not
match what is typically used in an ex ante analysis in support of a rule, the purpose is to gather
enough evidence on key drivers to establish a weight of evidence determination on the direction
of ex post costs compared to ex ante costs. These approaches are briefly described in section 4 of
the report. Are some approaches more defensible than others? If so, which ones and why? Which
of the methodologies are more likely to yield reliable results and why? Or are the merits of each
method dependent upon characteristics of the case study to which they are being applied? If some
methodologies are more applicable under specific contexts or rule settings, please identify and
describe. Are there other methodologies that should be considered for ex post cost analyses?
Part II: Evaluating the Case Study Approach
Conducting ex post cost analyses for the selected rules proved to be more difficult than expected. We
encountered a number of challenges, both data-related and analytical in the process. Please answer
questions 4 through 6 for each case study.
(4) Each case study encountered a number of data-related challenges. Comprehensive, detailed data
on compliance costs simply was not available for any of the case studies. In some cases, we
obtained detailed data on compliance costs for a small segment of the affected industry. In others,
we obtained aggregate level data for a larger proportion of the industry.
a. Under what conditions are different sources of ex post data useful/not useful for assessing
ex post costs? In responding, please consider each of the following sources of data:
• aggregate information from states,
• data from demonstration projects,
• detailed data for a small portion of a heterogeneous industry,
• detailed data on what technologies have been adopted but unit costs for a
"typical" entity,
• expert opinion on costs without external validation,
• cost estimates from contractors that worked on the original rule
b. Is there anything more we could do with the data we have for these case studies that
would yield meaningful conclusions?
A-3
-------
c. Do you have ideas on ways we can collect better ex post information for these case
studies given limited resources? Are there data sources we have not considered that we
could use?
(5) When conducting these ex post cost analyses, we also struggled with a number of analytic
challenges including establishing a clear counterfactual and disentangling costs incurred in
response to a regulation from costs associated with other activities pursued simultaneously (e.g.,
system upgrades, product redesigns).
a. Do you have suggestions on ways to better meet these challenges?
b. Do you have any thoughts or recommendations on how EPA can estimate or better
apportion costs across activities pursued by industry and differentiate between the
regulatory driven and non-regulatory driven changes?
c. What should we do in cases where there is a real paucity of data that limits our analytic
options? Are some methods of estimation more or less useful in these cases (e.g. expert
opinion, maintain consistency with ex ante; a rough estimate based on assumptions and
publically available data)?
(6) Given the various data and analytic challenges, is it appropriate to draw "weight of evidence"
conclusions on compliance costs based on the examination of key drivers of cost?
a. Are there cases where drawing conclusions is less or more defensible?
b. Some previous retrospective exercises have used hard metrics for evaluating whether
costs are over or underestimated (e.g., Harrington et al. used +/- 25 percent). This same
metric was applied in the case studies presented in the Interim Report. Should we
continue to use this metric? What are the drawbacks, if any, to applying a consistent
metric across the case studies given their differences in data quality?
Part III: Moving Forward
(7) It is difficult to make general statements about the accuracy of ex ante estimates of the costs of a
regulation because the promulgation of every regulation is a unique event. Different
considerations of timing, technology, industry structure, and a host of other factors go into the
estimation of ex ante costs and determine the accuracy of those ex ante estimates relative to ex
post experience. With this in mind, how can we meaningfully make generalizations concerning
ex ante cost estimates based on ex post comparisons? How can ex post cost comparisons be used
to inform how the Agency estimates costs ex ante for future rules?
(8) Previous studies in the literature have focused on the relative proportions of over- and
underestimates of costs or the average ratio of ex ante to ex post cost estimates. In section 3, we
have suggested reasons for which these indicators might not reveal a bias in ex ante cost
estimates, and have suggested a regression-based procedure for making such a determination.
a. Is it possible to make general statements as to the accuracy of ex ante cost estimates? If
so, what might be the best way to evaluate their accuracy?
b. Is it appropriate to concentrate on the bias of ex ante cost estimates, or might other
statistical measures be more revealing?
(9) The rules addressed in the Interim Report were selected so as to cover a range of media. Rules
identified for Phase 2 of the project were randomly selected using stratified sampling. For both
Phase 1 and Phase 2 rules, we have encountered (and continue to encounter) data challenges that
make it difficult to draw conclusions regarding realized compliance costs. If we continue with
A-4
-------
this project, is there a way to credibly identify rules that would lead to informative ex post cost
studies from which we can draw conclusions? Specifically,
a. What should our priorities be with regard to how we select the sample of rules for
evaluation? Should we compromise the random selection of rules in favor of selecting
rules with more readily available data? If so, on what types of rules should we focus?
What types of rules could we defensibly leave out? What kind of selection biases would
we introduce under different sampling methods? For instance, if we exclude rules where
the industry is particularly heterogeneous or data are difficult to identify, can we still
draw conclusions that would be generally applicable?
b. How do we balance pragmatism vs. the purity of our sampling method, given our
experiences (e.g., the challenges/limitations we have faced due to lack of participation by
industry, an inability to identify industry compliance experts, potential contractor bias,
and data limitations)? Should our choice of methodology inform the way we select the
sample of rules or vice versa?
(10) What additional suggestions do you have on how best to build a database of ex ante versus ex
post cost comparisons of regulation given the difficulties we have faced thus far?
A-5
------- |