oEPA Evaluation of Three Environmental Results Programs August 2009 Fact Sheet Introduction http://www.eDa.gov/evaluate For more information on completed evaluations at EPA or the Evaluation Support Division, visit the above link. • There are an estimated 35,000 to 80,000 facilities across the country that are in the business of repairing and refinishing vehicles, particularly cars. Auto body shops present a wide array of environmental concerns, from use and emissions of hazardous materials such as methylene chloride, to discharges of polluted water into water systems, and worker exposure to toxic solvents and particulate matter. • The National Emission Standards for Hazardous Air Pollutants: Paint Stripping and Miscellaneous Surface Coating Operations at Area Sources, is a regulation that existing shops must comply with by January, 2011 and that new shops must comply with by January, 2008 or upon startup of operations. The rule requires that, for example, all spray painting must be done in a spray booth, painters must use spray guns and techniques that reduce overspray, all painters must receive training, and paint spray gun cleaning cannot release any mist of cleaning solvent to the air. • States are currently considering how to implement the new regulation; ERP is one potential policy tool for implementation. ERP is an innovative approach to improving facilities' management practices within small business sectors. ERP is an integrated system that incorporates plain language compliance assistance that promotes pollution prevention, facility self-assessment and self-certification, agency inspections, statistically-based performance measurement, and where necessary, comprehensive facility investigations and targeted enforcement actions. • The purpose of this evaluation is to inform states and EPA regions currently considering developing programs to encourage auto body shops to adopt best practices and improve compliance with environmental regulations. Evaluation Questions • To what extent have the ERPs in DE, RI, and ME led to actual and/or expected adoption of selected best practices that reduce the environmental footprint of auto body shops? • What environmental and health outcomes are estimated to result from auto body shops implementing these best practices? • What are the cost implications of each program for regulators and auto body shops initially and over time? • What is the cost-effectiveness of each program? • Overall, what are the advantages and disadvantages of each of the three ERPs in terms of reaching auto body shops, generating environmental and worker health outcomes, and achieving cost-effective results? • What factors influenced the outcomes of each program (e.g., existing or impending regulations, regulatory/assistance offices involved in conducting the program, and extent of coordination with industry representatives)? ------- Evaluation Questions Con't. • What do our findings suggest regarding the circumstances under which ERPs are likely to produce cost-effective results? • What is the current status of each program? • What are the primary implementation challenges states faced in developing and implementing their programs? • What factors influenced the states' decisions to continue, not continue, or modify their programs after initial pilots? • How does the state and/or EPA Region involved in implementing each program view the program's results, and why? Evaluation Methods • To assess the behavioral and environmental outcomes of the ERPs, the evaluation primarily relied on existing data reported by each program, which captured what percentage of facilities use certain best practices during independent, random inspections before and after ERP implementation. The analysis compares the percentage of shops using certain best practices before and after the ERP program and translates those changes in behaviors to environmental outcomes, where possible. Where this translation is not possible, we qualitatively discuss the benefits we would expect facilities would observe as a result of adopting these behaviors. • We cannot prove the extent to which ERP led to observed outcomes, in comparison to other factors (e.g., regulations or permit requirements) happening concurrently with ERP. • ERP states did not collect quantitative data on long-term outcomes (e.g., emissions reductions), and therefore we were limited to estimating this information where we could. • To assess the remaining evaluation questions, the evaluation relies on interviews with state program staff, EPA staff involved in supporting ERP, representatives of the States' ERP Consortium, and selected auto body shops. For example, these interviews provided insights on program status, costs, factors that influenced outcomes, and implementation experiences. Key Findings • Our findings suggest that ERP is associated with improved business practices in the auto body sector and is regarded as successful by both state and industry representatives. Quantifying environmental outcomes associated with ERP is difficult, and those outcomes that we were able to quantify were relatively small. In addition, sustaining the program has proven to be difficult given resource constraints and overall regulatory priorities. ------- Key Findings Con't. • Each state selected between 19 and 24 indicators of environmental performance. States observed improved performance between the samples of facilities measured for the vast majority of indicators (observed performance improved for 54 out of 65 indicators (83 percent) between baseline and post- certification). Of these 54 indicators, 29 (54 percent of the indicators where observed performance improved, and 45 percent of all indicators) were found to have statistically significant changes in performance. In addition, we found no statistically significant declines in performance. • The greatest percentage of the total number of indicators with statistically significant improvements were observed in the air emissions and worker health and safety categories. In both of these categories, half of the total set of indicators measured (not solely those improving, but the total number of indicators) showed statistically significant improvements. • With regard to air emissions, we estimate that improvements in usage of low- VOC/waterbased solvents may have reduced VOC emissions by as much as 1.7 tons per year for all auto body shops in each of two states: Delaware and Maine (actual amounts could be less). In addition, we estimate that the material usage associated with auto body shop spray gun cleaning operations declined by between 0.6 and 2.0 tons per year for all shops in Delaware. Finally, we estimate that improvements in hazardous waste management in Delaware may have increased the amount of hazardous waste being properly identified by as much as 22,440 pounds per month (actual amounts could be less). All three of these estimates rely on numerous assumptions, and have considerable uncertainties, which are described in the full evaluation report. We were not able to quantify reductions in water discharges or improvements to worker health and safety. • Note that while the overall trend in improving performance measured by states is consistent with the hypothesis that ERP leads to adoption of selected best practices, we cannot be sure that the observed changes can be fully attributed to the presence of ERP. • The three states included in the evaluation spent a range of resources developing and implementing their ERPs. We estimate government costs (including state resources and an EPA State Innovation Grant) of $800 to $2,000 per auto body shop in the population for states conducting a single cycle of ERP. These costs are not inclusive of all of the state resources spent on staff throughout the program. States also dedicated staff time that is not fully captured in these cost estimates. For Rhode Island, costs range from $400 to $700 per shop for two cycles of ERP (in other words, the costs per shop per cycle would be half as much). • Compared to the alternative policy tools that these three states considered, this evaluation finds that ERP was a high-cost/high-value approach. • A key factor related to cost effectiveness is the number of facilities targeted by ERP. Specifically, ERP can be more cost effective for larger populations of facilities, so long as the population is relatively homogenous, with a common set of regulatory requirements and best practices. ------- Key Findings Con't. • Another factor related to the likelihood of a state achieving cost-effective results through ERP is the degree to which state staff can build partnerships within (e.g., between different environmental media offices) and outside agency walls (e.g. Rhode Island's partnership with the University of Rhode Island). State staff within all of the ERPs described in this evaluation forged partnerships within and outside their agencies to implement the ERP. Successful design and implementation of ERPs that deal with multiple environmental media require the cooperation among and buy-in from the various offices that are affected. Outside partners may provide support in the form of technical expertise and/or funding. • Baseline performance and the likely extent of improvement (i.e., to what extent are facilities likely to be able or willing to change their behavior) both have a direct effect on the extent to which statistically significant changes may be observed. • Key factors leading to success of an ERP include sufficient funding, the regulatory context in which the program was implemented, effective coordination and communication among involved offices, upper management buy-in, and continuing program support. Recommendations • Combine forces. States could work together to realize the economies of scale possible with ERP. • Decide on a set of common indicators. It would be helpful to be able to compare and aggregate ERP data for the same sector across programs. • Collect quantitative data on facility practices, not just information on the proportions of shops utilizing specific practices, for a small number of indicators. This could help quantify environmental outcomes. • Develop a tool to help states estimate environmental outcomes. • Un-package ERP. States could consider different ways to implement different components of ERP. • Consider implementing ERP primarily where larger populations of facilities are present. This approach has the potential to reduce per facility expenditures and increase the cost-effectiveness of the program. • Develop a clearer agreement between EPA and states as to whether or not ERP can be used to address core programs. Contact(s) • John Heffelfinger, EPA Office of Policy, Economics and Innovation, heffelfinQer.iohn@eDa.QQv • Scott Bowles, EPA Office of Policy, Economics and Innovation, bowles.scott@eDa.Qov Report Link: http://www.eDa.Qov/evaluate Date Completed: August, 2009 ------- |