Appendix:
ERP States Produce Results 2007 Report
December 2007
Document Number: EPA 100-R-07-008
The ERP States Produce Results 2007 Report (EPA 100-R-07-009), hereafter the "2007 Report,"
represents the first attempt to gather and assess data from multiple states on their Environmental
Results Programs (ERPs). Although each ERP follows essentially the same measurement
approach, based upon random inspections of facilities before and after self-certification, states
have found a variety of different and useful ways to report their own data. A challenge for the
U.S. Environmental Protection Agency (EPA) in developing this document was to tell a national
story for ERP while taking into account each state's unique approach to presenting its data. EPA
had to make many choices about how to present and analyze the data from states—choices that
would be cumbersome to explain in the 2007 Report itself. As much as possible, EPA chose to
minimize its own analysis of the data in order to allow the states' data to speak for itself. EPA
focused its analyses on areas deemed helpful to telling the story of ERP and to presenting
comparable data across states.
The purpose of this appendix is to provide more information about those background analyses
and to present detailed information on the sources of data—which range from peer-reviewed,
published findings on the Rhode Island auto body ERP to e-mail and telephone communications
with state personnel. EPA hopes that this transparent approach will make the results presented in
the 2007 Report more easily understood, while also highlighting issues worthy of further
attention and discussion among states—especially as the States ERP Consortium endeavors to
create minimum ERP reporting standards.
This appendix is organized topically, in the order in which analytical issues appear in the 2007
Report. Brief summaries of the findings associated with each topic are followed by discussions
of analytic methodologies and a list of information sources. The list of information sources has
detailed source-specific notes, as needed. Finally, the appendix closes with a description of
EPA's approach to data collection, verification, and analysis. Provided with that description is a
list of state personnel to whom EPA offered the opportunity to review and comment on the data
and analyses presented in the 2007 Report, its executive summary, and this appendix.
1. Observed Average Indicator Improvement
ERPs produce a wealth of data. States utilize Environmental Business Practice Indicators
(EBPIs) in order to focus their own attention on the most important performance issues. For an
individual ERP, the results for each EBPI can be presented in a readily digestible, one-page
format. However, EPA felt that presenting the results for all 135 EBPIs from the eight completed
ERPs would be overwhelming. Consequently, EPA looked for a way to summarize the
performance changes each state observed in its EBPIs. To that end, EPA developed a simple
-------
summary measure called the "observed average indicator improvement," which is intended to
fairly represent the progress shown in individual ERPs. (It also might have been quite instructive
to develop group compliance scores for all ERPs, similar to the Massachusetts group compliance
scores, but the detailed analysis required to develop such scores after-the-fact was beyond the
scope of the 2007 Report.)
Table A-l below reiterates data shown in Table 2 of the 2007 Report, as well as some additional
data provided to help users better understand the context in which EPA chose to present the
observed average indicator improvement metric.
Table A-l: Observed Average Indicator Improvement in First ERP Self-Certification Cycle
State
DE
FL
MA
MA
MA
MD
ME
Rl
Average
TOTAL
Sector
Auto Body
Auto Repair
Dry Cleaners
Photo Processors
Printers
Auto Body/Repair
Auto Body
Auto Body
All Sectors
All Sectors
#of
Indicators
19
17
15
8
25
5
22
24
16.875
135
# Improving
(# Significant)
17(13)
13(7)
5(0)
3(1)
17(1)
4(1)
18(3)
19(7)
12 (4.125)
96 (33)
# Worsening
(# Significant)
1 (0)
3(0)
5(0)
2(0)
6(0)
1 (0)
4(0)
3(0)
3.125(0)
25(0)
# No Change,
100% Meeting
Expectation
1
1
5
3
2
0
0
2
1.75
14
Avg. Pet.
Pt.
Change
30
7
5
12
13
12
10
21
N/A
N/A
Median
Pet. Pt.
Change
27.5
5.1
-0.8
4.0
9.0
16.0
6.8
16.0
N/A
N/A
Min. Pet.
Pt.
Change
-8.0
-12.6
-11.1
-9.2
-25.0
-0.1
-4.0
-7.0
N/A
N/A
Max. Pet.
Pt.
Change
77.0
26.7
55.0
34.0
100.0
22.0
55.9
54.0
N/A
N/A
For each state, the average percentage point improvement represents a simple average of the
percentage point changes in indicator values after the first round of self-certification. For
example, if 40% of facilities were in compliance with an indicator during the baseline round of
random inspections and 60% in the post-certification round of inspections, the change would be
20 percentage points. A state's average is the mean of all these percentage point changes for all
indicators, positive or negative.
The average excludes indicators that showed 100% performance levels at baseline and post-
certification random samples, because improvement is not possible in that circumstance. All
other indicators were included, even in instances in which the number of facilities was quite
small, because by their nature some indicators measure characteristics common to only a small
proportion of facilities.
Some of the important questions EPA considered in developing this summary performance table
are discussed on the next few pages.
Why Present the Average in the 2007 Report? EPA chose to present the average (i.e., mean)
percentage point improvement in the 2007 Report because it seemed to best and most easily
represent the progress shown by states. Examining the median improvement is also important;
the median is the point at which there are an equal number of data points whose values lie above
and below that value. The median values presented in Table A-l also largely suggest that the
performance of ERP groups with regard to EBPIs has improved over time. In all but one case,
the average improvement was larger (in some cases substantially) than the median. The
minimum and maximum percentage point changes (also shown in the table) help shed light on
-------
why: because the absolute value of the maximum percentage point change is for all states
substantially higher than the absolute value of the minimum percentage point change. Such large
percentage point changes may represent a substantial environmental impact, even if associated
with just a limited number of indicators.
Why Utilize All Observed Values? EPA decided to average all observed values, rather than
just statistically significant changes, which is an approach sometimes taken in summary analyses
like this. EPA chose this approach in part because it is a more conservative one. If EPA had
chosen to summarize values only from significant results, the percentage point changes would
have been higher, because none of the observed decreases were statistically significant and the
statistically significant improvements are generally greater in magnitude than non-significant
ones. Further, EPA saw value in presenting all results, especially for ERPs that saw limited
statistically significant change in the first ERP cycle. Statistical significance is much less likely
to be observed at a 95% confidence level, used by most states, especially with the small sample
sizes most states have employed. (Sample sizes of most ERPs have ranged from 35 to 50 per
round of inspections.)
Should the Analysis Include Indicators for Which There Are Small Sample Sizes? Some
indicators may apply to only a small subset of facilities visited during random inspections. It is
fair to ask whether it is appropriate to include such indicators in the average analysis. In some
cases, excluding indicators with samples of under 10 facilities could noticeably change the
"average percentage point improvement" for a state, either up or down. For instance, one of the
indicators for Massachusetts' dry cleaners had a sample of 4 facilities in the baseline and 10 in
the post-certification inspections, and contributed a 55 percentage point change that was much
higher than other observed improvements. Excluding this indicator from the calculations would
have resulted in an "average percentage point improvement" for the dry cleaners ERP of negative
0.8. (This slightly negative result should not be viewed in isolation. The 2007 Report provides
data showing that the Massachusetts dry cleaners ERP made important progress in reducing
perchloroethylene waste and emissions in the first ERP cycle; dry cleaners were already
performing at high levels for 5 indicators, limiting the potential for improvement; and the "group
compliance score" discussed in section 3.2 of the 2007 Report shows improvement in the second
ERP cycle, after no change in the first cycle.)
EPA chose to include all indicators, regardless of sample size. Excluding any indicators would
have required setting a sample size threshold, which EPA did not feel was appropriate because it
would require an arbitrary determination of the threshold. Also, indicator-level sample size data
were not available for Florida's auto repair ERP. Consequently, it would not have been possible
to determine whether any of the Florida indicators required exclusion because of small sample
sizes.
EPA also considered developing a weighted average. A weighted average could account for
variations in sample sizes for individual EBPIs without excluding the results for EBPIs with
small sample sizes, by giving each EBPI a weight relative to the number of facilities for which it
was relevant. While EPA may consider also presenting weighted averages in future reports, this
approach was not pursued in this 2007 Report, for several reasons: (1) insufficient data were
available from Florida; (2) an index-based indicator for Maryland, showing its only statistically
-------
significant change, would have had to be excluded (see Maryland methodological notes for more
information on the indicator); and (3) a weighted average may in some cases create an
unintended bias, because it assumes that the importance of each EBPI is proportional to the
number of facilities to which that EBPI applies. To the contrary, states may indeed take this
issue into account when deciding upon what practices to track as EBPIs in the first place. States
may also consider, in the future, developing weights for EBPIs based upon relative importance,
but this requires assigning numerical values and also could inject bias.
1.1. SOURCES AND STATE-BY-STATE METHODOLOGICAL NOTES
DELAWARE, AUTO BODY. Source: Delaware Department of Natural Resources and Environmental Control.
Final Report. EPA 2002 State Innovation Pilot Grant Program December 2005. Notes: Delaware's report indicated
the state was assessing the significance of performance changes using a 90% confidence level, but the significance
levels of individual EBPIs were not provided in the Delaware report. Consequently, EPA utilized data in the
Delaware report to assess significance at a 95% confidence level, using recognized statistical techniques for one-
tailed hypothesis testing. A 95% confidence level was chosen for consistency with the data presented for other
states.
FLORIDA, AUTO REPAIR Source: Florida Department of Environmental Protection. "Compliance
Certification Program (CCP). Compliance Assistance for Auto Repair. PILOT PROJECT Preliminary Results."
PowerPoint presentation, Slides 36 and 37. 2003. Notes: None.
MAINE, AUTO BODY. Sources: (1) For proportions of facilities achieving EBPIs: Maine Department of
Environmental Protection, Office of Innovation. Auto Body Environmental Results Program: Final Report. May 7,
2007. (2) For determinations about statistical significance: Lippert, Sara. Maine Department of Environmental
Protection. "Maine EBPI Data-revised 1-07," unpublished preliminary results from Microsoft Excel spreadsheet. E-
mail communication with U.S. EPA contractor. January 11, 2007. Notes: None.
MARYLAND, AUTO REPAIR/AUTO BODY. Source: McGrath, Dennis. Report of the Analysis of Self-
Certification and Inspections of Park Heights Neighborhood Auto Mechanic and Auto Body Shops for the Park
Heights Environmental Compliance Assistance Program Schaefer Center for Public Policy (University of
Baltimore). Received as Attachment 7 to the Maryland Department of Environment's Park Heights Project Final
Report, prepared by Bernard A. Penner. June 30, 2004. Notes: (1) Maryland report provides results for 9 EBPIs, but
EPA excluded 4 EBPIs from this analysis. Results for those EBPIs were only provided for those facilities sampled
in both rounds of inspections. Results for other Maryland EBPIs (and the other ERPs presented in EPA's 2007
Report) are based upon random samples of all facilities, whether or not inspected in both rounds of inspections.
Including those four EBPIs in this analysis could create an upward bias in the findings, assuming such facilities are
likely to perform better than facilities that were not inspected in the initial round. (2) Three of Maryland's remaining
five EBPIs are similar to other ERP EBPIs, in that they measure the proportion of facilities meeting a certain
criterion (e.g., a compliance practice). However, two of the EBPIs incorporated in this analysis were presented as
Index scores, constructed from several related dichotomous (i.e., yes/no) questions: a "hazardous waste management
index" and a "painting operation compliance index." The percentage point increase associated with these index
scores represents the increase in the proportion of criteria being met by the average facility, which is slightly
different than the percentage point increase for individual EBPIs, but nonetheless seemed appropriate to include in
the calculation of average percentage point improvement. (3) With regard to the EBPI tracking discharge of
contaminated wastewater, the Maryland report acknowledged that three facilities in each round of inspections were
found to be discharging without a permit. However, the report did not provide relevant sample size in each round.
Consequently, actual proportions of facilities in compliance in each round could not be calculated. This, in turn,
meant the percentage point change could not be calculated. In this special case, EPA conservatively assumed the
percentage point change to be negative 0.1 to avoid potentially understating the number of EBPIs with negative
performance change. If EPA had assumed a percentage point change of zero, the number of EBPIs with negative
performance changes in the table would be lower. If EPA excluded the EBPI from analysis, the number of EBPIs
with negative performance changes in the table would be lower and Maryland's observed average indicator
improvement would be higher.
-------
MASSACHUSETTS, DRY CLEANERS. Source: Tetra Tech EM, Inc., for Massachusetts Department of
Environmental Protection. Environmental Results Program for Dry Cleaners and Photo Processors. Round 1 Versus
Round 2 and Serf-Certification Data Analysis. Final Report. Appendix G (detailed EBPI results) and page 8 (Table
2, listing EBPIs, direction of change, and significance associated with them). June 22, 2000. Notes: Five of
Massachusetts' EBPIs for dry cleaners are classified as "No Change, High Initial Compliance" in Table 2 of this
TetraTech EM, Inc., report (2000). EPA determined that classification meant that Massachusetts inspectors
observed 100% performance on such EBPIs in both rounds of data collection. EPA based that determination on
baseline inspection data for those EBPIs presented in Appendix A of TetraTech EM, Inc. (2001), the source cited
below for EBPI data from the Massachusetts printers ERP. Exact performance levels for those EBPIs are not
presented in TetraTech EM, Inc. (2000).
MASSACHUSETTS, PHOTO PROCESSORS. Source: Tetra Tech EM, Inc., for Massachusetts Department of
Environmental Protection. Environmental Results Program for Dry Cleaners and Photo Processors. Round 1 Versus
Round 2 and Serf-Certification Data Analysis. Final Report. Page 9 (Table 4, listing EBPIs, direction of change,
and significance associated with them) and Appendix K (detailed EBPI results). June 22, 2000. Notes: Three of
Massachusetts' EBPIs for photo processors are classified as "No Change, High Initial Compliance" in Table 4 of this
TetraTech EM, Inc., report (2000). EPA determined that classification meant that Massachusetts inspectors
observed 100% performance on such EBPIs in both rounds of data collection. EPA based that determination on
baseline inspection data for those EBPIs presented in Appendix A of TetraTech EM, Inc. (2001), the source cited
below for EBPI data from the Massachusetts printers ERP. Exact performance levels for those EBPIs are not
presented in TetraTech EM, Inc. (2000).
MASSACHUSETTS, PRINTERS. Source: Tetra Tech EM, Inc., for Massachusetts Department of Environmental
Protection. Environmental Results Program for Dry Cleaners and Photo Processors. Round 2 Versus Round 3 and
Self-Certification, and Printers Round 1 Versus Round 2 and Serf-Certification Data Analysis. Final Report Page 7
(Table 7, listing EBPIs, direction of change, and significance associated with them) and Appendix A (detailed EBPI
results). July 16, 2001. Notes: Two of Massachusetts' EBPIs for printers are classified as "No Change, High Initial
Compliance" in Table 7 of this TetraTech EM, Inc., report (2001). EPA determined that classification meant that
Massachusetts inspectors observed 100% performance on such EBPIs in both rounds of data collection. EPA based
that determination on baseline and post-certification inspection data for those EBPIs presented in Appendix A of
TetraTech EM, Inc. (2001).
RHODE ISLAND, AUTO BODY. Source: Enander, Richard T., et al. "Environmental Health Practice:
Statistically Based Performance Measurement." Pages 819-824. American Journal of Public Health. Volume 97.
2007. TVotes: None.
2. Massachusetts Group Compliance Scores
Section 3.2 of the 2007 Report provides a graph (Figure 5 in the 2007 Report) showing the
performance of Massachusetts ERP sectors over time, as measured by the "group compliance
score." As noted in the 2007 Report, the group compliance score is a measure of the extent to
which facilities are achieving compliance-related EBPIs, as observed by inspectors during
random visits to facilities. A score of 80%, for instance, would mean that, on average, each
facility is achieving 80% of the indicators that apply to it.
The group score is calculated by first calculating a facility-level compliance score, then
averaging facility compliance scores across all inspected facilities. For instance, imagine
inspectors visited 5 facilities, and evaluated the extent to which those facilities were achieving
desired performance on 10 EBPIs. (Note: ERP random samples typically consist of about 35 to
50 inspections; this small number of inspections is used only for the purpose of illustrating the
calculation.) Table A-2 (next page) provides example data, showing how the group compliance
score would be calculated for these imaginary facilities.
-------
Table A-2: Illustrative Calculation of Group Compliance Score
Facility ID
Facility A
Facility B
Facility C
Facility D
Facility E
EBPIs Achieved
3
5
4
6
7
Relevant EBPIs
10
10
8
10
7
Group Compliance Score
Compliance Score
30%
50%
50%
60%
100%
58%
Note that each facility's compliance score is based upon the EBPIs that are relevant to it. For
instance, not all EBPIs are relevant to the processes utilized at facilities C and E, above, so their
compliance score is based upon the relevant EBPIs they are achieving.
Readers may have encountered references to Massachusetts' group or sector "performance score"
in prior documentation about Massachusetts ERPs. Massachusetts has used the term
"performance score" to refer to performance on both compliance and pollution prevention (P2)
EBPIs. The term "group compliance score" is used in this 2007 Report to add greater clarity
about the makeup of each score. In the case of dry cleaners and photo processors, all the EBPIs
are compliance-oriented, so the group compliance score and group performance score are equal.
In the case of printers, however, 12 of the 26 EBPIs represent voluntary P2 practices. To enable
greater comparability to the compliance-oriented performance scores of the other two sectors,
Massachusetts has long reported three different performance scores for printers: (1) just
compliance EBPIs; (2) just P2 EBPIs; and (3) all EBPIs combined. Table A-3 (below) presents
each of those three scores over time for printers, along with the scores for dry cleaners and photo
processors.
Table A-3: Massachusetts Group Performance Scores (1997-2003)
Sector
Dry
Cleaners
Photo
Processors
Printers
Score Type
Compliance
Compliance
Compliance
Pollution
Prevention
(P2)
Total (All
EBPIs)
1997
84%
57%
—
—
—
1998
84%
71%
82%
45%
60%
1999
—
—
89%
55%
68%
2000
97%
96%
—
—
—
2001
—
—
—
—
—
2002
98%
98%
—
—
—
2003
—
—
86%
43%
61%
Readers should also recognize that changes in scores over time have not been assessed for
statistical significance, nor have confidence intervals been established for individual scores.
Doing so is possible, but would be resource-intensive after the fact, and was beyond the scope of
this 2007 Report.
Finally, readers should recognize that Massachusetts frequently reports these scores on a scale of
1 to 10, rather than on a percentage basis. For instance, the 1997 score for photo processors
-------
might be reported as 5.7 on a scale of 10, rather than 57%. The percentage basis is used in this
2007 Report because it was considered easier to communicate in this context.
2.1. SOURCES
PHOTO PROCESSORS AND DRY CLEANERS DATA. DeGabriele, Steven. Director, Business Compliance
Division, Massachusetts Department of Environmental Protection. "Massachusetts Environmental Results Program."
Slides 15-16. Presentation to Minnesota Pollution Control Agency. June 8, 2006.
PRINTERS DATA FOR BASELINE (1998) AND FIRST POST-CERTIFICATION ROUND OF
INSPECTIONS (1999). Tetra Tech EM, Inc., for Massachusetts Department of Environmental Protection.
Environmental Results Program for Dry Cleaners and Photo Processors. Round 2 Versus Round 3 and Self-
Certification, and Printers Round 1 Versus Round 2 and Self-Certification Data Analysis. Final Report. Pages 2,
23. July 16,2001.
PRINTERS DATA FOR SECOND POST-CERTIFICATION ROUND OF INSPECTIONS (2003). Peck,
Susan. Associate Division Director, Massachusetts Department of Environmental Protection Business Compliance
Division. "Re: resolution of data issues." E-mail communication to U. S. EPA contractor. April 27, 2007.
3. Massachusetts Dry Cleaners' Reductions in Perchloroethylene
Emissions and Waste
Section 3.3 of the 2007 Report presents estimates of average reductions in perchloroethylene
emissions and waste by dry cleaners of 32% and 28%, respectively, between the first and second
self-certification cycles. These reductions equate to an estimated 135-ton reduction in annual
perc emissions and a 151-ton reduction in annual perc waste. These estimates are based upon
self-certification data from dry cleaners describing their perchloroethylene usage and
perchloroethylene waste shipments. By subtracting the weight of waste shipments from usage
weight, Massachusetts estimated air emissions.
Based on the estimated air emissions, EPA also estimated the volume of perchloroethylene air
emissions that were avoided would have been enough to fill the lungs of over 3.6 million adults.
The two subsections below present additional information on the estimation of perchloroethylene
waste and emissions, and on the lung-related calculations.
Estimating Reductions in Perchloroethylene Emissions and Waste. The data underlying the
analysis are from the "ERP Credit Matrix" documents cited below. The analysis for the 2007
Report corrected minor calculation errors found in the "ERP Credit Matrix." Tables A-4 and A-5,
next page, present the detailed source data and calculations from this analysis, including how the
reductions were adjusted to reflect a drop in the number of dry cleaners from 623 in 1997 to 612
in 1998.1
:Figure of 612 dry cleaners in 1998 (provided in the "ERP Credit Matrix") is different from the 1998 population of
648 dry cleaners presented in Section 5 of this appendix. Section 5's figure is based upon more recent data provided
by Massachusetts. EPA was unable to resolve this inconsistency prior to publishing the 2007 Report, but the
inconsistency does not have any adverse impact on the 2007 Report's findings of substantial emissions and waste
reductions. Employing the figure of 648 dry cleaners for the 1998 population would signal a population increase
from 1997 to 1998, rather than the population decrease the analysis is currently based upon. Therefore, normalized
estimates of emissions and waste reductions would be larger than those presented here.
-------
Table A-4: Estimated Reductions in Average
Perchloroethylene Usage, Emissions and Waste among MA Dry Cleaners (1997-1998)
Amount of Usage/Waste/Emissions 1997 1998 Source of Figure
Universe of Certifiers
Perc Usage (gallons)
Perc Usage (Pounds)
Perc Usage (Tons)
Average Perc Usage (Gallons/Facility)
Percentage Reduction in Average Perc Usage (From Baseline)
Perc Waste (Pounds)
Perc Waste (Tons)
Average Perc Waste (Pounds/Facility)
Percentage Reduction in Average Perc Waste (From Baseline)
Perc Emissions (Pounds) [Usage Minus Waste]
Perc Emissions (Tons)
Average Perc Emissions (Pounds/Facility)
Average Perc Emissions (Tons/Facility)
Percentage Reduction in Average Perc Emissions (From Baseline)
623
144,807
1 ,950,550
975
232.4
n/a
1 ,097,084
549
1761.0
n/a
853,466
427
1369.9
0.68
n/a
612
99,787
1,344,131
672
163.1
29.9%
774,839
387
1266.1
28.1%
569,292
285
930.2
0.47
32.1%
From "ERP Credit Matrix," p. 2. (See
Sources, below.)
From "ERP Credit Matrix," p. 2.
Calculated. Basis is 13.47 Ibs/gallon, from
"ERP Credit Matrix," p. 3.
Calculated.
Calculated.
Calculated
From "ERP Credit Matrix," p. 2.
Calculated.
Calculated.
Calculated.
Calculated.
Calculated.
Calculated.
Calculated.
Calculated.
Table A-5: Estimated, Normalized Gross Reductions in
Perchloroethylene Usage, Emissions and Waste among MA Dry Cleaners (1997-1998)
Amount of Usage/Waste/Emissions 1997* 1998 Reduction (1997 minus 1998)
Perc Usage (gallons)
Perc Usage (Pounds)
Perc Usage (Tons)
Average Perc Usage (Gallons/Facility)
Percentage Reduction in Average Perc Usage (From Baseline)
Perc Waste (Pounds)
Perc Waste (Tons)
Average Perc Waste (Pounds/Facility)
Percentage Reduction in Average Perc Waste (From Baseline)
Perc Emissions (Pounds) [Usage Minus Waste]
Perc Emissions (Tons)
Average Perc Emissions (Pounds/Facility)
Average Perc Emissions (Tons/Facility)
Percentage Reduction in Average Perc Emissions (From Baseline)
142,250
1,916,110
958
232.4
n/a
1,077,713
539
1761.0
n/a
838,397
419
1369.9
0.7
n/a
99,787
1,344,131
672
163.1
29.9%
774,839
387
1266.1
28.1%
569,292
285
930.2
0.5
32.1%
42,463
571 ,980
286
69.4
n/a
302,874
151
494.9
n/a
269,105
135
439.7
0.2
n/a
* 1997 figures calculated by multiplying 1997 average usage/waste/emissions by 612, the number of dry cleaners in existence in 1998.
Readers should also recognize that the emissions and waste reductions occurred over the course
of only one year (from 1997 to 1998), so it reduces the likelihood that intervening factors other
than ERP played a significant role in the decrease. Further, certification data presented in the
"ERP Credit Matrix" also show that reductions continued in subsequent years.
Volume of Perchloroethylene Emissions, Expressed in Terms of Adult Lungs. To illustrate
the emissions reductions in more intuitive terms, EPA estimated the number of adults whose
lungs would be filled by the volume of perc emissions reduced. To do so, EPA divided the
-------
volume of the reduction in perchloroethylene emissions (17,660 m3) by the typical vital lung
capacity of an adult male (4.8 L, or 0.0048 m3). Volume of perchloroethylene emissions
reductions is derived by dividing the estimated weight of emissions reductions (122,064 kg) by
the estimated density of perchloroethylene vapor at 20°C (6.91 kg per m3); density of
perchloroethylene vapor is 5.76 times the density of air, which has a density of 1.2 kg per m3.
The exact, resulting calculation is 3,679,108 adults.
3.1. SOURCES
MASSACHUSETTS PERCHLOROETHYLENE EMISSIONS AND WASTE DATA. Massachusetts
Department of Environmental Protection. "MassDEP ERP Dry Cleaner Program Description ('ERP Credit
Matrix')." Corrected version of Addendum 1 to July 20, 2006, U.S. EPA memo from Granta Y. Nakayamato
Robert W. Varney regarding "Approval of Recognition and Resource Flexibility Credit for the Massachusetts
Environmental Results Program in the Dry Cleaner Sector." September 30, 2005 (with August 2006 edits).
VITAL LUNG CAPACITY, ADULT MALE. "Lung." Microsoft Encarta. Accessed October 8, 2007
http://encarta.msn.com/encvclopedia 761570316/Lung.html.
PERCHLOROETHYLENE VAPOR DENSITY RELATIVE TO AIR MIDSUN Group "Perchloroethylene
Material Safety Data Sheet (MSDS)." April 4, 2006.
4. Massachusetts Printers' Reduction in VOC Emissions Due to
Increased Use of UV Ink
Section 3.3 of the 2007 Report provides an estimated range for reductions in volatile organic
compound (VOC) emissions related to an observed increase in the proportion of printers utilizing
environmentally friendly ultraviolet (UV) inks. In 1998, Massachusetts inspectors observed that
1 out of 32 (3.1%) randomly visited, relevant printers were using UV ink. In 1999, the figure
was 3 out of 34 (8.8%). In 2003, 5 out of 22 (22.7%). The difference between 1998 and 1999 is
not statistically significant at a 95% confidence level, but the difference between 1998 and 2003
is.
To estimate the VOC reductions based upon the change in behavior, EPA utilized a spreadsheet
tool that Massachusetts' Department of Environmental Protection (DEP) had commissioned a
contractor to develop (see source notes, beginning next page). Calculations of environmental
outcomes in that spreadsheet tool rely upon (1) inspector-collected data, for compliance/pollution
prevention practices; (2) facility certification data, particularly for facility characteristics such as
size; and (3) assumptions, based on published information or professional judgment, for such
issues as emission rates associated with particular facility practices. In developing the UV ink-
related emissions reduction estimate, EPA used newer data representing the change in
performance, and the appropriate number of printers in the state's inventory, but did not alter any
other assumptions in the Massachusetts tool.
EPA based calculations upon a population of 723 printers at the time of the 1998 random
inspections. Reductions are not adjusted for population changes, but any impact is expected to
be minimal, because 728 printers were in operation at the time of the 2003 random inspections. If
anything, EPA's approach slightly understates potential emissions reductions.
-------
The spreadsheet tool provided a point estimate for emissions reductions, based upon the
observed change. EPA created a range, by developing a confidence interval for the difference in
proportions. We can be 95% confident that the increase in performance between 1998 and 2003
was between 1.0 and 38.2 percentage points. EPA created a range of estimated emissions
reductions by using the spreadsheet tool to create point estimates for a 1.0 percentage point
change and a 38.2 percentage point change.
That process yielded an estimated range of reductions from 52,428 gallons to 2,002,750 gallons
per year. EPA decided to convert this range into the more commonly used metric for VOC
emissions—tons. The basis for this conversion is an estimate in Massachusetts' ERP compliance
assistance workbook (see sources, below) that density of printers inks ranges from 6 pounds per
gallon to 8 pounds per gallon. EPA multiplied the lower end of the emissions reductions range
by 6 pounds per gallon, and multiplied the upper end by 8 pounds per gallon. This resulted in
the estimated range of annual VOC emissions reductions of between 157 and 8011 tons.
In the process of developing Section 3.3 of the 2007 Report, EPA also explored a number of
other measures of change in environmental condition offered by Massachusetts DEP. These
measures related to potential changes in VOC emissions due to other printer practices, and to
potential changes in discharges of heavy metals and other toxic chemicals. A number of these
showed statistically significant changes (at a 95% confidence level) between 1998 and 1999, but
none of these measures showed statistically significant change when comparing 2003 to 1998.
EPA and Massachusetts agreed that it would be most appropriate to take a conservative approach
and only report results for measures showing statistically significant change (from 1998 to 2003)
at a 95% confidence level, the confidence level historically used by Massachusetts DEP.
4.1. SOURCES
METHODOLOGY FOR CALCULATING CONFIDENCE INTERVAL FOR THE DIFFERENCE IN
PROPORTIONS. U.S. EPA. ResultsAnalyzer2006.xls. 2006. Note: The Results Analyzer is a tool for ERP
states to calculate results. The formulas used to calculate a confidence interval for the difference are based upon the
following: Kish, Leslie. Survey Sampling, p. 41. New York: John Wiley & Sons, Inc., 1965.
METHODOLOGY FOR ULTRAVIOLET INK OUTCOME. Tetra Tech EM, Inc., prepared for Massachusetts
Department of Environmental Protection. "Massachusetts Department of Environmental Protection Environmental
Results Program Environmental Outcomes for Printers." Microsoft Excel spreadsheet. Provided by Susan Peck of
Massachusetts Department of Environmental Protection by e-mail as file named "Printing Environmental Outcomes
final with ERP numbers.xls." E-mail communication to U.S. EPA contractor. December 7, 2006.
PROPORTIONS OF FACILITIES UTILIZING ULTRAVIOLET INK (1998 and 1999). Tetra Tech EM, Inc.,
for Massachusetts Department of Environmental Protection. Environmental Results Program for Dry Cleaners and
Photo Processors. Round 2 Versus Round 3 and Self-Certification, and Printers Round 1 Versus Round 2 and Self-
Certification Data Analysis. Final Report Page 7 and Appendix A. July 16, 2001. Note: Appendix provides
detailed performance data, by question number. Microsoft Excel spreadsheet cited in the previous paragraph
("Printing Environmental Outcomes final with ERP numbers.xls.") identifies question 5 Ic as the relevant question
for usage of ultraviolet ink. Table 7 on page 7 of the July 16, 2001, Final Report shows that printers' performance
with regard to question 5 Ic increased from Round 1 to Round 2, and that the increase was not statistically
significant.
10
-------
PROPORTIONS OF FACILITIES UTILIZING ULTRAVIOLET INK (2003). Peck, Susan. Massachusetts
Department of Environmental Protection, "cl.xls," unpublished Microsoft Excel spreadsheet. E-mail
communication with U.S. EPA contractor. December 7, 2006. Note: Ultraviolet ink usage data is associated with
checklist question number 64, in row 112 of the "prcyOS" worksheet of the Microsoft Excel spreadsheet.
PRINTERS POPULATION IN 1998 AND 2003. Peck, Susan. Massachusetts Department of Environmental
Protection, "erp universes by year.xls," unpublished Microsoft Excel spreadsheet. E-mail communication with
U.S. EPA contractor. March 23, 2007.
RANGE OF DENSITIES OF PRINTERS INK. Massachusetts Department of Environmental Protection.
Printers Environmental Certification Workbook. Page 54. May 25, 2004.
5. Comparing Massachusetts and Michigan Approaches for
Regulating Dry Cleaners
Section 4.9 of the 2007 Report compares the cost-effectiveness of the approaches to regulating
dry cleaners used by Massachusetts and Michigan. To do so, the 2007 Report compares 2006
baseline ERP results for Michigan to the most recent data from random inspections for
Massachusetts, 2002. (Massachusetts is conducting another round of random inspections in
2007, but those data were not available for this 2007 Report. EPA is not aware of any reason
that comparing data collected from the two different points in time would present a substantial
bias for the purpose of this comparison—e.g., EPA is unaware of any industry or regulatory
changes between 2002 and 2006 that would significantly bias performance.) The 2007 Report
also compares the number of full-time equivalent (FTE) employees needed to manage the
Michigan program with the number of FTEs required for the Massachusetts approach, on a per-
dry-cleaner basis. Overall, the analysis found no difference in environmental performance
between the two states' dry cleaners on six comparable indicators, and determined that
Massachusetts uses approximately 50% of the FTEs (on a per dry cleaner basis) of Michigan in
managing their dry cleaners program.
Performance Comparison. For each state, EPA examined EBPIs chosen by that state, and
looked for comparable measures collected by the other state. EPA worked with staff from the
two states to confirm that six measures were comparable. Measures/data were accepted for
comparison if the other state had similar question language, regardless of whether the other state
had selected the item as an EBPI. This examination revealed six questions that met the
following characteristics:
• EBPI for Massachusetts and/or Michigan,
• Language (and therefore data) that was closely comparable,
• Both states had relevant facilities to which the question applied, and
• Each state's regulatory approach could arguably have had an impact on performance.
11
-------
Four similarly worded EBPIs were excluded from the analysis. One related to compliance with a
particular air requirement by 1993; it was excluded because the deadline for full compliance
occurred before ERP was initiated.2 The other three excluded indicators related to wastewater
and hazardous waste issues that the Michigan Air Quality Division has not historically examined
during its annual inspection process; Michigan's Department of Environmental Quality (DEQ)
tracked those issues in its 2006 baseline results only because its pilot ERP effort involved
multiple environmental media. Performance on these indicators, therefore, would not be
reflective of Michigan's annual inspection approach. Several partially related EBPIs were also
found, but these were excluded from the analysis because of comparability concerns. Readers
should recognize that non-EBPI questions were not examined for comparability.
As noted in the 2007 Report, the ability to draw conclusions from the comparison is somewhat
limited as a result of the small number of indicators identified as comparable. A more
comprehensive analysis would be helpful in the future—for this sector and other sectors—but
was beyond the scope of this 2007 Report.
For each EBPI, EPA calculated the difference between the states in observed proportions of
facilities in compliance. On average, 82% of dry cleaners visited by Michigan inspectors were
achieving the EBPIs in question; for Massachusetts, the figure is also 82%. A look at
performance on individual EBPIs shows similar equality: each state's randomly inspected dry
cleaners were performing better than those of the other state on three of the six indicators. Also,
the largest observed difference in compliance proportions is six percentage points, and none of
the observed performance differences are statistically significant. Significance was evaluated
using one-tailed tests at a 95% confidence level. Table A-6 (next page) presents data
summarizing each state's dry cleaners' performance on the selected indicators.
It is important to note that future analyses would benefit from both states having large sample
sizes, although this may not be practical in the face of state resource constraints. In this
comparison, the small size of Massachusetts' sample makes that state's figures much less precise
than Michigan's. As an illustration, a state conducting a simple random sample of 250 facilities,
out of a population of 1000, would generally expect a maximum margin of error of ±5.3
percentage points at a 95% confidence level. A state conducting a simple random sample of 25
facilities with the same population and confidence level would generally expect a maximum
margin of error of ±18.0 percentage points—over three times that of the other state.3 Such large
imprecision can mean that differences in performance must be rather large in order to be deemed
statistically significant.
Specifically, the question relates to whether all dry-to-dry machines installed before December 9, 1991, have a
refrigerated condenser or a carbon adsorber that was installed prior to September 22, 1993. The removal of this
question has negligible impact on the overall conclusions. Michigan observed 88.4% compliance and Massachusetts
observed 90% compliance with this requirement. The difference is not statistically significant.
3 Calculations based upon a Score confidence interval. (Agresti and Coull. "Approximate is Better than 'Exact' for
Interval Estimation of Binomial Proportions." The American Statistician. Volume 52, Number 2, Pages 119-126.
1998.)
12
-------
Table A-6: Michigan and Massachusetts Performance Comparison
Michigan DEQ Inspector
Checklist Question
Number and Language
1.03. Is the dry cleaning
machine door kept closed,
except for loading and
unloading?
1.04. Does facility keep a
log of the gallons of perc
purchased each month?
1.07. Does facility inspect
the following components of
the machine weekly for
leaks?
(All hose and pipe
connections, fittings,
couplings, and valves; Door
gaskets; Filter gaskets;
Pumps; Solvent tanks and
containers; Muck cookers,
stills; Water separator;
Exhaust dampers; Diverter
valves; Cartridge filter
housing)
1.13. Do all dry to dry
machines installed after
12/9/91 have a refrigerated
condenser?
1.16. Is the date and
temperature sensor
monitoring results recorded
weekly?
3.10. Are containers in good
condition and kept closed
except when adding or
removing waste?
Average
# Relevant
Sample Size*
255
255
253
230
231
246
Observed %
Compliant
(Weighted)**
98.8%
65.7%
78.6%
90.3%
65.5%
93.8%
82.1%
Massachusetts DEP
Inspector Checklist
Question Number and
Language
8. Are all machine
doors closed
immediately after
transferring articles and
kept closed at all times
except during
maintenance?
1. Are previous
month's perc purchases
recorded on the first
day of each month?
10. Is leak detection
performed weekly,
following the workbook
protocol and using
proper leak detection
equipment?
19. Do all dry to dry
machines installed after
12/9/91 have a
refrigerated condenser?
20c. Is the temperature
at the end of cycle
measured on the outlet
side of the refrigerated
condenser and
recorded weekly?
38. Are containers in
good condition and kept
closed?
Average
# Relevant
Sample Size*
23
23
22
18
21
23
Observed %
Compliant
100.0%
69.6%
72.7%
94.4%
61.9%
91.3%
81.7%
Observed Difference, in
Percentage Points***
(MA % minus Ml %)
1.2
3.9
-5.9
4.1
-3.6
-2.5
-0.5
* "Relevant Sample Size" refers to the number of inspected facilities for whom the question was relevant, because not all regulatory issues apply to all
facilities.
** Michigan DEQ conducted a stratified random sample, with each of Michigan's four inspectors visiting a random set of facilities in the geographic region
assigned to that inspector. Total number of inspections conducted was 262. Because the number of sites visited by each Michigan inspector was not
proportional to that region's share of the overall population, proportions have been weighted according to population share. This approach was not
necessary for the Massachusetts sample, which was a simple random sample consisting of 25 inspections.
*** Readers may observe some minor discrepancies, due to rounding.
13
-------
Comparison of State Effort, on a Per-Dry-Cleaner Basis. EPA developed estimates of staff
hours utilized per dry cleaner regulated, using each program's estimated full-time equivalent
(FTE) employees utilized in dry cleaner management and the populations of dry cleaners in each
state during the time frame examined for FTE usage. The analysis found that the Massachusetts
approach, on average, has utilized approximately 50% of the staffing utilized under the Michigan
approach. The subsections that follow explain, in detail, how EPA arrived at this figure.
Massachusetts Hours per Dry Cleaner Estimate. Massachusetts Department of Environmental
Protection (DEP) estimates it expends a maximum of 2 FTEs (as defined by that state) per year
to manage its perc dry cleaners, and approximately 1.5 FTEs in years in which it does not
conduct random inspections. On that basis, Massachusetts' average FTE usage for the timeframe
of this comparison can be calculated as 1.83 per year, because the state conducted four rounds of
random inspections during the six years from the start of ERP in 1997 through 2002, the last year
for which performance data are available.
EPA then converted Massachusetts FTEs into hours worked per year, for comparability to
Michigan. Based on feedback from the state, EPA assumed Massachusetts ERP personnel
typically earn 20 days annual leave per year. Massachusetts personnel also receive 13 holidays
per year, and earn 15 sick days and three personal days per year. EPA assumed, for both states,
that personnel utilized 100% of their annual leave, 50% of their sick leave, and 33% of personal
leave. Thus, out of 260 possible working days, a typical Massachusetts ERP staff person would
work 218.5 days. Applying Massachusetts' standard 7.5-hour workday, the Massachusetts FTE
is estimated equivalent to 1638.75 hours per year. Multiplying that times the estimated average
FTE usage of 1.83 per year results in average annual staff effort of 3004.4 hours per year.
To normalize staff effort by the number of dry cleaners, EPA divided the annual staff effort by
622, which is Massachusetts' average number of perchloroethylene dry cleaners for 1997-2002.
Those are the basis years for the average FTE data used in this analysis. That calculation results
in an estimate of 4.828 hours per dry cleaner.
Table A-7, below, presents the number of Massachusetts perchloroethylene dry cleaners for the
years 1997-2006. The population size has declined over time, but using a different population
does not have a substantial impact on the findings. For instance, using 2006's population of 560
dry cleaners—the lowest since 1997—results in an estimate of 5.365 hours per dry cleaner, still
just 56% of Michigan's effort estimate.4
Table A-7: Massachusetts Population of Perchloroethylene Dry Cleaners (1997-2006)
Year
Number of Dry Cleaners
1997
623
1998
648
1999
650
2000
637
2001
633
2002
543
2003
571
2004
591
2005
571
2006
560
As noted in footnote 1 in Section 3 of this appendix, the 1998 population figure for dry cleaners presented in this
section (648) is different from the number used in Section 3 of this appendix (i.e., 612), because of discrepancies in
the sources used for each analysis. EPA was unable to resolve this inconsistency prior to publishing the 2007
Report. However, utilizing the alternate population figure in this analysis has a negligible impact on the findings.
Specifically, using 612 dry cleaners would result in a Massachusetts effort estimate of 4.875 hours per dry cleaner
(51% of Michigan's).
14
-------
Michigan Hours per Dry Cleaner Estimate. Michigan DEQ utilized approximately 4.75 FTEs
(as defined by that state) in 2006 to manage its dry cleaners program, inspecting each facility
once a year (including both facilities that use perc and those using other solvents). EPA
understands 2006 to have been a historically typical year of FTE usage for the Michigan dry
cleaners program. (Note: The Michigan dry cleaners program experienced budget cuts in late
2007.)
As it did with the Massachusetts data, EPA converted Michigan FTEs into hours worked per
year. Based on feedback from the state, EPA assumed Michigan dry cleaners personnel earn
19.5 days annual leave per year. Just like Massachusetts, Michigan personnel receive 13
holidays per year. Michigan personnel also earn 13 sick days and no personal days per year. As
with Massachusetts, EPA assumed that Michigan dry cleaners personnel utilized 100% of their
annual leave and 50% of their sick leave. Thus, out of 260 possible working days, a typical
Michigan dry cleaners program staff person would work 221 days each year. Applying
Michigan's standard 8-hour workday, the Michigan FTE for the purpose of this analysis is
estimated equivalent to 1768 hours per year. Multiplying that times the 2006 estimated FTE
usage of 4.75 results in an estimated 2006 staff effort of 8398 hours.
To normalize staff effort by the number of dry cleaners, EPA first accounted for the fact that
only 810 of Michigan's 878 dry cleaning establishments in 2006 (92.3%) utilized
perchloroethylene. EPA assumed that staff time is proportionally distributed across all dry
cleaners, so that 92.3% of the total staff effort, or 7747.6 hours, is directed at dry cleaners using
perchloroethylene. Dividing this figure by the number of perchloroethylene dry cleaners results
in an estimate of 9.565 hours per dry cleaner.
Comparison of State Effort per Dry Cleaner. EPA completed the analysis by dividing the
Massachusetts normalized effort (4.828 hours per dry cleaner) by the Michigan normalized effort
(9.565 hours per dry cleaner), with a result of 50.48%, rounded down to 50% for the purposes of
the 2007 Report. Looked at another way, the Massachusetts perchloroethylene dry cleaners
population would have to drop to 314, without any change in Massachusetts effort, for the per-
dry-cleaner effort in each state to be the same.
5.1. SOURCES
MASSACHUSETTS DRY CLEANERS 2002 PERFORMANCE DATA. Peck, Susan. Massachusetts
Department of Environmental Protection, "cl.xls," unpublished Microsoft Excel spreadsheet. E-mail
communication with U.S. EPA contractor. December 7, 2006.
MASSACHUSETTS FTE ASSUMPTIONS. Peck, Susan. Massachusetts Department of Environmental
Protection. "Re: FTE definition assumptions." E-mail communication with U.S. EPA contractor. March 7, 2007.
MASSACHUSETTS FTE ESTIMATE. Peck, Susan. Massachusetts Department of Environmental Protection.
"Re: outstanding information for Biennial Report." E-mail communication with U.S. EPA contractor. February 16,
2007.
MASSACHUSETTS NUMBER OF DRY CLEANERS. Peck, Susan. Massachusetts Department of
Environmental Protection, "erp universes by year.xls," Microsoft Excel spreadsheet. E-mail communication with
U.S. EPA contractor. March 23, 2007.
15
-------
MICHIGAN EBPI LIST, FTE ESTIMATE, AND NUMBER OF DRY CLEANERS. Ostrowski, James A
Michigan Department of Environmental Quality. Michigan Department of Environmental Quality Dry Cleaning
Program 2006 Inspection Summary. January 19, 2007.
MICHIGAN FTE ASSUMPTIONS. Ostrowski, James. Michigan Department of Environmental Quality. "Re:
working hours per FTE." E-mail communication with U.S. EPA contractor. March 7, 2007.
MICHIGAN RESULTS DATA (2006). (1) Information on each stratum and sample size, by inspector:
Ostrowski, James. Michigan Department of Environmental Quality. "Michigan ERP Baseline Inspection Results
Summary Table." Unpublished document provided via e-mail communication with U.S. EPA contractor.
November 30, 2007. (2) Performance data for each question, by inspector: Ostrowski, James. Michigan
Department of Environmental Quality. "ERP-Total Summary BY INSPECTOR.xls," unpublished Microsoft Excel
spreadsheet. E-mail communication with U.S. EPA contractor. November 27, 2007.
6. EPA's Data Collection and Verification Process
EPA requested from states the most up-to-date data for use in the development of the 2007
Report. As much as possible, EPA utilized existing state data and analysis, as reported by states,
because re-analyzing all results was beyond the scope of the 2007 Report and appendix. For the
limited number of data inconsistencies that were identified, EPA worked closely with states to
resolve them, when possible. When EPA was not able to resolve inconsistencies, EPA in some
cases chose not to utilize such data. Whether or not data with unresolved inconsistencies were
utilized, EPA strived to transparently identify such inconsistencies for readers and (where such
data were utilized in the 2007 Report) to note how EPA handled such inconsistencies in its
analyses.
All data presented in the 2007 Report, its executive summary, and this appendix underwent a
quality assurance process to ensure calculations were properly conducted and to ensure data were
appropriately characterized and accurately transcribed from state sources. Further, EPA
provided several opportunities for review and comment to each of the seven states whose ERP
data are presented in the 2007 Report, its executive summary, and this appendix: Delaware,
Florida, Maine, Maryland, Massachusetts, Michigan, and Rhode Island. Their input is reflected
in these documents. Reviewers from each state are identified below.
Delaware.
Kimberly Chesser
Small Business Ombudsman
Department of Natural Resources and Environmental Control
Florida.
Michael Redig
Bureau of Solid and Hazardous Waste
Department of Environmental Protection
Maine.
Sara Lippert
Office of Innovation
Department of Environmental Protection
16
-------
Maryland.
Bernard Penner
Environmental Crimes Unit
Office of the Attorney General
Formerly of Department of the Environment
Massachusetts.
Steven DeGabriele, Susan Peck and Paul Reilly
Business Compliance Division
Department of Environmental Protection
Michigan.
James Ostrowski
Environmental Science and Services Division
Department of Environmental Quality
Karen Kajiya-Mills
Air Quality Division
Department of Environmental Quality
Rhode Island.
Richard Enander, Ph.D.
Office of Technical and Customer Assistance
Department of Environmental Management
17
------- |