United States
Environmental Protection
Agency
Office of Policy,
Economics and Innovation
(1807T)
August 2010
EPA-100-R-10-003
Evaluation of the
EPA Region 1
New England
Marina Initiative
Promoting Environmental Results
•4
Through Evaluation
-------
Acknowledgements
This evaluation was conducted by Industrial Economics, Incorporated (lEc) and Eastern
Research Group, Inc. (ERG) for EPA's Office of Policy, Economics, and Innovation under
Contract EP-W-04-023. An Evaluation Team guided the effort consisting of Larry Wells of
EPA's Region 1, Matt Keene of EPA's Office of Policy, Economics, and Innovation; and Lou
Nadeau of ERG, Inc. Matt Keene also served as the technical program evaluation advisor.
Kimberly Damm of Brown University provided valuable support in editing and reviewing the
final report. We would also like to thank the representatives from state government, marina
owners and others individuals committed to the health of marina communities who agreed to be
interviewed for this evaluation.
This report was developed under the Program Evaluation Competition, co-sponsored by EPA's
Office of Policy, Economics and Innovation and the Office of the Chief Financial Officer. To
access copies of this or other EPA program evaluations, please go to EPA's Evaluation Support
Division's website at http:www.epa.gov/evaluate.
-------
TABLE OF CONTENTS
TABLE OF CONTENTS n
TABLES AND FIGURES ERROR! BOOKMARK NOT DEFINED.
ACRONYMS 3
EXECUTIVE SUMMARY 4
CHAPTER 1: INTRODUCTION 6
I. Purpose and Objectives of the Evaluation 6
II. Overview of the New England Marina Initiative 7
III. Evaluation Questions 10
CHAPTER 2: METHODOLOGY FOR THE NEW ENGLAND MARINA INITIATIVE EVALUATION 12
I. Data Sources 12
II. Data Collection 12
III. Analytical Approaches 15
IV. Confounding Factors 16
CHAPTER 3: RESULTS AND FINDINGS 20
Question 1: Regulatory Compliance 20
Question 2: Best Environmental Practices (BEPs) 26
Question 3: Usefulness of the Activities and Materials 30
Question 4: The Checklist 31
Question 5: Environmental Health of Marina Communities 33
Question 6: Improvements to the Initiative 34
Question 7: Transferring the Initiative 35
CHAPTER 4: DISCUSSION 36
CHAPTER 5: RECOMMENDATIONS 41
Recommendation 1: Clarify the theory of change 41
Recommendation 2: Emphasize near-term, practical outcomes 41
Recommendation 3: Prioritize goals and objectives 41
Recommendation 4: Specify environmental objectives 41
Recommendation 5: Formalize the Initiative 42
REFERENCES 43
APPENDIX A: INTERVIEW GUIDE FOR STAKEHOLDER INTERVIEW A-l
APPENDIX B: EPA REGION 1, MARINA CHECKLIST FOR DATA COLLECTION DURING SITE VISITS B-l
APPENDIX C: CUSTOMER SATISFACTION SURVEY QUESTIONNAIRE C-l
APPENDIX D: PROPENSITY SCORE MATCHING ANALYSIS D-l
APPENDIX E: PROPENSITY SCORE MATCHING E-l
-------
TABLES AND FIGURES
Figure 2-1. EPA Region 1 New England Marina Initiative Logic Model 9
Table 3-1. Final Evaluation questions and the components of the logic model to which they correspond
11
Table 4-1. Link Between Refined Evaluation Questions, Data Sources, and Analytical Approaches and
Tools 18
Table 5-1. Propensity Score Matching Analysis Results for Regulatory Outcomes 21
Table 5-2. Self-Disclosed Violations and EPCRA Tier II Filings Among Marinas, Before 2001, 2001-
2004, and After 2004 25
Table 5-3. Propensity Score Matching Analysis Results for Best Environmental Practices Outcomes .27
Table 5-4. Number of Respondents That Were "Very Satisfied" or "Satisfied" With Initiative
Materials' Ability to Address Different Best Environmental Practices [a] 30
Table 5-5. Material Potentially Developed by State Using new England Marinas Initiative Assistances 1
Table 5-6. Percent of Respondents with Less than Three Years of Experience for Selected best
Environmental Practices 33
Table B-l. Categorization of Checklist Items Between regulatory Compliance and Best Environmental
Practices Outcome Categories B-4
Propensity Score Matching Analysis D-l
Figure D-l. 2004 Regulatory Compliance Indicators Assessed Qualitatively (Good, Moderate, and Poor) Grouped
By 2001 Qualitative Assessments D-6
Figure D-2. 2004 Best Management Practice Indicators Assessed Qualitatively (Good, Moderate, and
Poor) Grouped By 2001 Qualitative Assessment D-7
-------
ACRONYMS
BEP
BMP
EPA's STORE!
EPCRA
FY
ICR
LEPCs
LTM
MSDS
NGOs
NPDES
OPEI
OSHA
PSM
PWW
RCT
SERCs
SPCC
TIME
USGS
Best Environmental Practice
Environmental Management Plan
STOrage and RETrieval
Emergency Planning and Community Right to Know Act
Fiscal Year
Information Collection Request
Local Emergency Planning Committees
Long Term Monitoring
Material Safety Data Sheet
Non-Governmental Organizations
National Pollutant Discharge Elimination System
Office of Policy Economics and Innovation
Occupational Safety and Health Administration
Propensity Score Matching
Pressure Wash Wastewater
Randomized Control Trial
State Emergency Response Commissions
Spill Prevention, Control and Countermeasure Plan
Temporarily Integrated Monitoring of Ecosystems
United States Geological Survey
-------
EXECUTIVE SUMMARY
An evaluation of the EPA Region 1 New England Marina Initiative was selected as one of five
program evaluations in Fiscal Year (FY) 2008 under EPA's Office of Policy, Economics and
Innovation (OPEI) 2007 Program Evaluation Competition. This report presents a description of
the New England Marinas Initiative and the questions that the evaluation is designed to answer,
the methodology used to answer the evaluation questions, the findings of the study, and useful
conclusions and recommendations. The purpose of the evaluation is to assess the effectiveness
and transferability of the New England Marinas Initiative.
In 2001, EPA launched the New England Marinas Initiative to address the potentially significant
environmental impacts of marinas. The regionally coordinated initiative is designed to improve
marina environmental performance by implementing an effective regional education and
outreach campaign that improves awareness and compliance, increases the use of best
environmental practices (BEPs), and enhances the current assistance provider network to help
achieve sustained industry-wide environmental support. The evaluation was conducted between
September 2007 and May 2009 based on behavioral data collected between 2001 and 2004 as
well as information collected after 2004.
The key findings of this evaluation were developed in response to seven key evaluation questions
regarding regulatory compliance improvements, increased use of BEPs, usefulness of program
activities and materials, improvements to the program's data collection instruments (i.e. marinas
checklist of regulatory compliance and BEPs), increased environmental health of marina
communities, improvements to program structure, and interest in applying the program in other
EPA Regions and states. To answer the seven evaluation questions, the evaluation team used
quantitative (propensity score matching analysis) and qualitative methods to analyze information
collected from stakeholder interviews, customer satisfaction surveys and application of the
marinas checklist tool to statistically valid samples of New England marinas.
Summary of findings include:
• Statistically significant increases in the percentage of facilities with:
o Spill prevention procedures in place for transferring oil within the facility
o MSDs used for training employees that handle hazardous chemicals
o NPDES storm water permits
o Spill Prevention, Control and Countermeasure Plan (SPCC) on site
o Assessment of the methods they're using to prevent oil and fuel releases
o Structural changes to riprap
o Structural changes to placement of filters with drains
• Statistically significant decrease in the percentage of facilities with:
o Spill prevention procedures in place for fuel dispensing
o Fuels, solvents and paints stored in a protected, secure location, away from drains
o Fuels, solvents and paints plainly labeled
o Biodegradable cleaners
• No statistically significant change in the majority of indicators used to measure
regulatory compliance and BEPs
-------
• Statistically significant evidence of the regulatory compliance activities and BEPs that
marina owners generally do and do not practice
• The marinas checklist is generally perceived as a useful tool to raise awareness among
marina operators of regulatory issues and BEPs
• Respondents agree that the checklist is not sufficiently specific for marina owners to
understand what it means to be in compliance; a state-specific checklist tool would
account for differences in regulations between states and reduce confusion.
• Though states conducted activities and developed materials that contributed to the New
England Marinas Initiative, states did not view them as part of the Initiative
• Overall satisfaction amongst the Initiative's participants with its materials and activities
• Stakeholders suggest: specific improvements to the checklist as well as workshops and
seminars, more on-site demonstrations of BEPs, and more outreach efforts
• Without more clearly defined objectives, the evaluation team could not identify
appropriate data sources for determining the program's impact on the health of marina
communities
• Interest in transferring the Initiative to other states and regions is unclear
Given the growing emphasis on the value of networks and collaborative partnerships in
achieving measurable environmental goals, the EPA Region 1 New England Marinas Initiative is
a valuable model that demonstrates lessons about developing stakeholder networks, improving
regulatory compliance and adoption of BEPs, articulating and clarifying program theory, and
measuring and evaluating performance. Overall, EPA, states, trade associations and marina
owners, and other stakeholders have established a strong foundation for the maintenance and
growth of the New England Marinas Initiative.
Recommendations from this evaluation are as follows:
• Clarify the Initiative's theory of change by clearly stating which activities and materials
are intended to lead to which desired change amongst marina owners and others.
• Emphasize near-term, practical outcomes by making changes in behavior a top priority
and the foundation of the Initiative's learning objectives; hone the effectiveness of
education and technical assistance tools and strategies through a combination of ongoing
measurement and evaluation
• Given the large number of desired outcomes for the Initiative and limited resources, use
the analysis of regulatory compliance and BEPs as well as stakeholder input documented
in this report to prioritize goals, objectives and activities
• Refine the Initiative's environmental objectives and related measurement and evaluation
work by identifying specific pollutants and/or ecological characteristics most relevant to
the health of marina communities in New England
• Actively brand and promote the New England Marinas Initiative to improve
communications with Initiative partners as well as states, regions and others that could
benefit from implementing a similar initiative or its basic components
Appendices include the following resources: Interview Guide for Stakeholder Interview,
Checklist for Data Collection During Site Visits, Customer Satisfaction Survey Questionnaire,
Propensity Score Matching Analysis and Method Description of Propensity Score Matching.
-------
CHAPTER 1: INTRODUCTION
This program evaluation assesses the effectiveness of the New England Marina Initiative (the
Initiative) at building cooperative networks that support improved regulatory compliance and the
adoption of best environmental practices (BEPs) among marinas. Started by EPA Region 1, this
regionally coordinated initiative builds cooperative networks and partnerships among federal and
state environmental agencies, states, regional trade associations, NGOs, and marinas to provide
educational and technical assistance to marina owners in the Northeast United States. The
evaluation was conducted between September 2007 and May 2009 based on behavioral data
collected between 2001 and 2004 as well as information collected after 2004 including:
stakeholder interviews, a customer-satisfaction survey and background research on the Initiative
and environmental monitoring databases. This report provides a description of the New England
Marinas Initiative, the evaluation questions and the methodology used to answer the questions, a
summary of results and findings, and conclusions and recommendations for future actions.
I. PURPOSE AND OBJECTIVES OF THE EVALUATION
The purpose of this evaluation is to assess the effectiveness and transferability of the New
England Marinas Initiative.
The evaluation of the New England Marinas Initiative was selected as one of five program
evaluations in FY 2008 under EPA's Office of Policy, Economics and Innovation (OPEI) 2007
Program Evaluation Competition. Region 1 received funding and technical assistance from the
Evaluation Support Division within OPEI in support of the evaluation. This document provides
the results of the evaluation. Specifically, the assessment is intended to help EPA Region 1:
• Determine the effect of the Initiative on the practices of marina owners with respect to
regulatory compliance and use of BEPs,
• Better understand what factors influence the practices of marina owners,
• Determine the usefulness of the Initiative's activities and materials,
• Determine the appropriateness of instruments that EPA Region 1 used to measure the
performance of the Initiative,
• Understand the influence of the Initiative on the environmental health of marina
communities, and
• Identify opportunities to improve and learn from the Initiative and determine States' and
Regions' interest in replicating the model.
EPA Region lisa primary audience for this evaluation. They will use this evaluation to inform
continued improvement of the New England Marinas Initiative as well as document lessons
learned thus far. Another primary audience for this evaluation are state and regional programs
and organizations similar to the Marinas Initiative with respect to compliance/best practices of
marinas, performance measurement, program design, building partnerships/networks, education
and outreach, and technical assistance. OPEI will use the results and learning from this
evaluation to inform planning, management and evaluations of other environmental programs
nationwide.
-------
This report presents: 1) a description of the New England Marinas Initiative and the questions
that the evaluation is designed to answer, 2) the methodology used to answer the evaluation
questions, 3) the findings of the study, and 4) conclusions and recommendations for the
audiences' use.
II. OVERVIEW OF THE NEW ENGLAND MARINA INITIATIVE
In 2001 EPA Region 1 launched the New England Marinas Initiative to address the potentially
significant environmental impacts of marinas. The regionally coordinated initiative is designed to
improve marina environmental performance by implementing an effective regional education
and outreach campaign that improves awareness and compliance, increases the use of best
environmental practices (BEPs), and enhances the current assistance provider network to help
achieve sustained industry-wide environmental support.
The Initiative focused on several categorical areas where marina owners are responsible for
managing potentially significant environmental hazards that arise in the operation of marinas:
• Management of hazardous wastes—Many marina by-products can pose a substantial or
potential hazard to human health or the environment when inappropriately managed.
Examples of substances that may need to be managed and disposed of according to
federal or state hazardous waste requirements could include waste gasoline, solvents,
lead-based paint chips, and waste batteries.
• Management of stormwater runoff—Stormwater discharges are generated by runoff from
land and impervious areas such as paved streets, parking lots, and building rooftops
during rain and snow. They often contain pollutants in quantities that could adversely
affect water quality. Most storm water discharges are point source discharges and require
coverage by a National Pollutant Discharge Elimination System (NPDES) permit.
• Management of waste oil and fueling facilities—A common stormwater pollutant is
petroleum hydrocarbons discharged by improper management of waste oil or spills
during boating and tank fueling.
• Other marina related environmental impacts include: boat sewage pumpouts; air
emissions from engines; non-hazardous waste generation (e.g., shrink wrap); transfer of
non-indigenous aquatic species by boaters; and use of toxic products for activity such as
cleaning, painting, and antifreeze protection.
Marinas are often small businesses and many lack the environmental expertise and resources to
cope with these potential hazards. In New England there are more than 1,200 marinas governed
by federal environmental regulations as well as state regulations that differ across the region.
The New England Marinas Initiative is designed to increase the flow of information and sharing
of best environmental practices. It focuses on maintaining and enhancing the current network
that provides compliance assistance, pollution prevention materials, outreach and training to
marina owners, staff and other partners including state environmental agencies, state and
regional trade associations and non-governmental organizations (NGOs). The Initiative aims to
educate participants and disseminate knowledge to marina owners about regulations and BEPs
with the following activities, products and services:
-------
• Workshops and training
• Technical assistance
• Regulatory interpretations
• Marinas Checklist
• Marina web site
• Fact sheets and other guidance documents
• Marina Environmental Management Plan Workbook
• Inventory of marina owners
To illustrate the overall design, planning and implementation of the Initiative, EPA Region 1
developed a logic model for the New England Marina Initiative. A logic model depicts a
program's theory. It is a picture of the relationships between a program's activities, outputs and
outcomes. A logic model documents and explains the elements of the program and their
interrelationships, providing an understanding about how the program works - ultimately, it
clarifies the underlying assumptions, expectations, and objectives of the program. Logic models
are used in evaluations to shape and prioritize the questions of interest for the evaluation.
Two distinct phases are depicted in the Initiative's logic model (Figure 2-1): (1) a project
planning and assessment stage; and (2) a project implementation stage. Key components in the
Initiative's logic model include:
• Resources—the basic inputs of funds, staffing, and knowledge dedicated to the program (not
depicted in Figure 2-1).
• Activities—specific processes and actions, such as education and technical assistance as well
as measurement, which focused primarily on outcomes in the implementation stage.
• Outputs—immediate products and services that result from activities often used to measure
short-term progress. These actions include workshops, the Marina website, and fact sheets.
• Customers—are the users of the activities and outputs, such as the state environmental
agencies and state and regional trade associations.
• Short-Term Outcomes—intended changes in awareness, attitudes, understanding, knowledge,
and skills. For state environmental agencies this means that agencies are aware and agree
that marina owners are in need of assistance. For marina owners and staff awareness of
regulations and BEPs are short-term outcomes.
• Intermediate Outcomes—changes in behavior, such as the collection of additional compliance
data, increased efforts to improve communication, and marina owners' adoption of BEPs.
Improved regulatory compliance and use of best environmental practices is anticipated.
• Long-Term Outcomes—changes in a condition. Improved regulatory compliance in the
marina sector, improved environmental performance from marina owners and
environmentally healthy marina communities are examples of long-term outcomes.
• Contextual/External Factors—potential influences not directly controlled by the Initiative or
its entities. For example, local or national economic forces, the industries that marinas serve
and that serve marinas or seasonal cycles in the business of marinas.
-------
Figure 2-1. EPA Region 1 New England Marina Initiative Logic Model
Program Goal: Improve marina environmental performance through implementing an effective regional education and outreach campaign that includes: improving awareness and compliance,
increasing implementation of best management practices, and enhancing the current assistance provider network to help achieve sustained industry environmental support.
Outcomes
Activities
Outputs
Customers
Shorter-term
Intermediate
Longer-term
PLANNING
Establish goals, identify resources &
gaps
Design strategic projects to achieve
goals
Identify Marina owners in Region 1
Monitor progress
OUTREACH/NETWORKING
Facilitate Regional Marina Workgroup
Conduct internal/external meetings
Prepare written updates
Establish partnerships
MEASUREMENT
Develop measurement process using
statistical principles, random sampling
techniques
Develop environmental indicators
Develop measurement tool (checklist of
indicators)
Conduct baseline on-site visits
Aggregate results
EDUCATION/TECHNICAL ASSISTANCE
• Conduct environmental workshops
• Conduct special demonstration training
events
• Provide technical assistance w/
Regional Marina Workgroup
• Develop technical assistance tools
• Develop & maintain Marina website
• Provide regulatory guidance
Inventory of marina
owners
Strategic plans
Meetings
Written updates
Communication
letters
Network of
Partners
• State
Environmental
Agencies
(Regional
Marina
Workgroup)
• State &
Regional
Marine Trades
Associations
•NEWMOA
•NEIWPCC
•NGOs
• Baseline data of
Marina performance
• Presentations on
aggregated results
Region 1& Partners'
awareness &
understanding of
marina's
environmental impact
improves
Region 1& Partners
explore ways to
address issues
Partners agree on
need for assistance in
this sector
Region 1 & Partners
capabilities to assist
marina owners is
enhanced
Region 1 & Partners
decide to provide
resources to address
issues
• Workshops
•Technical
assistance
• Marina Guidance
Documents
• Fact sheets
• Marina
Environmental
Management Plan
Workbook
• Marina Website
• Regulatory
interpretations
Improved
regulatory
compliance
from marina
owners
Improved
environmental
performance
from marina
owners
1
Health!
marina
commu
t
Note: Gray boxes in the "Project Planning and Needs Assessment Stage " reflect activities in the "Project Implementation Stage. "
-------
III. EVALUATION QUESTIONS
Evaluation questions were developed by the evaluation team and are based on those components of the
logic model most critical to program success and most important to Region 1. The question design also
considers the data and information available for evaluation. The evaluation is designed to answer seven
groups of questions which each address one or more components of the logic model (Table 3-1).
1. To what degree did regulatory compliance at in-scope marinas improve between 2001 and 2004?[c]
a. To what extent did the New England Marina Initiative activities and/or materials influence
regulatory compliance?
b. To what extent did non-program resources (e.g., consultants, non-program guidance documents)
influence regulatory compliance?[e]
c. To what extent did the state of the marina sector (for example, educationally, economically, and
demographically) influence regulatory compliance?
2. To what degree did in-scope marinas increase the use of best environmental practices (BEPs) between
2001 and 2004?[d],[e]
a. To what extent did the New England Marina Initiative activities and/or materials influence
implementation of BEPs? [f]
b. To what extent did non-program resources (e.g., consultants, non-program guidance documents)
influence implementation of BEPs? [g]
c. To what extent did the state of the marina sector (for example, educationally, economically, and
demographically) influence implementation of BEPs?
3. To what degree were the New England Marina Initiative activities and/or materials useful to in-scope
marinas in improving regulatory compliance or implementing best environmental practices (BEPs)?
a. To what extent were non-program resources (e.g., consultants, non-program guidance documents)
useful in improving regulatory compliance or implementing BEPs?
b. What materials did states develop, and what are states doing, as part of the New England Marinas
Initiative or as part of their own associated activities related to marinas?
4. What items should be added or dropped from the Region 1 Marina Checklist? [h]
a. Were any significant items missing from the Region 1 Marina Checklist?
5. To what extent did the New England Marina Initiative lead to healthier marina communities (e.g.,
improved environmental conditions)?
6. What recommendations for improvement to the program structure can be made?
a. Are there specific improvements for transferability to other EPA regions?
7. Among EPA Regions and States is there interest in applying the Region 1 Marina Initiative approach?
10
-------
Table 3-1. Final evaluation questions and the components of the logic model to which they correspond
Evaluation Questions
1 . To what degree did regulatory compliance at in-scope marinas improve
between 2001 and 2004?[c]
1 A. To what extent did the New England Marina Initiative activities
and/or materials influence regulatory compliance at in-scope
marinas?
IB. To what extent did non-program resources influence regulatory
compliance at in-scope marinas?
1C. To what extent did the state of the marina sector influence regulatory
compliance at in-scope marinas?
2. To what degree did in-scope marinas increase their use of best
environmental practices between 2001 and 2004?
2A. To what extent did the New England Marina Initiative activities
and/or materials influence implementation of best environmental
practices at in-scope marinas?
2B. To what extent did non-program resources influence implementation
of best environmental practices at in-scope marinas?
2C. To what extent did the state of the marina sector influence
implementation of best environmental practices at in-scope marinas?
3 . To what degree were the New England Marina Initiative activities and/or
materials useful to in-scope marinas in improving regulatory compliance
or implementing best environmental practices?
3a. To what extent were non-program resources useful to in-scope
marinas in improving regulatory compliance or implementing best
environmental practices?
3b. What materials did states develop, and what activities are states
doing, as part of the New England Marinas Initiative and their own
associated activities related to marinas?
4. What items should be added or dropped from the Region 1 Marina
Checklist?™
5. To what extent did the New England Marina Initiative lead to healthier
marina communities?
6. What recommendations for improvement to the program structure can be
made?
6a. Are there specific improvements for transferability to other EPA
regions?
7. Among EPA Regions and States is there interest in applying the Region 1
Marina Initiative approach?
Key Components of the Logic Model
• Improved regulatory compliance from marina owners
• Improved regulatory compliance from marina owners
• Enhanced support for marinas
• Improved regulatory compliance from marina owners
• Improved regulatory compliance from marina owners
• Increased implementation of best environmental
practices from marina owners
• Increased implementation of best environmental
practices from marina owners
• Enhanced support for marinas
• Increased implementation of best environmental
practices from marina owners
• Increased implementation of best environmental
practices from marina owners
• Enhanced support for marinas
• Enhanced support for marinas
• Enhanced support for marinas
• Improved regulatory compliance from marina owners
• Increased implementation of best environmental
practices from marina owners
• Healthier marina communities
• Enhanced support for marinas
• Enhanced support for marinas
• Enhanced support for marinas
lb| Many of these questions require a definition of marinas that are in-scope for this evaluation. Section 3.2 below discusses scope.
'°' The regulatory compliance items will be derived from the Checklist that was used in both the baseline and follow-up data collections.
[dl The best environmental practice items will be derived from the Checklist that was used in both the baseline and follow-up data collections.
'e' The original refined evaluation question was "To what degree did in-scope marinas increase their use of BEPs between 2001 and 2004,
between 2001 and 2007, and between 2004 and 2007?" However, due to issues in using the customer satisfaction ICR, statistically valid
sampling data were not available for 2007.
'^ The original refined evaluation question was "To what extent did the New England Marina Initiative activities and/or materials influence
implementation of best environmental practices at in-scope marinas between 2001 and 2007 and between 2004 and 2007?" However, due to
issues in using the customer satisfaction ICR, statistically valid sampling data were not available for 2007.
[gl The original refined evaluation question was "To what extent did non-program resources (e.g., consultants, non-program guidance documents)
influence implementation of best environmental practices at in-scope marinas between 2001 and 2007 and between 2004 and 2007?" However,
due to issues in using the customer satisfaction ICR, statistically valid sampling data were not available for 2007.
[h] Checklist version dated 7/19/04.
11
-------
CHAPTER 2: METHODOLOGY FOR THE NEW ENGLAND MARINA
INITIATIVE EVALUATION
The scope and content of the evaluation methodology is primarily shaped by the program's
theory (as represented in the logic model), the evaluation questions, and the data available to
answer the questions. Answering each evaluation question requires information and decisions
related to data sources, collection methods (e.g. survey, research), collection strategies (e.g.
sampling strategy), and the appropriateness of analytical tools and approaches. The systematic
consideration and documentation of decisions related to each of these issues results in an
evaluation methodology.
This evaluation methodology balances qualitative and quantitative data and analytical methods to
answer questions about the Initiative's outcomes, for instance regulatory compliance and use of
BEPs, and about the Initiative's process, such as measurement and educational activities and
outputs. The data available enables both quasi-experimental and non-experimental study
designs.
I. DATA SOURCES
Six data sources were used in this evaluation:
• Site-visit data—data collected from marinas during site visits in 2001 (baseline) and 2004
(follow-up). The site visit data represent a key focus of the Initiative itself. Specifically, the
Initiative developed a rigorous and statistically valid data collection method and collected
information from marinas before and after program implementation.
• Stakeholder interviews—interviews of Initiative stakeholders, including EPA, state agencies,
and trade associations.
• Customer satisfaction survey—a survey of marina owners in the New England region.
• Additional Data
o Data on self-disclosures of violations and Emergency Planning and Community
Right-to-Know Act (EPCRA) Tier IIfilings—EPA Region 1 supplied data on both
self-disclosures of violations and EPRCA Tier II filings.
o Information on installations of boat pressure wash control systems—representatives
from EPA Region 1, industry and trade associations were interviewed regarding the
installation of these systems.
o Environmental databases and other relevant contextual information—research for
secondary data on water quality and potential non-program influences (e.g., economic
state of the marina sector).
II. DATA COLLECTION
SITE VISITS
The New England Marina Initiative included site visits to 70 marinas in 2001 and another 70 in
2004. The purpose of the site visits was to collect statistically valid data on regulatory
12
-------
compliance and use of best environmental practices at marinas both before and after program
implementation. Full compliance was necessary in order for a marina to be considered in
compliance. During the visits the Region performed some compliance assistance when
requested. During the site visits, EPA staff collected data using the New England Marina
Initiative Checklist found in Appendix B. Table B-l of Appendix B provides a list of the data
elements collected through the checklist and whether those questions can be classified as
"descriptive," "regulatory compliance," or "best practices."
These data were provided in hardcopy form and used to create a Microsoft Access database form
to facilitate conversion to electronic data. The raw data was entered via a double entry procedure
to adhere to corporate quality assurance protocols. The data were coded to allow for statistical
analysis.
The data provide both baseline and follow-up data on a number of regulatory compliance and
best practices that the Initiative was designed to address. The marinas visited in the fall of 2001
differ from those visited in the fall of 2004. Thus, there are no cases where there are before and
after observations of the same marina. To make valid and reliable comparisons between the two
years' data, propensity score matching (PSM) methods are incorporated into the evaluation
methodology. PSM is discussed in Appendix E.
STAKEHOLDER INTERVIEWS
In-depth interviews with state agencies and trade associations' staff provide insight on external
factors impacting the state of the marina industry, feedback on stakeholders' opinions about
guidance materials and best environmental practices (BEPs), and recommendations to improve
the program. The interviews also helped to shape the customer satisfaction survey (next section).
The interview format allowed for respondents to provide detailed responses and for interviewers
to probe for more thorough responses. A copy of the complete interview guide used in the
stakeholder interviews can be found in Appendix A of this report.
Interviews lasted about one hour, and questions focused on:
• Past and present involvement with the Initiative
• Environmental challenges faced by marinas
• Economic obstacles threatening the marina industry
• Other factors affecting marinas
• Interviewee thoughts on the Marina Checklist
• Interviewee thoughts on other program materials and guidance documents
• Initiative influence on state activities
• Recommendations for the Initiative
These interviews were conducted during March and April of 2008 and then again in February
2009. Nine people were interviewed: six state agency staff members (in five interviews), two
trade association personnel members, and EPA Region 1's lead for the initiative.
13
-------
CUSTOMER SATISFACTION SURVEY
The evaluation method included a small-scale customer satisfaction survey for this evaluation
using EPA's generic Information Collection Request (ICR) (EPA ICR No. 1711.05, OMB
Control No. 2090-0019) from Office of Management and Budget. The purpose of the survey was
to collect information on:
• Awareness among in-scope marinas of the compliance assistance materials developed by the
Initiative,
• Use of the compliance assistance materials among in-scope marinas,
• Satisfaction among in-scope marinas with the compliance assistance materials,
• Suggestions for improvements to the compliance assistance materials from the in-scope
marinas, and
• Experience with a set of specific best environmental practices.
The survey was implemented as a mail survey and recipients were randomly selected. Prior to
selection, the list of marinas (provided by EPA Region 1) was sorted by state to ensure
proportional representation among the New England states. Each survey recipient was sent a
reminder letter approximately a week to 10 days following the initial mailing. A copy of the
survey can be found in Appendix C.
The survey was sent to 224 marinas (including replacements for out of scope entities) with the
intent of collecting a total of 86 valid responses. Twenty-two responses were received. There
were multiple reasons for the poor response:
• Timing—due to delays in the review process, the survey was sent to marinas during one of
the busiest times for marinas (late Spring/early summer). Thus, many marinas, which are
primarily small operations, had little time to respond to the survey.
• Marinas inventory—Region 1's inventory of New England Marinas, including physical
addresses, was used to develop the sampling frame. Twelve percent (27) of surveys were
returned to sender (not delivered).
• Survey letter—the survey letter sent to marina owners listed an incorrect phone number for
the survey help desk. This may have affected the response rate; however, the letter included
an email address and only one question concerning the survey was received via email. The
correct contact phone number and e-mail address for the survey help desk were provided on
the reminder letter. One marina operator inquired about the survey by phone.
ADDITIONAL DATA AND INFORMATION
• Self-disclosed violations data—a list of self-disclosed violations by marinas in the northeast
was provided (but not audited for completeness) by Region 1. It is administrative data
maintained by EPA Region 1 that may serve as an indicator regarding the effect of the
Initiative on self-disclosures and indicate the potential effect that the Initiative has had on
improving compliance and safeguarding public health.
14
-------
• EPCRA Tier IIfilings among marinas—this is a listing of EPCRA Tier II filings among
marinas. Facilities that are covered by EPCRA are required to file Emergency and Hazardous
Chemical Inventory Forms with their Local Emergency Planning Committees (LEPCs), State
Emergency Response Commissions (SERCs), and local fire departments each year. Marinas
are required to submit a "Tier II" form. These data may serve as an indicator of the effect of
the Initiative on Tier II filings and indicate the potential effect that the Initiative has had on
improving compliance and safeguarding public health.
• Information on Increased Use of Boat Pressure Wash Equipment Control Technologies—
boat pressure wash wastewater (PWW) treatment is a focus of the Initiative. When boats are
pressure washed, harmful residues, paints, and other materials may contaminate surrounding
land and water. Wastewater from processes like these is regulated under the Clean Water
Act, and marinas are required to have discharge permits or to capture and dispose of the
wastewater responsibly. EPA Region 1 supplied reference to information on the increased
installation of boat pressure wash equipment control technologies. The evaluation team
obtained information from Region 1, the Trade Associations, and several suppliers of PWW
treatment technologies for marinas to examine the relationship between the Initiative and the
number of system installations.
• Secondary Data Sources on Water Quality and External Factors—the evaluation identified
three potential data sources for assessing the effect of the Initiative on water quality: 1) U.S.
Geological Society (USGS) Water Data for the Nation1, 2) EPA National Water Quality
Inventory Report to Congress (305(b) reports)2, and 3) EPA's STORET (short for STOrage
and RETrieval) data3.
• External/Contextual Factors—The evaluation did not identify external data sources but did
discuss the impact of external factors on the Initiative's outcomes with stakeholders during
the interviews.
III. ANALYTICAL APPROACHES
Keeping an eye on the relationships between evaluation questions, data sources and analytical
methods is important to maintaining the focus of an evaluation - Table 4.1 provides a quick
reference to those relationships. The importance of the questions and the quality of the data
guide decisions about study design within an evaluation. This evaluation has access to a range of
data types that encourage the use of a variety of analytical tools.
First, the site visit data is based on a statistically valid sample surveyed by Region 1 and enables
the use of a quasi-experimental evaluation design to quantitatively answer some evaluation
questions related to the state of and changes in regulatory compliance and the use of BEPs in the
marinas sector. Specifically, this evaluation used the Propensity Score Matching (PSM) method
which compares outcomes between treatment and control groups in cases where a randomized
control trial (RCT) has not been used. See Appendix F for a detailed discussion on the use of the
PSM method in the evaluation. An alternative approach would have been standard statistical
hypothesis testing on the differences between two mean values. PSM is a more rigorous
1 http://waterdata.usgs.gov/nwis
2
http://www.epa.gov/305b/
3 http://www.epa.gov/storet/index.html
15
-------
approach to looking for program impacts since it involves comparing matched units over time,
thus mimicking an RCT.
Second, due to limitations on the sample size of potential respondents and the lack of potential
control groups (no stakeholder interviews were conducted prior to Initiative implementation),
non-experimental direct analysis was used to analyze data gathered through stakeholder
interviews and interviews on pressure wash control systems. Tools used to analyze qualitative
data included content analysis and recursive abstraction. Generally, content analysis is an
approach to systematically and objectively make inferences, using quantitative or qualitative
techniques, by identifying specific characteristics of messages (e.g. the information recorded
during interviews). Recursive abstraction is the repeated distillation of data and information to
create compact summaries of qualitative information that emphasizes the key messages of
datasets. The stakeholder interview questions primarily collect qualitative information. From
interview notes, responses were summarized, condensed and categorized by common themes,
evaluation question, and stakeholder group. The framework for organizing information consisted
of a series of summary documents to catalog themes and corresponding passages. This format
enabled comparisons between interview responses within and across stakeholder groups.
Preliminary findings and recommendations were reviewed by the evaluation team.
Third, the customer satisfaction survey was sent to a statistically valid sample of the population;
however, the low response rate confounds the opportunity to make statistical inferences related
to the satisfaction of marina owners with the Initiative. The information gleaned from the survey
was coded and organized for data tabulation. Due to gaps in data and data credibility concerns,
the available Tier II and self disclosure violations data was also not suited for inferential
statistics and was similarly tabulated.
IV. CONFOUNDING FACTORS
The evaluation's results and recommendations are influenced by variety of factors that
confounded or limited data access and collection as well as analysis. First, the evaluation team
stated a clear interest in analyzing the relative effectiveness of the program's various education
and assistance tools (e.g. website, workbook, workshops). Given the resources allocated to this
evaluation and the time necessary to acquire an Information Collection Request to support a use
of a survey instrument that investigates behavior change, evidence of the relative effectiveness of
education and outreach tools is based on interviews of key informants and is useful in
prioritizing future planning efforts. Self disclosed and Tier II data was provided late in the
evaluation or after the evaluation was complete. The data is tabulated and the descriptive
statistics can be used to indicate program contribution and guide future planning and action.
Because the customer satisfaction survey suffered from a low response rate, the data it generated
could not be analyzed to provide statistically valid information on satisfaction. Results from the
Propensity Score Matching (PSM) analysis should be considered in light of its limitations. First,
State programs that may have affected the effectiveness of the Initiative are not accounted for in
the analysis. Second, the results of the PSM analysis are too narrow to be used in isolation and
the evaluation employs multiple methods to attempt to take account for context and non-
observed and unobservable variables that the PSM method does account for - PSM can not
match unmeasured variables. The value of the qualitative analysis of stakeholder interviews is
16
-------
limited by the relatively small number of interviews and findings may not be representative of all
program participants. In conserving resources for the evaluation, the evaluation method was
deliberate in selecting key individuals to interview based on their specific knowledge and
experience with the Marinas Initiative. We cannot assume that their experiences and perspectives
are necessarily representative of all involved, and the individuals selected to participate in an
interview may have vastly different experiences and views than those who were not selected.
17
-------
Table 4-1. Link Between Refined Evaluation Questions, Data Sources, and Analytical Approaches and Tools.
Refined Evaluation Question Based on Revised Logic Model
1. To what degree did regulatory compliance at in-scope
marinas improve between 2001 and 2004?
1 A. To what extent did the New England Marina Initiative
activities and/or materials influence regulatory
compliance at in-scope marinas following
implementation?
IB. To what extent did non-program resources (e.g.,
consultants, non-program guidance documents)
influence regulatory compliance at in-scope marinas
following implementation of the Initiative?
1C. To what extent did the state of the marina sector (for
example, educationally, economically, and
demographically) influence regulatory compliance at in-
scope marinas following implementation of the
Initiative?
2. To what degree did in-scope marinas increase their use of
best environmental practices between 2001 and 2004?
2A. To what extent did the New England Marina Initiative
activities and/or materials influence implementation of
best environmental practices at in-scope marinas?
2B. To what extent did non-program resources (e.g.,
consultants, non-program guidance documents)
influence implementation of best environmental
practices at in-scope marinas?
2C. To what extent did the state of the marina sector (for
example, educationally, economically, and
demographically) influence implementation of best
environmental practices at in-scope marinas following
implementation of the Initiative?
3. To what degree were the New England Marina Initiative
activities and/or materials useful to in-scope marinas in
improving regulatory compliance or implementing best
environmental practices?
Data Sources
See below
• Site visit data
• Stakeholder interviews
• Serf-Disclosure data
• Stakeholder interviews
• Information on the
installation of boat pressure
wash control technologies
• Stakeholder interviews
• Information on the economic
state of the marina sector
See below
• Site visit data
• Stakeholder interviews
• Customer satisfaction survey
• Stakeholder interviews
• Stakeholder interviews
• Information on the economic
state of the marina sector
• Stakeholder interviews
• Customer satisfaction survey
Analytical Approaches
See below
1 . Statistical comparisons, using Propensity Score Matching, of the 200 1
and 2004 values for compliance-related items on the site visit data.
2. Content analysis of site visit data and PSM results.
3 . Tabulation of serf-disclosure and TIER II data to determine the extent to
which these submissions increased among marinas following
implementation of the program.
4 . Qualitative analysis of information from stakeholder interviews and key
informants on the installation boat pressure washing control
systems at marinas.
5. Qualitative analysis of information from stakeholder interviews.
6. Tabulation and analysis of data from the customer satisfaction survey
and from the information on the installation of boat pressure wash
control technologies.
7. Qualitative analyses of information obtained through the stakeholder
interviews and information on the economic state of the marina sector.
See below
8. Statistical comparisons, using Propensity Score Matching, of the 2001
and 2004 values for best practices-related items on the site visit data.
9. Content analysis of site visit data and PSM results.
10. Qualitative analysis of information from stakeholder interviews and
customer satisfaction survey.
1 1 . Qualitative analysis of information from stakeholder interviews.
12. Qualitative analysis of information from stakeholder interviews and
information on the economic state of the marina.
13. Qualitative analysis of information from stakeholder interviews and
customer satisfaction survey.
18
-------
Refined Evaluation Question Based on Revised Logic Model
3A. To what extent were non-program resources (e.g.,
consultants, non-program guidance documents) useful to
in-scope marinas in improving regulatory compliance or
implementing best environmental practices?
3B. What materials did states develop, and what are states
doing, as part of the New England Marinas Initiative and
their own associated activities related to marinas?
4. What items should be added or dropped from the Region 1
Marina Checklist?
5. To what extent did the New England Marina Initiative lead
to healthier marina communities (e.g., improved
environmental conditions)?
6. What recommendations for improvement to the program
structure can be made?
6A. Are there specific improvements for transferability to
other EPA regions?
7. Among EPA Regions and States is there interest in applying
the Region 1 Marina Initiative approach?
Data Sources
• Stakeholder interviews
• Stakeholder interviews
• Site visit data
• Customer satisfaction survey
• Stakeholder interviews
• Stakeholder interviews (to
determine the availability of
data)
• Site visit data
• Customer satisfaction survey
• Stakeholder interviews
• Stakeholder interviews
• Stakeholder interviews
Analytical Approaches
14. Qualitative analysis of information from stakeholder interviews.
15. Qualitative analysis of information from stakeholder interviews.
16. Qualitative analysis of information from stakeholder interviews,
customer satisfaction survey, and results of PSM analysis.
17. Tabulate and assess data from the checklist.
18. Qualitative analysis of site visit data and information from stakeholder
interviews.
19. Qualitative analysis of information from stakeholder interviews the
customer satisfaction survey.
20. Tabulation and content analysis of site visit data.
2 1 . Qualitative analysis of information from stakeholder interviews.
22. Qualitative analysis of information from stakeholder interviews.
19
-------
CHAPTER 3: RESULTS AND FINDINGS
This section presents the results and findings of the evaluation of the New England Marina
Initiative. The section is organized by the seven key evaluation questions and findings are
summarized below.
QUESTION 1: REGULATORY COMPLIANCE
TO WHAT DEGREE DID REGULATORY COMPLIANCE AT IN-SCOPE MARINAS IMPROVE BETWEEN
2001 AND 2004?
PROPENSITY SCORE MATCHING ANALYSIS OF CHECKLIST DATA
A propensity score matching analysis (see Appendix E for details) was conducted to determine
changes in regulatory compliance amongst marinas between 2001 and 2004. Table B-l in
Appendix B presents a list of the data items from the marinas Checklist that were categorized as
either "regulatory compliance" or "best environmental practices" (BEPs). Most of the items on
the Checklist are simple yes/no questions and thus a percentage of marinas who reported a
specific outcome for 2001 and one for 2004 can be calculated and compared. Table 5-1 presents
the results of the analysis of regulatory compliance, including statistical significance of the
results. All of the indicators presented in Table 5-1 are regulatory requirements.
Requirements were categorized as follows: Hazardous Waste 10, Oil and Fuel 12 (storage 9,
SPCC 3) Hazardous Materials 3, Storm Water 9 (NPDES: permit 1, pressure washing 2, BEPs
6). After the baseline measure, the Region decided not to focus on oil and fuel storage due to
resource issues. Some requirements have thresholds. Facilities that fall under the thresholds are
not required to meet the requirement. Any facility that did not meet a threshold for a regulatory
requirement was not included in the analysis for the requirement.4
For facilities that reported an outcome in both 2001 and in 2004, the analysis identified six
compliance outcomes that changed significantly (Table 5-1 boldface boxes). The analysis
indicated a statistically significant:
• Increase in the percentage of facilities with:
o Spill prevention procedures in place for transferring oil within the facility (48.2 percent
in 2001; 74.5 percent in 2004).
o MSDs used in training employees that handle hazardous chemicals (52.2 percent in
2001; 73.5 percent in 2004).
o NPDES Storm Water Permits., among facilities requiring an NPDES Storm Water Permit
(2.4 percent in 2001; 35.4 percent in 2004).
• Decrease in the percentage of facilities with:
o Spill prevention procedures in place for fuel dispensing (96 percent in 2001; 83.7
percent in 2004).
4 Facilities falling under a regulatory threshold were included in the analysis for best environmental practices in
Section 5.2.1 below.
20
-------
o Fuels, solvents and paints are stored in a protected, secure location, away from drains
(95 percent in 2001; 80 percent in 2004).
o Fuels, solvents and paints plainly labeled (91.7 percent in 2001; 56.9 percent in 2004).
Three compliance outcomes showed statistically significant declines. In each case the 2001
percentage was above 90 percent. In testing statistically significant differences between
percentages, it is more common to find statistically significant differences when one value is
close to either zero (0%) or one (100%).
The analysis identified five statistically insignificant increases in outcomes (positive change)
where the increase was greater than 10 percentage points between 2001 and 2004. The analysis
also identified three statistically insignificant decreases in outcomes (negative change) where the
decrease was greater than 10 percentage points between 2001 and 2004.
Questions have been grouped to communicate the degree to which regulatory compliance
changed between 2001 and 2004 (Table 5-1). Green indicates that greater than 90 percent of
facilities achieved the regulatory outcome, yellow indicates between 50 and 90 percent and red
indicates that less than 50 percent of facilities reported compliance with the desired outcome. In
2001 eight outcomes were above 90% compliance (green). In 2004 four of those remained
above 90% and four dropped below 90%. There were two improvements from less than 50%
compliance to above 50% between 2001 and 2004.
PSM analysis does not calculate the statistical significance of differences when one value is
either zero or one. This occurred for three outcomes in the table. For two outcomes (secondary
containment and leak detection for below ground oil storage), compliance was 100 percent in
both 2001 and then in 2004, albeit among only two facilities in 2001 and three in 2004. For the
third outcome where PSM could not calculate a difference (SPCC plan posted in plain view at oil
storage locations), compliance was zero in 2001 (among 10 facilities) and was 7.7 percent in
2004 (among 13 facilities).
Table 5-1. Propensity Score Matching Analysis Results for Regulatory Outcomes.
Outcome
2001 Data
Marinas
With a
Reported
Outcome
Average
Value for
Outcome
2004 Data
Marinas
With a
Reported
Outcome
Average
Value for
Outcome
Propensity Score
Matching Analysis
Estimated
Difference
between
2004 and
2001
Facilities [a]
Statistically
Significant?
[b]
Hazardous Waste
Are Manifests Documenting Hazardous Waste
Shipments Kept, Going Back At Least 3
Years?
Do employees receive training in proper
handling of -wastes?
Do employees receive training in emergency
procedures?
55
63
63
Are Quantities Of Hazardous Waste \
Generated By The Marina Calculated Each ,,
Month, To Determine What Size Generator
The Marina Is? \
76.40%
81.00%
81.00%
15.90%
56
65
60
61
75.00%
73.80%
81.70%
14.80%
-0.047
-0.154
-0.064
-0.031
No
No
No
No
21
-------
Outcome
Are All Hazardous Wastes Stored In Labeled
Containers?
Are All Hazardous Wastes Stored In A
Dedicated Storage Area?
Are All Hazardous Wastes Stored Indoors Or
Covered?
Are All Hazardous Wastes Stored In An Area
With An Impervious Floor?
Are All Hazardous Wastes Stored With
Storage Area Spill Containment?
Are All Hazardous Wastes Shipped With A
Properly Licensed Transporter?
2001 Data
Marinas
With a
Reported
Outcome
59
60
60
60
60
49
Average
Value for
Outcome
57.60%
81.70%
71.70%
66.70%
45.00%
83.70%
2004 Data
Marinas
With a
Reported
Outcome
59
58
58
57
55
57
Average
Value for
Outcome
59.30%
86.20%
82.80%
75.40%
58.20%
86.00%
Propensity Score
Matching Analysis
Estimated
Difference
between
2004 and
2001
Facilities [a]
0.061
0.081
0.126
0.118
0.16
-0.023
Statistically
Significant?
[b]
No
No
No
No
No
No
Oil and Fuel
Is A Spill Prevention, Control And
Countermeasure Plan (SPCC) On Site? [c]
Is A Spill Prevention, Control And
Countermeasure Plan (SPCC) Signed By A
Registered Professional Engineer? fc]
Is A Spill Prevention, Control And
Countermeasure Plan (SPCC) Posted In
Plain View At Oil Storage Locations? [c]
Does Above Ground Oil Storage (Including
Piping System) Have Secondary
Containment? [cl
Does Above Ground Oil Storage (Including
Piping System) Have Leak Detection? [cl
Does Below Ground Oil Storage (Including
Piping System) Have Secondary
Containment? [c]
Does Below Ground Oil Storage (Including
Piping System) Have Leak Detection? [cl
Are Spill Prevention Procedures In Place For
Receiving Oil From A Supplier? [c]
Are Spill Prevention Procedures In Place For
Transferring Oil Within The Facility? [cl
Are Spill Prevention Procedures In Place For
Waste Oil Disposal? [cl
Does The Facility Have Spill Prevention
Procedures In Place For Fuel Dispensing?
Is Containment In Place In Case Of A Spill?
10
9
10
10
C)
>
2
41
27
37
50
52
20.00%
11.10%
0.00%
90.00%
55.60%
100.00%
100.00%
75.93%
48.21%
67.27%
96.00%
86.50%
14
13
13
15
13
3
3
27
35
40
43
47
35.70%
15.40%
7.70%
86.70%
61.50%
100.00%
100.00%
64.29%
74.47%
75.47%
83.70%
83.00%
0.192
0.083
-0.111
-0.038
0
0
-0.057
0.251
0.049
-0.151
-0.064
No
No
-
No
No
-
-
No
Yes
No
Yes
No
Hazardous Materials
Has The Amount Of Each Hazardous
Material Stored Onsite Been Calculated
(Including Motor Fuel In Above-Ground
Systems Greater Than 1 0,000 Lbs Capacity)
To Determine If Reporting To The Local
Emergency Planning Committee Is
Necessary?
Are Material Safety Data Sheets (MSDSs)
For All Hazardous Chemicals Kept On File?
63
63
23.80%
63.50%
51
66
25.50%
78.80%
-0.015
0.106
No
No
22
-------
Outcome
Are Material Safety Data Sheets (MSDSs)
Used For Training All Employees Handling
Hazardous Chemicals?
2001 Data
Marinas
With a
Reported
Outcome
67
Average
Value for
Outcome
52.20%
2004 Data
Marinas
With a
Reported
Outcome
49
Average
Value for
Outcome
73.50%
Propensity Score
Matching Analysis
Estimated
Difference
between
2004 and
2001
Facilities [a]
0.197
Statistically
Significant?
[b]
Yes
Stormwater
Does The Marina Have A NPDES Storm
Water Permit? [d]
When Pressure Washing Boats Coated With
Ablative Paints, Is Removed Material
Collected To Prevent Releases To Water?
When Pressure Washing Boats Coated With
Ablative Paints, Is Removed Material
Collected To Prevent Contamination Of
Land?
Are Blasting, Other Paint Preparation And
Painting Activities Contained Or Controlled
To Prevent Abrasives, Paint Chips, And
Overspray From Being Released To The
Water?
Are Blasting, Other Paint Preparation And
Painting Activities Contained Or Controlled
To Prevent Abrasives, Paint Chips, And
Overspray From Being Released To Land?
Are Blasting, Other Paint Preparation And
Painting Activities Contained Or Controlled
To Prevent Abrasives, Paint Chips, And
Overspray From Being Released To Protect
Employees?
Are All Engine Fluids Promptly Transferred
From Parts, Drip Pans, Used Filters And
Other Containers To Closed Receptacles For
Disposal Or Recycling?
Are Fuels, Solvents And Paints Stored In A
Protected, Secure Location, Away From
Drains?
Are Fuels, Solvents And Paints Plainly
Labeled?
41
61
61
51
51
52
57
60
60
2.40%
59.02%
55.74%
82.35%
82.35%
90.40%
96.50%
95.00%
91.70%
48
63
62
55
57
51
61
65
58
35.40%
63.49%
53.23%
81.82%
73.68%
92.20%
90.20%
80.00%
56.90%
0.354
-0.019
-0.072
-0.023
-0.114
0
-0.065
-0.125
-0.368
Yes
No
No
No
No
No
No
Yes
Yes
Key: Green = Indicator value above 90 percent; yellow = indicator value between 50 and 90 percent; and red = indicator value
below 50 percent.
[a] The estimated difference reflects the percentage point difference between the treatment group and control group included in
the analysis. Due to the nature of PSM methods, this will not be the same as the difference between the 2001 average and the
2004 average.
[b] This column indicates whether the PSM analysis found the difference between the 2001 and 2004 marinas to be statistically
significant using the 90 percent level of confidence (10 percent significance level).
[c] This outcome is a regulatory requirement if an above ground storage tanks holds 1,320 gallons or greater and/or a below
ground storage tank has a capacity of 42,000 gallons or more; therefore the PSM analysis for this outcome includes only
facilities where site visit data indicates that either of these criteria is met (i.e. if the answer to either of the last two parts of
question 4A of the Checklist is "Yes"). See Appendix B for the complete questions.
[d] This outcome is a regulatory requirement if a NPDES permit is required; therefore the PSM analysis for this outcome includes
only facilities where site visit data indicates that this criterion is met (i.e. if the answer to question 9A of the Checklist is
"Yes"). See Appendix B for the complete question.
23
-------
STAKEHOLDER INTERVIEWS
Analysis of stakeholder interviews provides insights into the impact of the Initiative on
regulatory compliance. Marina owners indicated they have a business interest in maintaining
high water quality (e.g. high water quality leads to increased customer satisfaction among marina
users) but often lack the knowledge about their individual responsibility and how it relates to
maintaining the water quality. Interviewees felt that the Initiative's cooperative approach that
allowed for the sharing of information among stakeholders has increased marinas' knowledge
about their compliance requirements and made a difference in regulatory compliance at marinas.
One interviewee pointed out that although marinas are made aware of regulatory compliance
issues through the Initiative and are taking steps to achieve compliance, many marinas are still
violating regulations simply because of recordkeeping issues.
Interviewees suggested that the state of the marina sector (for example, education, economics,
and demographics) had an influence on regulatory compliance following implementation of the
Initiative. Almost all interviewees said that the biggest external influence on regulatory
compliance was the marinas' ability to meet the costs of compliance. The EPA Region 1 lead for
the Initiative indicated that the cost of pressure wash wastewater control systems is a significant
economic obstacle as these systems can cost up to $200,000. Interviewees think that marinas are
adopting BEPs but that most do not have the financial resources to make bigger facility changes
or significant cash outlays.
In addition, interviewees felt that marinas face the same issues as other small businesses. For
instance, rapidly changing practices and regulations drastically increase costs and make it
prohibitively expensive for marinas to come into compliance in the short term. Interviewees also
said their organizational structure added difficulties to dealing with the environmental issues that
marinas face (e.g. high employee turnover rates and complex regulatory guidance). However,
regulations and areas of profitability (i.e. maintenance and repair, not storage), were also cited as
responsible for limiting the number and frequency of do-it-yourself activities for boaters. With
more of the environmentally-sensitive activities carried out by employees as opposed to boaters,
marinas are able to better monitor their actions and compliance.
Since the start of the Initiative, Region 1 became particularly interested in increasing the number
of marinas with MSDSs and NPDESs, which are important in at least two compliance areas: (1)
making hazardous waste determinations and (2) identifying hazardous material under the
Emergency Planning and Community Right to Know Act. NPDES storm water permits are
important for marinas since almost all marinas will be affected by storm water issues. To
achieve these and other priority regulatory outcomes identified by stakeholders, the EPA Region
1 lead suggested that local governments can influence marinas by setting requirements and
encouraging the use of best practices. Local agencies have considerable influence and a clear
incentive to provide additional sources of information to assist marinas in achieving compliance.
They often have regulatory requirements that are more stringent than or differ from EPA's.
Increased uses of MSDSs and NPDES storm water permits were among the three regulatory
compliance outcomes that were estimated to have a significant increase between 2001 and 2004.
Region 1 considers both of these to be significant improvements for a number of reasons.
MSDSs are important compliance areas because they aid in making hazardous waste
24
-------
determinations and in identifying hazardous material under the Emergency Planning and
Community Right to Know Act. Compliance with the Storm Water General Permit is a
fundamental environmental performance factor as storm water is an issue for most marinas.
SPCC plans were of particular interest to Region 1 because marinas may store large quantities of
oil and fuel in the immediate proximity of critical natural resources or densely populated areas.
Among facilities required to have a SPCC, the percentage in compliance increased from 20
percent to 36 percent, a 16 percentage point increase in compliance. However, the increased
compliance rate was not statistically significant in the PSM analysis (there were too few facilities
in each year to allow for finding a significant difference) and rate in 2004 was still below 50
percent compliance. Nevertheless, it should also be noted that seven facilities in the data had
SPCC in place even though it appears they were not required of those facilities.
ADDITIONAL DATA AND INFORMATION
• Self-Disclosures and EPCRA Tier II Filings Data
EPA Region 1 provided data on the number of self-disclosures and the number of EPCRA Tier II
filings dating back to the 1990s.5 The tabulations of these data for different time periods are
shown in Table 5-2. Since the New England Marinas program ran from 2001 to 2004, we
included three periods in the tabulation: before 2001, 2001-2004, and after 2004.
Table 5-2. Self-Disclosed Violations and EPCRA Tier II Filings Among Marinas, Before 2001,
2001-2004, and After 2004.
Type of Filing
Marinas Self Disclosures
All Self Disclosures
Marinas as Percent of All Self
Disclosures
Date Of Filing
Before
2001
0
24
0.0%
2001-
2004
16
240
6.7%
After
2004
2
127
1.6%
Marina EPCRA Tier II Filings
2
89
237
For the self-disclosures of violations6 prior to 2001, there were no self-disclosures reported
among marinas. Between 2001 and 2004, 16 self-disclosures were reported. The 16 reported
disclosures represented 6.7 percent of all reported disclosures. Following 2004 there were two
self-disclosures at marinas. Prior to 2001 two marinas completed EPCRA Tier II filings.
Between 2001 and 2004, 89 facilities filed, and between 2004-2007, filings increased to 237.
5 For the self-disclosure data, the earliest date in the data provided was June of 1999 and the most recent date was in
2006. For the Tier II filings, the earliest date in the data was for 1995 and the most recent was in 2007.
6 Each self-disclosure could involve one or more violations.
25
-------
INFORMATION ON THE INSTALLATION OF BOAT PRESSURE WASHING CONTROL SYSTEMS AT
MARINAS
EPA Region 1 observed three pressure wash wastewater (PWW) treatment installations during
compliance assistance visits after 2004. These systems were not observed at marinas during 2001
or 2004 site visits. A representative of the MA Marine Trades Association indicated that in early
2008 the Trade Association coordinated the bulk purchase of PWW systems for a number of
marinas in the association. The representative indicated that the marinas with which he is
familiar had a PWW control system installed or on order by 2008.
Four PWW treatment suppliers were contacted about the demand for their products pre- and
post-2004.
• Company A installed 30 systems as of February 2009. The company has seen a noticeable
increase in demand for its products among New England marinas since 2004; however, the
company did not initiate sales of pressure wash products to marinas until late 2004 or early
2005. Company A speculates that there was no demand for these products before 2004
because the issue had not been widely understood as a priority by marinas, the public, or the
regulatory community. Since 2004 demand increased and triggered Company A to develop
their technology for the marina industry.
• Company B has seen no increase in sales or inquiries about products from marinas in New
England for PWW recycle systems since 2004.
• Company C reported completing four installations since 2004.
• Company D reported recently installing two systems in Connecticut.
QUESTION 2: BEST ENVIRONMENTAL PRACTICES (BEPs)
TO WHAT DEGREE DID IN-SCOPE MARINAS INCREASE THEIR USE OF BEPS BETWEEN 2001 AND
2004?
PROPENSITY SCORE MATCHING ANALYSIS OF CHECKLIST DATA
A propensity score matching analysis was conducted in parallel with the analysis of compliance
outcomes discussed in the previous section to determine to what degree in-scope marinas
increased their use of BEPs between 2001 and 2004. Table 5-3 presents the results of the
analysis for BEP outcomes, including the statistical significance of the results. Two types of
indicators are presented in Table 5-3 to reflect best environmental practices: (1) Indicators that
are not regulatory requirements, and (2) Indicators that are regulatory requirements, but that have
a threshold. In this case facilities that fall under the threshold are not required to meet the
requirement. Thus, the indicator then represents a BEP for facilities that fall under the regulatory
threshold.
Five BEP outcomes had a statistically significant difference (Table 5-3 boldface boxes) between
facilities that reported an outcome in 2001 compared with facilities that reported an outcome in
2004. The analysis indicated a statistically significant:
26
-------
Increase in the percentage of facilities:
o
o
With a Spill Prevention, Control and Countermeasure Plan (SPCC) on site among
facilities where it is not required, (from 2.8 percent in 2001 to 24.1 percent in 2004).
With an assessment of the effectiveness of the methods they're using to prevent oil and
fuel releases. (from an average value of 2.87 in 2001 to an average value of 3.18 in 2004,
both based on a four-point scale where 4 was "Excellent" and 1 was "Poor").
That made structural changes to riprap (21.9 percent in 2001; 44.8 percent in 2004).
That made structural changes to placement of filters and drains (14.1 percent in 2001;
42 percent in 2004).
• Decrease in the percentage of facilities that:
o Switched to biodegradable cleaners (88.7 percent in 2001; 72.7 percent in 2004).
The analysis identified 2 statistically insignificant increases in outcomes (positive change) where
the increase was greater than 10 percentage points between 2001 and 2004. The analysis also
identified two statistically insignificant decreases in outcomes (negative change) where the
decrease was greater than 10 percentage points between 2001 and 2004.
Similar to the previous section, questions have been grouped to highlight the degree to which
marinas in New England changed their use of BEPs between 2001 and 2004 (Table 5-3). Green
indicates that greater than 90 percent of facilities implemented the BEP, yellow indicates
between 50 and 90 percent and red means that less than 50 percent of facilities reported adoption
of the BEP. In 2001, four outcomes were above 90 percent compliance (green). In 2004, three of
those remained above 90 percent and one dropped below 90 percent. Of the ten BEPs used by
fewer than 50 percent of facilities in 2001, one BEP was used by more than 50 percent of
marinas by 2004. In 2001, ten BEPs were applied by 50 to 90 percent of facilities and in 2004,
one had improved to above 90 percent adoption and two BEPs dropped to below 50 percent of
marinas
Table 5-3. Propensity Score Matching Analysis Results for Best Environmental Practices Outcomes.
Outcome
2001 Data
Marinas
With a
Reported
Outcome
Average
Value for
Outcome
2004 Data
Marinas
With a
Reported
Outcome
Average
Value for
Outcome
Propensity Score
Matching Analysis
Estimated
Difference
between
2004 and
2001
Facilities fa]
Statistically
Significant?
[b]
Oil and Fuel
Is A Spill Prevention, Control And
Countermeasure Plan (SPCC) On Site? |c]
Is A Spill Prevention, Control And
Countermeasure Plan (SPCC) Signed By A
Registered Professional Engineer? [cl
Is A Spill Prevention, Control And
Countermeasure Plan (SPCC) Posted In Plain
View At Oil Storage Locations? [c]
Does Above Ground Oil Storage (Incl. Piping
System) Have Secondary Containment? [cl
36
36
36
10
2.80%
2.80%
2.80%
50.00%
29
17
16
18
24.10%
5.90%
12.50%
38.90%
0.267
0.062
0.144
-0.017
Yes
No
No
No
27
-------
Outcome
Does Above Ground Oil Storage (Incl. Piping
System) Have Leak Detection? [c]
Does Below Ground Oil Storage (Incl. Piping
System) Have Secondary Containment? [cl
Does Below Ground Oil Storage (Incl. Piping
System) Have Leak Detection? [cl
Does The Facility Have More than One Spill
Prevention Procedure In Place For Fuel
Dispensing? [d]
How Would You Evaluate The Effectiveness Of
The Methods You Are Using To Prevent
Releases? [e]
Does The Marina Have A NPDES Storm Water
Permit? [f]
Has The Marina Made Structural Changes To
Minimize Surface Water Runoff?
Has The Marina Made Structural Changes To
Berming?
Has The Marina Made Structural Changes To
Vegetation?
Has The Marina Made Structural Changes To
Riprap?
Has The Marina Made Structural Changes To
Drains?
Has The Marina Made Structural Changes To
Placement Of Filters In Drains?
2001 Data
Marinas
With a
Reported
Outcome
11
28
28
42
53
Average
Value for
Outcome
45.50%
96.40%
96.40%
87.50%
2.8679
2004 Data
Marinas
With a
Reported
Outcome
19
21
22
34
49
Average
Value for
Outcome
Propensity Score
Matching Analysis
Estimated
Difference
between
2004 and
2001
Facilities [a]
36.80% 0.009
85.70%
90.90%
94.40%
3.1837
-0.062
-0.037
0.094
0.288
Statistically
Significant?
[b]
No
No
No
No
Yes
Stormwater
14
65
65
64
64
64
64
100.00%
69.20%
55.40%
51.60%
21.90%
25.00%
14.10%
8
62
60
62
58
55
50
100.00%
71.00%
40.00%
51.60%
44.80%
29.10%
42.00%
0
0.047
-0.167
-0.012
0.212
-0.017
0.24
-
No
No
No
Yes
No
Yes
Other Best Environmental Management Practices
Has The Facility Switched To Alternative
Materials Or Products To Reduce Toxicity Or
Other Hazards To Health, Safety, Or The
Environment?
Has The Facility Switched To Safer Paint
Stripping?
Has The Facility Switched To Safer Painting?
Has The Facility Switched To Safer MSD Odor
Chemicals?
Has The Facility Switched To Dust Collection?
Has The Facility Switched To Phosphate-Free
Cleaners?
Has The Facility Switched To Biodegradable
Cleaners?
Has The Facility Switched To Safer Antifreeze?
Does The Facility Have A Sewage Pumpout
System?
66
41
52
57
50
53
62
60
65
89.40%
46.30%
73.10%
26.30%
66.00%
37.70%
88.70%
95.00%
69.20%
57
40
51
44
51
52
55
53
69
87.70%
52.50%
62.70%
31.80%
78.40%
48.10%
72.70%
96.20%
62.30%
-0.017
0.071
-0.116
0.034
0.141
0.098
-0.174
0.005
-0.039
No
No
No
No
No
No
Yes
No
No
Key: Green = Indicator value above 90 percent; yellow = indicator value between 50 and 90 percent; and red = indicator value
below 50 percent.
[a] The estimated difference reflects the percentage point difference between the treatment group and control group included in
the analysis. Due to the nature of PSM methods, this will not be the same as the difference between the 2001 average and the
2004 average.
28
-------
[b] This column indicates whether the PSM analysis found the difference between the 2001 and 2004 marinas to be statistically
significant using the 90 percent level of confidence (10 percent significance level).
[c] This outcome is only a regulatory requirement if an above ground storage tanks holds 1,320 gallons or greater and/or a below
ground storage tank has a capacity of 42,000 gallons or more; otherwise, it is likely a best environmental practice. Therefore,
the PSM analysis for this outcome includes only facilities where site visit data indicates no tanks above these limits are on
the premises (i.e. if the answer to either of the last two parts of question 4A of the Checklist is "No"). See Appendix B for the
complete questions.
[d] Facilities are required to have at least one spill prevention procedure in place for dispensing fuel. For facilities that are in
compliance with this regulatory requirement, this outcome is determined as a best practice based on whether the facility has
at least two of the following procedures in place: an overfill alarm, automatic shutoff, fuel collars, employee monitoring of
fueling, or other procedures.
[e] This outcome was based on a four-point scale (where 4 was "Excellent" and 1 was "Poor"), as opposed to the rest of the
outcomes in the table that all had Yes/No responses. Thus, the "Average Value For Outcome" columns (for both the 2001
and 2004 data) for this outcome contain the actual average rating for the group (as opposed to the percentage of "Yes"
answers that is indicated for the other outcomes).
[f] This outcome is only a regulatory requirement if a NPDES permit is required; otherwise, it is simply a best environmental
practice. Therefore, the PSM analysis for this outcome includes only facilities where site visit data indicates no permit is
required (i.e. if the answer to question 9A of the Checklist is "No"). See Appendix B for the complete question.
STAKEHOLDER INTERVIEWS
Stakeholders were asked if they thought the Initiative's materials influenced marinas' adoption
of BEPs. Interviewees viewed the Checklist as a high-level document that prompts owners to
think about BEPs and leads to greater regulatory awareness but does not go into enough depth
for marina owners to understand what it means to be in compliance with regulations.
Interviewees felt that the Environmental Management Plan (EMP) workbook is put together
well, but they were unsure if marina owners use it; many thought that distributing it by mail and
through the web site probably would not result in owners filling the booklet out. While
interviewees thought the workshops and training were generally informative, some found them
overly technical, hindering their effectiveness to influence adoption of BEPs.
Interviewees were also asked about external factors affecting the adoption of BEPs. They
suggested that employee training as an issue - marinas do not generally have the capacity to offer
formal training, and the seasonal nature of the business of marinas results in a high employee
turnover rate. Generally this leads to a lower overall awareness of BEPs amongst employees,
which affects marinas' and boaters' capacity to use BEPs. Interviewees also said marinas that
conduct technical work and building services currently have a shortage of skilled labor and that
retaining skilled employees in general is difficult in the marinas industry. Again, issues with
employee retention and knowledge of the general operations of marinas have an impact on the
awareness of environmental issues and best practices at marinas.
CUSTOMER SATISFACTION SURVEY DATA
The customer satisfaction survey asked respondents to rate their satisfaction with different
Initiative materials with respect to selected BEPs. Table 5-4 presents the number of respondents
that were "Very Satisfied" or "Satisfied" with Initiative materials' ability to address different
BEPs. Almost all respondents were very satisfied or satisfied with all Initiative materials with
respect to each of the BEPs. For workshops and training, all respondents indicated that they were
satisfied with the coverage of the use of closed receptacles for disposal or recycling and the
installation of sewage pump-out systems. For the website, all respondents were satisfied with
coverage of spill prevention procedures and use of closed receptacles. The satisfaction scores for
29
-------
the guidance materials were lower overall than for workshops and training and the website. In
open ended responses, stakeholders indicated that EPA's workshops were too technical, that
workshops should be no more than 1.5 hours driving distance for participants and that better
publicity of the website and notification of updates and new materials will help marinas take
advantage of program materials.
Table 5-4. Number of Respondents That Were "Very Satisfied" or "Satisfied" With Initiative Materials'
Ability to Address Different Best Environmental Practices [a].
Best Environmental Practice
Spill prevention procedures or devices (e.g. overfill alarms)
Use of closed receptacles for disposal or recycling
Clear labeling and secure storage of fluids (e.g. fuels and
paints)
Structural changes to minimize surface water runoff (e.g.
berming, filters and riprap)
Use of alternative practices or materials to reduce toxicity or
other hazards
Installation of a sewage pump-out system (onshore or on-boat)
Initiative Materials
Workshops/
Training
5 of 7
8 of 8
8 of 9
6 of 11
7 of 10
7 of 7
Web Site
8 of 8
8 of 8
8 of 9
7 of 8
8 of 9
7 of 7
Guidance
Materials
9 of 10
9 of 10
8 of 10
7 of 10
7 of 10
9 of 10
[a] Number of respondents (of total) that indicated that they were either "very satisfied" or "satisfied." Respondents were asked if
they had used the material in questions 1, 7 and 14 of the Customer Satisfaction Survey; the satisfaction score data refer to
questions 4,11, and 17 (see Appendix B for complete questions).
QUESTION 3: USEFULNESS OF THE ACTIVITIES AND MATERIALS
TO WHAT DEGREE WERE THE NEW ENGLAND MARINA INITIATIVE ACTIVITIES AND/OR
MATERIALS USEFUL IN IMPROVING REGULATORY COMPLIANCE OR IMPLEMENTING BEPS?
The customer satisfaction survey and the stakeholder interviews were designed to determine the
usefulness of program resources to improving compliance and use of BEPs. That information has
been documented in the findings related to questions 1 and 2.
WERE NON-PROGRAM RESOURCES USEFUL?
As for the usefulness of non-program resources, restrictions to the questions admissible under the
customer service ICR limited the collection of information on non-EPA resources. The
stakeholder interviews provided little additional information on non-program resources that were
useful to marinas.
WHAT MATERIALS DID STATES DEVELOP AND WHAT ARE STATES DOING AS PART OF THE
INITIATIVE?
Information collected during stakeholder interviews and a review of state websites revealed that
states conducted activities and developed materials contributing to the New England Marinas
Initiative as well as independent activities, which could be considered non-program resources.
State contacts did not view their activities and materials as part of the Initiative and indicated that
they had either developed their own prior to the Initiative or developed them independent of the
Initiative. Region 1 noted that EPA had assisted the states in developing the materials listed in
30
-------
Table 5-5 and that the states may have been unaware that Region 1 was providing that assistance
as part of the Initiative. The New England Marinas Website's links to state programs were used
to find materials developed in the 2001-2005 time frame.7 Table 5-5 lists some of the more
prominent materials that may have been influenced by the Initiative.
Table 5-5. Material Potentially Developed by State Using New England Marinas Initiative Assistance
State/Agency
Connecticut Department of
Environmental Protection
Maine Department of Environmental
Protection
Massachusetts Office of Coastal Zone
Management
Massachusetts Department of Marine
Fisheries
New Hampshire Department of
Environmental Services
Rhode Island
Vermont
Materials
• CT Clean Marina Award Checklist
• CT Clean Marina Compliance Checklist
• CT Clean Marina Guidebook
• Clean Boater's Program Tipsheets
• Brightwork: Best Environmental Practices Manual for Maine's
Boatyards and Marinas
• Maine Pumpout Station Guide
• Massachusetts Clean Marina Guide
• A Guide to Selecting Pressure Washing Management Practices and
Technologies: Supplement to the Massachusetts Clean Marina
Guide
• Boater Fact Sheets
• MA Clean Vessel Act Program Brochure
• Best environmental practices for New Hampshire Marinas,
Guidelines for Environmentally Proactive Marinas
• Best environmental practices for New Hampshire Marinas, Pocket
Version
• Management of Engine Test Tank Wastewater for Marinas, Fact
Sheet
• A Boaters Guide to Sewage Pumpout Discharge Regulations and
Pumpout Stations
• "RI's Clean Marina Guidebook and other materials are under RI
CRMC at: http://www.crmc.ri.sov/marinas.html "
• Marinas Web
QUESTION 4: THE CHECKLIST
WHAT ITEMS SHOULD BE ADDED OR DROPPED FROM THE REGION 1 MARINA CHECKLIST?
STAKEHOLDER INTERVIEWS
To assess the need for changes in the Checklist, stakeholders were asked for overall thoughts
about the Checklist as well as which items should be added or dropped. Most interviewees felt
the Checklist is a useful tool to raise awareness among marina operators of regulatory issues and
BEPs. However, many interviewees indicated that the Checklist overall was not appropriate for
their state and that state-specific checklists might work better because of the differences in
regulations between states. Interviewees also suggested that additional information on any
applicable exemptions be included to avoid confusion. For instance, the hazardous waste section
(Question 3) should include an option asking for the marina's generator status because marinas
7 EPA Region 1 indicated that materials developed through 2005 may have been influenced by the Initiative's
activities.
-------
may be in a hazardous waste conditionally exempt status which is relatively less stringent. EPA
Region 1 noted, however, that the Checklist is not intended as a comprehensive evaluation tool
but an indicator tool that the Region can use in assessing a facility.
The EPA Region 1 lead suggested adding questions to the Checklist to determine whether
marinas had come into compliance with EPRCA regulations. This would include if a marina has
sulfuric acid in excess of one thousand pounds and if a marina has chlorine in excess of one
hundred pounds on the premises, as well as a question about whether or not hazardous waste
determinations have been completed (and a follow-up question concerning whether the
determinations have been documented).
Interviewees suggested that data that are already available to EPA on the web should be removed
from the Checklist (e.g. Question 9, concerning NPDES permits) to make the document shorter
and avoid confusion. It was also noted that parts of the section on storm water are required by all
marinas, so the questions in this section should ask whether the marina is in compliance, not
whether the marina is subject to compliance. In addition, one interview pointed out that Question
4B (SPCC) on the checklist needs to be updated because marinas can now certify themselves.
Interviewees thought that because pressure washing has been a focus for EPA in recent years, it
is almost a non-issue now and could be removed from the Checklist. An interviewee was
concerned that Question 8 (MSDS) refers to an Occupational Safety and Health Administration
(OSHA) rule, and if so, suggested it could be removed from the Checklist as well. Other
interviewees thought the Checklist should include questions on bilge water, boat washing, and
wetlands. In addition, interviewees said questions on pollution prevention, parts washing,
engines, and test tank water discharge would be helpful.
In order to make the Checklist more useful in general to marina operators, interviewees suggest
making the Checklist more explanatory because the typical marina owner is probably unfamiliar
with a lot of the details in the Checklist. One interviewee suggested adding a footnote to link
directly to the EPA website and materials on the subject, so users could dynamically see the rule
and documentation (information on the actual rule, how to comply with it, and best practices
related to the issue).
CUSTOMER SATISFACTION SURVEY
One of the goals of the Checklist is to increase awareness among marinas of BEPs and regulatory
requirements. The customer satisfaction survey did not ask respondents about the Checklist
specifically, but the survey did ask respondents to indicate how many years of experience they
have had with a number of BEPs. Table 5-6 presents the percent of respondents with less than
three years of experience with a number of best environmental practices. This suggests more
items related to the best practices with the highest percent of respondents with less than 3 years
of experience (i.e. the best practices that marina owners/employees seem to have the least
amount of experience with) could increase awareness among users. Respondents have the least
amount of experience with implementing structural changes, alternative practices to reduce
toxicity, and installation of sewage pump-out systems
32
-------
Table 5-6. Percent of Respondents with Less than Three Years of Experience for Selected
Best Environmental Practices.
Best Environmental Practice
Structural changes to minimize surface water runoff (e.g. berming,
filters and riprap)
Use of alternative practices or materials to reduce toxicity or other
hazards
Installation of a sewage pump-out system (onshore or on-boat)
Spill prevention procedures or devices (e.g. overfill alarms)
Use of closed receptacles for disposal or recycling
Clear labeling and secure storage of fluids (e.g. fuels and paints)
Percent of Respondents
with Less than 3 Years
of Experience
27.3%
27.3%
27.3%
22.7%
13.6%
13.6%
QUESTION 5: ENVIRONMENTAL HEALTH OF MARINA COMMUNITIES
TO WHAT EXTENT DID THE NEW ENGLAND MARINA INITIATIVE LEAD TO HEALTHIER MARINA
COMMUNITIES?
To reliably assess water quality impacts from marinas, data should be collected before (e.g., in
2000) and after the Initiative (e.g., 2004) from fixed monitoring stations in the immediate vicinity
o/'marinas. Of stakeholders interviewed none could identify data sources on water quality useful
to assessing water quality impacts of marinas. Some interviewees suggested possible sources of
data but indicated that any data found were likely to be unreliable. Three potential sources of
water quality data were reviewed: 1) U.S. Geological Society (USGS) Water Data for the
Nation8, 2) EPA National Water Quality Inventory Report to Congress (305(b) reports)9, and 3)
EPA's STORE! (short for STOrage and RETrieval) data10. Each of the three of the potential
sources on water quality data was comprehensive in nature but each source had limitations for
use in the evaluation.
o U.S. Geological Society (USGS) Water Data for the Nation11—these data provide
information on flows and quality of waterways in the United States. Data are
collected at different sites across the country. However, effective use of these data
would require obtaining exact latitude and longitudes for marinas and matching those
to the lat-longs of the data collection sites, a process beyond the scope of this
analysis. It is unlikely that this process would reveal appropriate sampling sites near
marinas. Additionally, the data collection sites in the USGS data do not necessarily
provide the same data over time and some represent only one-time data collections.
o EPA National Water Quality Inventory Report to Congress (305(b) reports) u—these
data are compiled to track general water quality conditions in the United States. The
data track supported use by type of water body (e.g., coastline, bay, etc.). There are
some data at the watershed level, but for the most part these data summarize
http://waterdata.usgs.gov/nwis
9 http://www.epa.gov/305b/
10 http://www.epa.gov/storet/index.html
11 http://waterdata.usgs.gov/nwis
12 http://www.epa.gov/305b/
33
-------
conditions at the state level, especially in older reports (pre-2002). Additionally, the
data rely on assessments that were performed. Both Maine and Massachusetts have
significant amounts of coastlines that were not assessed.
o EPA's STORE! (short for STOrage and RETrieval) data13—the STORE! data
provide a repository of water quality measurement data from different sites across the
country. However, as with the USGS data, use of these data for this analysis would
require obtaining exact lat-longs for marinas and matching those to the lat-longs of
the data collection sites. Furthermore, the data contained in STORET may not contain
enough data points over time to support the analysis needed for this project. Although
each is comprehensive in nature, each source had limitations for use in the evaluation.
QUESTION 6: IMPROVEMENTS TO THE INITIATIVE
WHAT RECOMMENDATIONS FOR IMPROVEMENT TO THE PROGRAM STRUCTURE CAN BE MADE?
STAKEHOLDER INTERVIEWS
Stakeholders identified specific improvements that can be made to the program. They made suggestions
regarding: (1) improvements to the checklist; (2) improvements to the workshops and seminars; (3)
clarification of state-specific requirements and regulations; (4) providing more on-site demonstrations of
BEPs and (5) providing more outreach efforts.
Interviewees said that though the BMP workbook is well structured few marinas use it. They suggested
that EPA hold seminars that teach owners how to properly use the workbook and also make it available in
an electronic—and more interactive—form. Some interviewees said that the workshops and seminars
currently offered are overly technical and that regulation requirements should be translated into plain
English. A trade association representative could attend workshops as a facilitator since they may have a
better understanding of regulations in the real-world. Interviewees felt that a trade association
representative would have a better understanding of the issues marinas face and that the audience would
be more comfortable with this format than an EPA-only workshop or seminar.
Almost all interviewees highlighted the state-to-state differences in requirements as a problem. Additional
language could be added ("The federal rule says...") to Initiative materials to minimize confusion about
state vs. federal regulations. Interviewees noted that more on-site presence by EPA staff would be useful.
They suggested that EPA bring someone along who is qualified to give state-specific compliance
information during outreach efforts. Region 1 suggested that demonstration of BEPs and new
technologies on-site is an effective way to provide training and suggested scaling up this type of
assistance. Also, since some marina owners may not be comfortable with visits from EPA representatives,
the Region could send a regulation expert not affiliated with the agency to conduct site visits and provide
technical assistance.
13 http://www.epa.gov/storet/index.html
34
-------
QUESTION 7: TRANSFERRING THE INITIATIVE
AMONG EPA REGIONS AND STATES is THERE INTEREST IN APPLYING THE REGION 1 MARINA
INITIATIVE APPROACH?
The data source identified to answer this question, stakeholder interviews, provided little
information regarding specific states or regions with an interest in the applying the Initiative.
EPA Region 1 considered itself, and stakeholders viewed Region 1, as a partner with the states
and other stakeholders. Stakeholders, however, were unaware that Region 1 had named the
Initiative The New England Marinas Initiative. All of the interviewees were familiar with the
work of Larry Wells, the EPA Region 1 lead for the New England Marinas Initiative. Each
respondent was familiar and had experience with many of the components of the Initiative as
they are represented in the Initiative's logic model. The various components of the initiative
were generally viewed favorably and no respondent suggested terminating any of the
components of the Initiative with which they were familiar.
It was not the intent of the Region to formally promote the effort as a regional initiative. The
existence of a formal Initiative was unknown to respondents; most did not have a definitive view
on whether the Initiative should be transferred to another region or State. Specific considerations
for transferability of the New England Marinas Initiative will be considered further in Chapters 4
(Discussion) and 5 (Recommendations).
35
-------
CHAPTER 4: DISCUSSION
Given growing emphasis on the value of networks and collaborative partnerships in achieving
measurable environmental goals, the EPA Region I New England Marinas Initiative is a valuable
model that demonstrates lessons about developing stakeholder networks, improving regulatory
compliance and adoption of BEPs, articulating and clarifying program theory, and measuring and
evaluating performance.
Overall, EPA, states, trade associations and marina owners, and other stakeholders have
established a strong foundation for the maintenance and growth of the New England Marinas
Initiative. That the Initiative took action on the implementation stage as described in the
program logic model (See Figure 2-1) is evidence of the Initiative's success in creating
partnerships and a network of individuals and organizations with an interest in regulatory
compliance and adoption of BEPs at marinas. According to the Initiative leaders and as
documented in the logic model, creating partnerships and enhancing the assistance provider
network were critical to the success of the Initiative. The implementation stage, including the
development and delivery of education and technical assistance, would not have been possible
without thorough execution of the project planning and needs assessment stage.
In the implementation stage, statistically valid samples of the population of marinas in New
England and the use of a measurement instrument (the Checklist) to collect pre- and post-
implementation data enabled a rigorous quasi-experimental study of the validity of multiple
aspects of the program's theory. The statistical analysis, using a method called propensity score
matching (PSM), of pre- and post-implementation data detected few improvements in regulatory
compliance and adoption of BEPs among marinas in New England between 2001 and 2004.
Though the PSM method makes a definitive statement that a set of the Initiative's desired
outcomes were not achieved between 2001 and 2004, the analysis provides no insights as to why
this is the case, whether outcomes may be achieved in the future, or what factors influenced
changes in compliance and use of BEPs. Results of the PSM analysis (both positive and
negative) and analysis of qualitative data indicate compliance improvements in several areas and
encourage the use of all of the available data and information to more completely answer such
questions as "Why are we where we are?" and "How can we improve?" For the Marinas
Initiative, there is much to be learned by considering these two evaluative questions in the
context of the Initiative's program specific evaluation questions.
Theory of Change
The theory of change - the steps necessary to bring about long-term goals - illustrated in the
implementation stage of the Initiative's logic model (Figure 2-1) is not as clear as the much more
detailed planning and needs assessment stage. This suggests that the implementation stage has
not been as thoroughly considered. For example, exactly which educational activity, product, or
service (e.g. workshop or EMP workbook or website) will lead to achieving which and what kind
of change in understanding (e.g. understanding of environmental significance, regulatory
compliance, or how-to skills)? Without articulating these causal linkages more explicitly, there
is an increased likelihood that the Initiative operates on inaccurate and/or incomplete
assumptions about how the program will achieve its goals, which, in turn, could compromise the
Initiative's capacity to measure its effectiveness. For instance, the design of the Initiative
36
-------
assumes that education and technical assistance lead to increased knowledge which leads to
actions taken by marina owners which results in better compliance at marinas and more frequent
use of BEPs. Although these are two major assumptions on which the success of the Initiative is
based, there is little evidence of their validity.
First, there is limited measurement of learning. If the Initiative is based on the assumption that
learning leads to change, establishing the effectiveness of education strategies is critical.
Stakeholders think the EMP booklet is designed well but were unsure if marina owners use it;
they agree that workshops and training are informative but also agree that they are overly
technical and thus unclear and confusing. The Initiative resulted in a plethora of materials and
activities aimed improving compliance and use of BEPs and the Initiative recognizes the
importance of their usefulness to marina owners as evidenced by an evaluation question about
that issue. Usefulness was not clearly defined by the Initiative but its definition could be
clarified through further development of the logic model. For instance, is a particular activity or
education tool intended to be useful in improving understanding or a skill and which one,
changing behavior, or building networks and partnerships? The materials could be useful in
achieving those things but other factors prevent use (access, cost, time, and relevance) or marina
owners could have viewed the materials as useful but applied the tool outside the measurement
window (2001-2004). A starting place for hints at changes in awareness and learning amongst
marina owners could be found by looking for changes in "don't know " responses on the
checklist between 2001 and 2004. Significant changes in certain outcomes could be tracked back
to the part of the activity or materials addressing that issue to search for insights more effective
educational tools or strategies. For instance, was an illustration, table or anecdote used more or
less often when a significant change in "don't know " is detected in the analysis?
Second, the initiative did not necessarily measure a change in behavior or the actions of marina
owners but a change in condition which would be caused by a change in behavior. Measurement
documented change in the status of the marina and did not document the intent, plans or ongoing
actions of marina owners with respect to compliance and BEPs. This suggests that the Initiative
measured the change in condition (a longer-term outcome) and did not measure shorter term
outcomes such as change in awareness, understanding, skills, planning, or action - all of which
are integral to the assumptions on which the Initiative is based.
The PSM analysis identified relatively few positive changes in desired outcomes between 2001
and 2004; however, a timeframe is not specified in the logic model and outcomes may be
realized outside the 2001-2004 measurement window. For example, if assistance to marinas was
spread over 3 years, marinas may: 1) access assistance in the third year, 2) apply what was
gained from assistance after the 3rd year, 3) have access to assistance only in the 3rd year, or 4)
education and assistance materials and activities improved during the three years and marina
owners did not have access to the best quality assistance until the third year. In each of these
cases, marina owners would have no opportunity or only a small window of time to act on what
they learned from education and outreach before the second measurement in 2004.
Both self-disclosures of violations and EPCRA Tier II filings provide some evidence that
significant change may have gained momentum after 2004 (Table 5-2). First, the drop in the
number of self disclosures at marinas after the 2004 measurement could reflect increased
37
-------
compliance at marinas (fewer violations to disclose) or perhaps less motivation to disclose the
violations (due to less Initiative-related activities). If measurement continued at regular
intervals, the former (fewer violations) could be ruled out, and if either is true, the Initiative may
have been responsible for the effect. Second, trends in completed EPCRA Tier II filings
between 2001 and 2004 (89) and between 2004 and 2007 (237) is strong evidence that the
Initiative may be influencing marinas to make the appropriate filings. These two examples
demonstrate the potential benefits of going beyond, if possible, before- after-assessment of
marinas' status to get a clearer picture of trends in intent, behavior, and condition.
A few key pieces of information could be useful to understand and determine appropriate
measurement intervals. First, tracking the effectiveness of the education materials and
approaches will help to hone the use of education and outreach strategies (i.e. do recipients of
education activities and materials gain the knowledge and skills necessary to achieve the
Initiative's compliance and BEP goals?). Second, general information on the time and money
required for a marina owner to act on specific compliance and best practice indicators of interest
will add insight as to what is feasible to change in a given timeframe. Understanding what an
owner must invest to come into compliance or adopt a BEP may help to craft and prioritize the
use of indicators that will detect change within an appropriate measurement window. Third, the
content and timing of deployment of measurement instruments can be informed by a general
understanding of marina owners' intentions to act. That is, given a set of compliance issues or
BEPs, how does the marina owner prioritize when and what to do to improve compliance and
use of BEPs?
Prioritization
Some stakeholders suggested dropping a few indicators from the checklist and others suggested
that the checklist should include more indicators of environmental compliance or BEPs. These
suggestions provide evidence of important progress toward a willingness to discuss and general
awareness of challenging environmental issues - critical objectives of the Initiative. The
Initiative can use this evidence as a solid stepping stone to improve communication amongst
stakeholders and direct future education and technical assistance.
Besides the benefits of increasing awareness of environmental compliance and BEPs among
stakeholders, suggestions that items should be added or dropped from the checklist hint at the
perception amongst stakeholders that, for a wide variety of reasons, some issues are more
important than others. The creation of the checklist itself as well as categories within the
checklist illustrates the Region's process of prioritization. However, specific items or groups of
items were not weighted more or less important within the compliance areas. Prioritization
within compliance areas presents an opportunity to improve the focus of education and technical
assistance as well as refine the development, application and use of measurement instruments.
The statistically valid sampling approach applied for the collection of site data can be used for
purposes beyond the PSM analysis. The approach enabled the categorization of compliance and
BEP outcomes (indicated by red, yellow and green in Tables 5-1 and 5-3) so that it is clear
which compliance outcomes are achieved by greater than 90% of marinas across the population
of New England marinas as well as which outcomes are achieved by less than 50% of marinas in
the region. A better understanding of levels of compliance, regardless of change, could provide a
38
-------
solid foundation for prioritizing planning and action, especially if the Initiative groups and
weights items on the checklist according to criteria such as feasibility, importance, cost, time,
and environmental impact.
There are many potential explanations for an item that is red (<50% compliance or use) in 2001
to remain red in 2004. The marina owner may not be aware of the need to change or factors such
as cost, time, complexity, or lack of knowledge and skills may prevent or delay change. Or, if an
item remains green pre- and post- implementation then the outcome may be more easily achieved
or is already integral to the culture and operations of marinas and may require less frequent
measurement. Additional insight is available where change is observed. For example, a large
percentage change, either within or between categories, suggests that obstacles to change in the
short term are not so great that the indicator is fixed. Those items may be easier or more likely
to change than others, whereas some, such as PWW, may be prohibitively expensive in the short-
term so that detecting change in the measurement window is less likely. Using appropriate and
relevant criteria to classify and group each item of compliance and BEPs, the Initiative can better
prioritize where and how to direct resources and effort.
Stakeholders report dual and seemingly conflicting purposes for the checklist. Marina owners
thought that the lack of state-specific information made it too general and sometimes confusing.
The checklist has both been described as an indicator tool for EPA Region 1 to assess a facility
as well as a tool to raise marina owners' awareness of environmental issues. The checklist
achieved the former but struggles with the latter due to, among other reasons, state-to-state
regulatory variations. If the checklist is to raise awareness amongst all marinas, it needs to
account for state variations to increase specificity and reduce confusion amongst marina owners.
Increasing specificity on the checklist would enable members of the assistance network to use
the checklist data to prioritize planning and create opportunities to increase efficiency at the state
and local level.
Environmental Health
Improving the environmental health of marinas communities is one of the intended long-term
outcomes of the Marinas Initiative. Neither stakeholder interviews nor a search of environmental
databases yielded data sources obviously useful in answering questions about the environmental
impacts of the Initiative. Some of the reasons for the lack of relevant data are documented in a
recent evaluation of EPA's Temporarily Integrated Monitoring of Ecosystems (TIME) and Long
Term Monitoring (LTM) surface water monitoring programs
(http://www.epa.gov/evaluate/pdf/FactSheetTIME-LTM.pdf). The evaluation found that the
TEVIE/LTM data was managed by a small number of researchers, not widely accessible, and
primarily collected to assess the environmental status of surface waters and not generally useful
for assessing the environmental outcomes of policies or programs such as the Marinas Initiative.
Though the evaluation did not find clearly useful ecological data sources, the Initiative has not
asked clear questions about the environmental health surrounding marinas or the impact of the
Initiative on environmental conditions. With clear questions about environmental health related
to compliance and BEP priorities, the initiative can explicitly identify data sources and gaps. For
instance, if curbing the use or runoff of a particular chemical or group of hazardous wastes is a
top priority, then the Initiative can focus on determining the availability of pollutant-specific data
39
-------
or investigate the feasibility of collecting it. Region 1 identified checklist items (e.g. NPDES
permits) related to regulations protecting water quality surrounding marinas (Table 5-1). With
expert input from stakeholders, the Initiative may be able to use existing measures or develop
surrogate measures for environmental impact of Initiative.
Communication
The theory and goals of the Initiative place a strong emphasis on networks, partnerships and
coordination across all stakeholders including marinas. However, the customer satisfaction
survey, distributed by mail, suffered a very low response rate which reduced the quantity and
quality data and information available for the evaluation. A mail survey during the high season
for marinas was an inefficient way to collect data and the complications with the list of physical
addresses was only one issue preventing communication with the community of marina owners.
Email addresses, a marinas listserve, social media (Web 2.0 tools such as Linkedln and Twitter),
and instant and text messaging offer the Initiative powerful ways to communicate with their
customers and create the option for the Initiative to use web-based or email surveys to gather
data and information. Creating, maintaining and building a social network, like the network of
compliance assistance providers and marina owners, requires multiple types of contact
information and means of communication to suit the membership's diverse communication
styles and preferences. The communications tools provided to the network provide constant
conduits for stakeholder interaction and feedback as well as opportunities to efficiently distribute
information to stakeholders.
It is likely that marina owners did not recognize the significance of the survey and did not realize
it was a part of the New England Marinas Initiative. Stakeholders were aware of Region 1' s
work with marinas, and States in particular were grateful for EPA's assistance in developing
education materials for marinas. However, representatives from State agencies were clear that
the Initiative did not contribute to the development of the education materials listed in Table 5-6.
Stakeholders were unaware of an "Initiative" specifically aimed at improving compliance and
use of BEPs amongst marinas in New England. The lack of awareness of the New England
Marinas Initiative may have played a role in marina owners missing the connection between the
customer satisfaction survey and the Initiative and resulted in fewer completed and returned
surveys. If the formal, cooperative and concerted effort across many stakeholders is not known
as the New England Marina Initiative, it is more difficult to communicate the Initiative's value
and increase demand for the program. It is also more difficult for others to request the
information and support necessary to replicate the program if they are unaware that it (the
Initiative) exists. Inefficiently communicating the successes and lessons learned to those who
would gain from that information reduces the value of the Initiative and the benefits that it has to
offer.
40
-------
CHAPTER 5: RECOMMENDATIONS
RECOMMENDATION 1: CLARIFY THE THEORY OF CHANGE
The planning and needs assessment stage, as depicted in the logic model, was detailed and
successful as is evident by the implementation of the Initiative and its various components.
Improve the theory of the program's implementation stage by clarifying which activities and
materials are intended to lead to which desired change. For instance, which program activity or
output is intended to achieve which BEP for which audience? Clearly articulate how
measurement information will be used and by whom in order to improve the utility of
measurement instruments and to reduce confusion about the use of measurement data.
RECOMMENDATION 2: EMPHASIZE NEAR-TERM, PRACTICAL OUTCOMES
Because the assumption that providing knowledge to marina owners will lead to behavior change
is unproven, changes in behavior should be the foundation of the Initiative's learning objectives.
Education and technical assistance materials and activities should teach to change the intentions
of owners as well as their use of skills and knowledge that will improve compliance and use of
BEPs. Use peer teaching-learning education approaches when possible as a means to increase the
practicality of education and assistance activities.
Determine and hone the effectiveness of education and technical assistance tools and strategies
through a combination of ongoing measurement and evaluation of the Initiative, pilot testing,
focus groups, and learning from this evaluation. Where there are many options for achieving the
same critical objective (e.g. change in a high priority compliance item), consider measuring the
relative effectiveness of different approaches (activities, materials) and allocate resources
according to which ones work best. Pay close attention to the contextual influences on the
program such as employee turnover rate. That the turnover rate is high could suggest that marina
owners should not be taught what to do with as much emphasis as they are taught how to achieve
regulatory compliance and BEPs through seasonal employees.
RECOMMENDATION 3: PRIORITIZE GOALS AND OBJECTIVES
Because there are a large number of desired outcomes and limited resources available to achieve
them, use the data and analyses in this report (e.g. levels of and changes in compliance) to
prioritize program goals and objectives that serve to order desired outcomes and the resources
allocated to achieving them. Use priority goals and outcomes to prioritize learning objectives and
necessary resources for developing and deploying education and assistance activities and
materials. Clearly defined goals and objectives can be used as evidence to guide decisions about
resources allocated to measurement activities. For instance, the highest priority learning and
behavior change objectives are the first to receive the most measurement attention (funding,
sophistication, etc.).
RECOMMENDATION 4: SPECIFY ENVIRONMENTAL OBJECTIVES
To support goal prioritization, identify specific pollutants and/or ecological characteristics most
relevant to the health of marina communities in New England. Document the existence of or
gaps in data and information related to those pollutants and/or ecological characteristics and
document what is needed to fill those gaps. Based on access to information, consider specifying
41
-------
key measurable environmental objectives for the program. Consider whether compliance, BEPs
and other existing indicators serve as surrogate measures of environmental health in the near
term (e.g. NPDES reports).
RECOMMENDATION 5: FORMALIZE THE INITIATIVE
Actively brand and promote the New England Marinas Initiative to improve communications
with program partners as well as other local, state and regional governments and agencies that
could benefit from implementing the program or its components.
Opportunities to formalize and brand the Initiative include:
• Establishing an advisory group of key stakeholders to lead or guide the Initiative's
development and implementation.
• Drafting a New England Marina Initiative strategic plan or Memorandum of Understanding
and a budget history of the Initiative.
• Formalizing and improving communication within the Initiative, including the compliance
assistance network and marina owners, through a combination of:
o The collection and use of more and diverse contact information (home phone, cell, email,
web address)
o Social media (Linkedln, Twitter, Blogs);
o Regular communications (listserve, webinars, conference calls, survey monkey); and
o Regular events (annual or semi annual conferences, workshops).
• Giving multiple people or organizations access to contact information so that the loss of one
member of the network does not affect the functioning and usefulness of the network to all its
members.
42
-------
REFERENCES
Caliendo, Marco and Sabine Kopeinig, 2005. "Some Practical Guidance for the Implementation
of Propensity Score Matching," IZA Discussion Paper No. 1588, May.
Rosenbaum, P. and D. Rubin, 1983. "The Central Role of the Propensity Score in Observational
Studies of Causal Effects," Biometrika, vol. 70, pp. 41-50.
43
-------
-------
APPENDIX A: INTERVIEW GUIDE FOR STAKEHOLDER INTERVIEW
INTERVIEW QUESTIONS FOR STAKEHOLDERS
BACKGROUND/CONTEXT
1. Describe your involvement in the New England Marina Workgroup
2. How long have you been involved with the program?
STATE OF MARINA SECTOR
3. What do you see as the current environmental challenges facing the marina sector?
4. What economic obstacles may influence a marina's ability to meet regulatory
requirements or to use best environmental practices?
5. What other factors (e.g., marina owners, marina industry, boaters) may influence marinas
ability to meet regulatory requirements or to use best environmental practices?
PROGRAM MATERIALS/RESOURCES
6. The Marina Checklist (sent along with this set of questions) was designed as an
environmental indicator tool to allow marina owners and assistance providers to
determine to what degree that they are in regulatory compliance and implementing best
environmental practices. As you review the checklist are there any items that you would
add or ones that you would remove?
7. Are you familiar with any other guidance materials developed by the New England
Marina Workgroup (e.g., Marina Environmental Management Plan, storm water permit
fact sheet)?
7a. [If yes] Do you have any recommendations on how the materials could be
improved?
7b. [For state agencies] How did your participation in the marina initiative influence
your state's compliance assistance efforts? [If yes] Please describe.
8. Have you attended any workshops or training demonstrations that EPA has participated
in? [If yes], do you have any suggestions on how to improve these workshops?
IMPROVING PROGRAM STRUCTURE
A-l
-------
9. EPA is interested in recommendations you have regarding how the program began and
was implemented. Do you have any suggestions related to:
• Clarifying policies
• Streamlining procedures
• Enhancing communication systems
• Ensuring accountability of the program.
10. Do you have any lessons learned from your participation in this marina workgroup that
you would to pass on to similar workgroups?
HEALTHIER MARINA COMMUNITIES
11. Are you aware of any environmental indicator data that would help measure the impact
of marinas on the environment for marinas that you are familiar with?
A-2
-------
APPENDIX B: EPA REGION 1, MARINA CHECKLIST FOR DATA
COLLECTION DURING SITE VISITS
US Environmental Protection Agency, New England OMBSIOII-OOII
MARINA CHECKLIST
For All Questions, Please Use:
Y:Yes N: No DK:Don=tKnow NA: Not Applicable
Facility Type
1. a. How many employees work at the facility during peak recreational boating season? Full-
time Part-time
b. How many boats(capacity) are moored at the facility? Docked? Stored on land?
2. a. Are maintenance or repair operations performed at the facility?
b. If yes, are these primarily customer do-it-yourself activities?
Hazardous Waste
3. a. Are manifests documenting hazardous waste shipments kept, going back at least 3 years?
b. Do employees receive training in: Proper handling of wastes? Emergency
procedures? c. Are quantities of hazardous waste generated by the marina
calculated each month, to determine what size generator the marina is?
d. Are all hazardous wastes stored: In labeled containers? In a dedicated storage
area? Indoors or covered? In an area with an impervious floor?
With storage area spill containment?
e. Are all hazardous wastes shipped with a properly licensed transporter?
Oil and Fuel
4. a. Is oil (including motor fuel) stored above ground in any size tank(s) with a total aggregate
volume over 1320 gallons?
Below ground in any size tank(s) with a total aggregate volume over 42000 gallons?
b. Is a Spill Prevention, Control and Countermeasure plan (SPCC):
On site? Signed by a registered professional engineer? Posted in plain view at
oil storage locations?
5. a. Does above ground oil storage (including piping system) have: Secondary containment?
Leak detection?
b. Does below ground oil storage (including piping system) have: Secondary containment?
Leak detection?
B-l
-------
c. Are spill prevention procedures in place for:
Receiving oil from a supplier? Transferring oil within the facility?
Waste oil disposal?
6. a. Does the facility have spill prevention procedures in place for fuel dispensing?
Overfill alarm? Automatic shutoff? Fuel collars to capture splash/drips?
Employee monitoring of fueling? Other?
b. Is equipment available and procedures in place to contain a spill at the dispenser
location?
c. How would you evaluate the effectiveness of the methods you are using to prevent releases?
Excellent Good Fair Poor
Hazardous Materials
7. Has the amount of each hazardous material stored onsite been calculated (including motor
fuel in above-ground systems of greater than 10,000 pounds capacity) to determine if
reporting to the Local Emergency Planning Committee is necessary?
8. Are up to date Material Safety Data Sheets (MSDS) for all hazardous chemicals kept on
file? Used for training all employees handling hazardous chemicals?
Storm Water
9. a. Is a National Pollutant Discharge Elimination System (NPDES) Storm Water Permit
required of this facility?
b. Does the marina have a NPDES Storm Water Permit?
10. a. When pressure washing boats coated with ablative paints, are any efforts undertaken to
prevent removed material from releasing to water? from contaminating land?
b. Are blasting, other paint preparation and painting activities contained or controlled to
prevent abrasives, paint chips, and over spray from being released to the water? to
land? to protect employees?
c. Are all engine fluids promptly transferred from parts, drip pans, used filters and other
containers to closed receptacles for disposal or recycling?
d. Are fuels, solvents and paints stored in a protected, secure location, away from drains?
Plainly labeled?
11. Has the marina made structural changes to minimize surface water runoff? Berming?
Vegetation? Riprap? Drains? Placement of filters in drains? _
Other?
Other
12. Has the facility switched to alternative materials or products to reduce toxicity or other
hazards to Health, safety or the environment? Safer paint stripping? Safer
painting?
B-2
-------
Safer MSD odor chemicals? Dust collection? Phosphate free cleaners?
Biodegradable cleaners? Safer antifreeze? Other?
13. Does the facility have a sewage pump out system? Onshore Boat How
many gallons (approx) do you pump out per week? Do you use Clean Vessels
Act funds?
14. Has the marina taken any action to improve environmental performance not included above?
Please
explain
This checklist does not include all marina requirements and preferable practices. For further
information, see EPA New England=s regional marina website at:
(http//www.epa.gov/regionl/marinas) . In addition, your comments or suggestions can be sent
to: U.S. EPA New England, Mail Code SPP-Marinas, One Congress St., Boston, MA, 02114
B-3
-------
Table B-l. Categorization of Checklist Items Between Regulatory Compliance and
Environmental Practices Outcome Categories.
Best
Question #
Description
Regulatory
Complianc
e Measure
Best
Enviro-
nmental
Practice
Indication
Hazardous Waste
3a
Are manifests documenting hazardous waste shipments
kept, going back at least 3 years?
X
3b
Do employees receive training in proper handling of
wastes?
Do employees receive training in emergency procedures?
X
X
3c
Are quantities of hazardous waste generated by the marina
calculated each month, to determine what size generator
the marina is?
X
Are all hazardous wastes stored in labeled containers?
Are all hazardous wastes stored in a dedicated storage
area?
3d
Are all hazardous wastes stored indoors or covered?
Are all hazardous wastes stored in an area with an
impervious floor?
Are all hazardous wastes stored with storage area spill
containment?
X
X
X
X
X
3e
Are all hazardous wastes shipped with a properly licensed
transporter?
X
Oil and Fuel
4a
Is oil (including motor fuel) stored above ground in any
single tank with over 660 gallons capacity? [a]
Is oil (including motor fuel) stored above ground in total
aggregate capacity of over 1,320 gallons capacity?
Is oil (including motor fuel) stored below ground in total
aggregate capacity of over 42,000 gallons?
Is a Spill Prevention, Control, and Countermeasure plan
(SPCC) on site?
4b
Is a Spill Prevention, Control and Countermeasure plan
(SPCC) signed by a registered professional engineer?
Is a Spill Prevention, Control and Countermeasure plan
(SPCC) posted in plain view at oil storage locations?
X[b]
X[b]
X[b]
X[c
X[c
X[c
5a
Does above ground oil storage (including piping system)
have: Secondary containment?
Does above ground oil storage (including piping system)
have: Leak detection?
X[b]
X[b]
X[c
X[c
5b
Does below ground oil storage (including piping system)
have: Secondary containment?
X[b]
X[c
B-4
-------
Question #
Description
Does below ground oil storage (including piping system)
have: Leak detection?
Regulatory
Complianc
e Measure
X[b]
Best
Enviro-
nmental
Practice
Indication
X[c]
B-5
-------
Question #
5c
6a
6b
6c
Description
Are spill prevention procedures in place for: Receiving oil
from a supplier?
Are spill prevention procedures in place for: Transferring
oil within the facility?
Are spill prevention procedures in place for: Waste oil
disposal?
Does the facility have spill prevention in place for
dispensing fuel?
Does the facility have an overfill alarm?
Does the facility have automatic shutoff?
Does the facility have fuel collars to capture splash/drips?
Does the facility have employee monitoring of fueling?
Does the facility have other spill prevention procedures?
(fill in)
Is containment in place in case of a spill at dispenser
location?
How would you rate the effectiveness of the methods you
are using to prevent releases?
Regulatory
Complianc
e Measure
X
X
X
X
X
X
Best
Enviro-
nmental
Practice
Indication
X[dj
X[dj
X[dJ
X[dj
XM
X
Hazardous Materials
7
8
Has the amount of each hazardous material stored onsite
been calculated (including motor fuel in above-ground
systems of greater than 10,000 pounds capacity) to
determine if reporting to the Local Emergency Planning
Committee is necessary?
Are up to date Material Safety Data Sheets (MSDS) for all
hazardous chemicals kept on file?
Are up to date Material Safety Data Sheets (MSDS) for all
hazardous chemicals used for training all employees
handling hazardous chemicals?
X
X
X
Stormwater
9a
9b
lOa
Is a National Pollutant Discharge Elimination System (NPDES) Storm
Water Permit required of this facility?
Does the marina have a NPDES Storm Water Permit?
When pressure washing boats coated with ablative paints,
are any efforts undertaken to prevent removed material
from releasing to water?
When pressure washing boats coated with ablative paints,
are any efforts undertaken to prevent contamination of
land?
X[ej
X
X
B-6
-------
lOb
Are blasting, other paint preparation and painting activities
contained or controlled to prevent abrasives, paint chips,
and overspray from being released to the water?
ana overspray trom pemg released to tne water!
Are blasting, other paint preparation and painting activities
contained or controlled to prevent abrasives, paint chips,
and overspray from being released to land?
A -rci r\\ o 0+1 tt rr /~v+lt £*-t~ f\oi tt+ f\t*£if\o t*o+i r\-r\ o-r\ /~\ f\oi -
Are blasting, other paint preparation and painting activities
contained or controlled to prevent abrasives, paint chips,
and overspray from being released to protect employees?
X
X
X
Question #
Description
Regulatory
Complianc
e Measure
Best
Enviro-
nmental
Practice
Indication
lOc
Are all engine fluids promptly transferred from parts, drip
pans, used filters, and other containers to closed receptacles
for disposal or recycling?
X
lOd
Are fuels, solvents, and paints stored in a protected secure
location, away from drains?
X
Are fuels, solvents, and paints plainly labeled?
X
Has the marina made structural changes to minimize
surface water runoff?
X
Has the marina implemented berming to minimize surface
water runoff?
X
Has the marina made changes to vegetation to minimize
surface water runoff?
X
II
Has the marina made changes to riprap to minimize
surface water runoff?
X
Has the marina made changes to drains to minimize
surface water runoff?
X
Has the marina used the placement of filters in drains to
minimize surface water runoff?
X
Has the marina made other structural changes to minimize
surface water runoff? (fill in)
X
Other Best Environmental Management Practices
12
Has the facility switched to alternative materials or
products to reduce toxicity or other hazards to health,
safety, or the environment?
Has the facility switched to safer paint stripping to reduce
toxicity or other hazards to health, safety, or the
environment?
Has the facility switched to safer painting to reduce
toxicity or other hazards to health, safety, or the
environment?
X
X
X
B-7
-------
Has the facility switched to safer MSD odor chemicals to
reduce toxicity or other hazards to health, safety, or the
environment?
X
Does the facility use dust collection methods to reduce
toxicity or other hazards to health, safety or the
environment?
X
Has the facility switched to phosphate free cleaners to
reduce toxicity or other hazards to health, safety, or the
environment?
X
Has the facility switched biodegradable cleaners to
reduce toxicity or other hazards to health, safely, or the
environment?
X
Has the facility switched to safer anti-freeze to reduce
toxicity or other hazards to health, safety, or the
environment?
X
Has the facility switched to other materials or products to
reduce toxicity or other hazards to health, safety, or the
environment? (fill in)
X
B-8
-------
Question #
13
14
Description
Does the facility have a sewage pump out system?
Does the facility have a sewage pump out system onshore?
Does the facility have a sewage pump out system on-boat?
Has the marina taken any action to improve environmental
performance not included above?
Regulatory
Complianc
e Measure
Best
Enviro-
nmental
Practice
Indication
X
X
X
x
This question was eliminated in the 2004 survey since the regulatory threshold for above
ground storage tanks is 1,320 gallons.
[b] This item is a regulatory requirement if a facility's aggregate above ground storage is 1,320
gallons or more and/or a below ground storage tank has a capacity of 42,000 gallons or more.
[c] If oil is stored at a facility under the regulatory thresholds then implementation of these items
is a best environmental practice.
[d] Having spill prevention in place for dispensing fuel is a regulatory requirement (part one of
question 6 A) and indicates that a facility has implemented at least one of the items described in
the sub-questions of 6 A. However, if a facility has implemented more than one of the items
described, this is categorized as a best environmental practice.
[e] This indicator is a regulatory requirement if aNPDES permit is required.
[f| This indicator is a best environmental practice if a NPDES permit is not required.
B-9
-------
APPENDIX C: CUSTOMER SATISFACTION SURVEY QUESTIONNAIRE
EPAICR 1711.04
OMB Control Number 2090-0019
Expiration Date: 08/31/09
Customer Service Survey Questionnaire
New England Marinas Initiative
Environmental Protection Agency Region 1
1. Since 2001, have you attended any workshops or training sessions on marina
environmental issues that were either sponsored or co-sponsored by EPA?
Yes
No [Go to Question?]
2. How did you learn about the workshop(s) or training session(s) that you attended?
__EPA web site
Mailings
E-mail announcement
Word of mouth
Advertisement in trade publication (or other material)
Other EPA-sponsored events/training
Other (please specify)
3. What would be your most satisfactory way to learn about workshops and/or training(s)?
__EPA web site
Mailings
E-mail announcement
Word of mouth
Advertisement in trade publication (or other material)
Other EPA-sponsored events/training
Other (please specify)
4. On a scale of 1 to 4 (with 1 being "Very Dissatisfied" and 4 being "Very Satisfied"), how
satisfied were you with the content of the workshop(s) and/or training with respect to each of the
following practices:
Did not
cover/Do
not recall
Spill prevention procedures or devices (e.g. overfill alarms)
Use of closed receptacles for disposal or recycling
C-l
-------
Clear labeling and secure storage of fluids (e.g. fuels and paints)
Structural changes to minimize surface water runoff (e.g.
berming, filters and riprap)
Use of alternative practices or materials to reduce toxicity or
other hazards
Installation of a sewage pump-out system (onshore or on-boat)
5. On the whole, how satisfied were you with the content of those workshops or training
sessions?
Very satisfied
Somewhat satisfied
Somewhat dissatisfied
Very dissatisfied
6. Do you have any additional comments about your satisfaction with the workshops and/or
training sessions (e.g. content, location, instructors)?
7. Over the last three years, have you ever visited the EPA New England Marinas web site
to obtain information that would assist you in dealing with environmental issues at your
marina?
Yes, very often
Yes, somewhat often
Yes, but only rarely
No [Go to Question 14]
8. How did you learn about the EPA New England Marinas web site?
__EPA web site
Mailings
E-mail announcement
Word of mouth
Advertisement in trade publication (or other material)
Other EPA-sponsored events/training
Other (please specify)
9. What would be your most satisfactory way to learn about EPA New England Marinas
web site?
__EPA web site
Mailings
C-2
-------
E-mail announcement
Word of mouth
Advertisement in trade publication (or other material)
Other EPA-sponsored events/training
Other (please specify)
10. How satisfied were you with the design of the site [i.e., how easy was it to locate the
materials that you were looking for on the web site]?
Very satisfied
Somewhat satisfied
Somewhat dissatisfied
Very dissatisfied
I was unable to locate the materials I was looking for [Go to Question 15]
11. On a scale of 1 to 4 (with 1 being "Very Dissatisfied" and 4 being "Very Satisfied"), how
satisfied were you with content of the web site with respect to the following practices:
1
Did not
234 cover/Do
not recall
Spill prevention procedures or devices (e.g. overfill alarms) D D D D D
Use of closed receptacles for disposal or recycling D D D D D
Clear labeling and secure storage of fluids (e.g. fuels and paints) D D D D D
Structural changes to minimize surface water runoff (e.g.
berming, filters and riprap)
Use of alternative practices or materials to reduce toxicity or
other hazards
Installation of a sewage pump-out system (onshore or on-boat)
12. On the whole, how satisfied were you with the content of the web site?
_Very satisfied
_Somewhat satisfied
_Somewhat dissatisfied
Very dissatisfied
13. Do you have any additional comments about your satisfaction with the web site (e.g.
content, technical complexity, additional resources, format)?
C-3
-------
14. Since 2001, have you used any EPA or state guidance documents or fact sheets to assist
with the marina's environmental compliance or suggestions about best environmental
practices?
_Yes
No...
.[Go to Question20]
15. How did you learn about the guidance documents or fact sheets?
_EPA web site
_Mailings
_E-mail announcement
_Word of mouth
_Advertisement in trade publication (or other material)
Other EPA-sponsored events/training
Other (please specify)
16. What would be your most satisfactory way to learn about guidance documents or fact
sheets?
_EPA web site
_Mailings
_E-mail announcement
_Word of mouth
_Advertisement in trade publication (or other material)
_Other EPA-sponsored events/training
_Other (please specify)
17. On a scale of 1 to 4 (with 1 being "Very Dissatisfied" and 4 being "Very Satisfied") how
satisfied were you with the content of the guidance materials with respect to the
following practices:
Did not
cover/Do
not recall
Spill prevention procedures or devices (e.g. overfill alarms)
Use of closed receptacles for disposal or recycling
Clear labeling and secure storage of fluids (e.g. fuels and paints)
Structural changes to minimize surface water runoff (e.g.
berming, filters and riprap)
Use of alternative practices or materials to reduce toxicity or
other hazards
Installation of a sewage pump-out system (onshore or on-boat)
C-4
-------
18. On the whole, how satisfied were you with the content of the guidance materials?
Very satisfied
Somewhat satisfied
Somewhat dissatisfied
Very dissatisfied
19. Do you have any additional comments about your satisfaction with the guidance materials
(e.g. content, technical complexity, additional resources, format)?
Finally, in order to understand which facilities are most satisfied with which products and
services and to improve the products and services available to your facility, we have a few
more questions.
20. How many full- and part-time employees work at your facility?
Full time employees
Part time employees
21. How many boats are moored, docked, or stored on land at your facility?
Boats Moored
Boats Docked
Boats stored on land
22. Are maintenance or repair operations performed at the facility?
Yes, they are primarily do-it-yourself activities performed by boat owners
Yes, but they are not primarily do-it-yourself activities performed by boat owners
No
23. How many years of experience do you have with the following environmental
practices at your marina?
T ^, 0 Between ,. ^
xT T, . Less than 3 _ , , More than
No Expenence 3 and 6 ,
years 6 years
years
Spill prevention procedures or devices (e.g. overfill i—i PI PI PI
alarms)
Use of closed receptacles for disposal or recycling CJ D D D
C-5
-------
Clear labeling and secure storage of fluids (e.g.
fuels and paints)
Structural changes to minimize surface water runoff
(e.g. berming, filters and riprap)
Use of alternative practices or materials to reduce
toxicity or other hazards
Installation of a sewage pump-out system (onshore
or on-boat)
Thank you for your participation in the survey! If we may contact you to clarify your
responses, please provide an e-mail address:
Burden Statement: Public reporting burden for this collection of information is estimated to average
10 minutes per response, including the time for reviewing instructions, gathering information, and
completing and reviewing the collection of information. Send comments on the Agency's need for this
information, the accuracy of the provided burden estimates, and any suggestions for reducing the burden,
including the use of automated collection techniques to the Director, OEI Collections Strategies Division,
United States Environmental Protection Agency, Mail Code 2822,1200 Pennsylvania Avenue, N.W.,
Washington, DC 20460-0001; and to the Office of Information and Regulatory Affairs, Office of Management
& Budget, 725 17th Street NW, Washington, DC 20503. Attention: Desk Officer for EPA. Include the EPA
ICR number (1711.05) and the OMB control number (2090-0019) in any correspondence.
C-6
-------
APPENDIX D: PROPENSITY SCORE MATCHING ANALYSIS
Outcome
Are Manifests Documenting Hazardous Waste Shipments Kept,
Going Back At Least 3 Years?
Do employees receive training in proper handling of wastes?
Do employees receive training in emergency procedures?
Are Quantities Of Hazardous Waste Generated By The Marina
Calculated Each Month, To Determine What Size Generator
The Marina Is?
Are All Hazardous Wastes Stored In Labeled Containers?
Are All Hazardous Wastes Stored In A Dedicated Storage
Area?
Are All Hazardous Wastes Stored Indoors Or Covered?
Are All Hazardous Wastes Stored In An Area With An
Impervious Floor?
Are All Hazardous Wastes Stored With Storage Area Spill
Containment?
Are All Hazardous Wastes Shipped With A Properly Licensed
Transporter?
Is Oil (Including Motor Fuel) Stored Above Ground In Any
Single Tank With Over 660 Gallons Capacity?
Is Oil (Including Motor Fuel) Stored Above Ground In Total
Aggregate Capacity Of Over 1,320 Gallons?
Is Oil (Including Motor Fuel) Stored Below Ground In Total
Aggregate Capacity Of 42,000 Gallons?
Is A Spill Prevention, Control And Countermeasure Plan
(SPCC) On Site? [b]
Is A Spill Prevention, Control And Countermeasure Plan
(SPCC) On Site? [c]
Is A Spill Prevention, Control And Countermeasure Plan
(SPCC) Signed By A Registered Professional Engineer? [b]
Is A Spill Prevention, Control And Countermeasure Plan
(SPCC) Signed By A Registered Professional Engineer? [cl
Is A Spill Prevention, Control And Countermeasure Plan
(SPCC) Posted In Plain View At Oil Storage Locations? [bl
2001 Data
Marinas
With a
Reported
Outcome
55
63
63
63
59
60
60
60
60
49
57
54
54
10
36
9
36
10
Average
Value for
Outcome
76.40%
81.00%
81.00%
15.90%
57.60%
81.70%
71.70%
66.70%
45.00%
83.70%
26.30%
18.50%
100.00%
20.00%
2.80%
11.10%
2.80%
0.00%
2004 Data
Marinas
With a
Reported
Outcome
56
65
60
61
59
58
58
57
55
57
1
58
59
14
29
13
17
13
Average
Value for
Outcome
75.00%
73.80%
81.70%
14.80%
59.30%
86.20%
82.80%
75.40%
58.20%
86.00%
100.00%
25.90%
100.00%
35.70%
24.10%
15.40%
5.90%
7.70%
Propensity Score Matching Analysis
Entities in
Treatment
Group
51
60
55
57
54
53
53
52
50
53
1
54
54
8
22
8
14
8
Matched
Controls
53
61
61
61
58
56
56
56
56
48
13
53
53
9
33
9
33
10
ATT
-0.047
-0.154
-0.064
-0.031
0.061
0.081
0.126
0.118
0.16
-0.023
0.692
0.03
0
0.192
0.267
0.083
0.062
0
Standard
Error
0.091
0.081
0.076
0.071
0.099
0.076
0.084
0.093
0.1
0.079
-
0.084
0
0.247
0.103
0.206
0.08
0
t statistic
-0.52
-1.89
-0.834
-0.428
0.616
1.06
1.494
1.276
1.598
-0.291
-
0.361
-
0.778
2.578
0.404
0.775
-
Outcome
Category
[a]
R
R
R
R
R
R
R
R
R
R
-
-
-
R
BP
R
BP
R
D-l
-------
Outcome
Is A Spill Prevention, Control And Countermeasure Plan
(SPCC) Posted In Plain View At Oil Storage Locations? \c\
Does Above Ground Oil Storage (Including Piping System)
Have Secondary Containment? [bl
Does Above Ground Oil Storage (Including Piping System)
Have Secondary Containment? [c]
Does Above Ground Oil Storage (Including Piping System)
Have Leak Detection? [b]
Does Above Ground Oil Storage (Including Piping System)
Have Leak Detection? [cl
Does Below Ground Oil Storage (Including Piping System)
Have Secondary Containment? [b]
Does Below Ground Oil Storage (Including Piping System)
Have Secondary Containment? [c]
Does Below Ground Oil Storage (Including Piping System)
Have Leak Detection? [bl
Does Below Ground Oil Storage (Including Piping System)
Have Leak Detection? [c]
Are Spill Prevention Procedures In Place For Receiving Oil
From A Supplier?
Are Spill Prevention Procedures In Place For Transferring Oil
Within The Facility?
Are Spill Prevention Procedures In Place For Waste Oil
Disposal?
Does The Facility Have Spill Prevention Procedures In Place
For Fuel Dispensing?
Does The Facility Have More than One Spill Prevention
Procedure In Place For Fuel Dispensing?
Is Containment In Place In Case Of A Spill?
How Would You Evaluate The Effectiveness Of The Methods
You Are Using To Prevent Releases? [d]
How Would You Evaluate The Effectiveness Of The Methods
You Are Using To Prevent Releases? [dl
Has The Amount Of Each Hazardous Material Stored Onsite
Been Calculated (Including Motor Fuel In Above-Ground
Systems Greater Than 10,000 Lbs Capacity) To Determine If
Reporting To The Local Emergency Planning Committee Is
Necessary?
2001 Data
Marinas
With a
Reported
Outcome
36
10
10
9
11
2
28
2
28
41
27
37
50
42
52
53
53
63
Average
Value for
Outcome
2.80%
90.00%
50.00%
55.60%
45.50%
100.00%
96.40%
100.00%
96.40%
75.93%
48.21%
67.27%
96.00%
87.50%
86.50%
2.8679
2.8679
23.80%
2004 Data
Marinas
With a
Reported
Outcome
16
15
18
13
19
3
21
3
22
27
35
40
43
34
47
49
49
51
Average
Value for
Outcome
12.50%
86.70%
38.90%
61.50%
36.80%
100.00%
85.70%
100.00%
90.90%
64.29%
74.47%
75.47%
83.70%
94.40%
83.00%
3.1837
3.1837
25.50%
Propensity Score Matching Analysis
Entities in
Treatment
Group
13
9
12
8
14
1
19
1
19
38
43
48
38
31
42
45
45
42
Matched
Controls
33
10
9
9
10
2
27
2
26
54
56
55
49
47
51
52
52
61
ATT
0.144
-0.111
-0.017
-0.038
0.009
0
-0.062
0
-0.037
-0.057
0.251
0.049
-0.151
0.094
-0.064
0.288
0.288
-0.015
Standard
Error
0.11
0.184
0.243
0.268
0.231
-
0.083
-
0.084
0.101
0.100
0.096
0.071
0.073
0.082
0.128
0.128
0.094
t statistic
1.31
-0.604
-0.069
-0.143
0.04
-
-0.747
-
-0.44
-0.567
2.507
0.512
-2.115
1.290
-0.779
2.241
2.241
-0.161
Outcome
Category
[a]
BP
R
BP
R
BP
R
BP
R
BP
R
R
R
R
BP
R
BP
R
R
D-2
-------
Outcome
Are Material Safety Data Sheets (MSDs) For All Hazardous
Chemicals Kept On File?
Are Material Safety Data Sheets (MSDs) Used For Training All
Employees Handling Hazardous Chemicals?
Is A National Pollutant Discharge Elimination System
(NPDES) Storm Water Permit Required Of This Facility?
Does The Marina Have A NPDES Storm Water Permit? [e]
Does The Marina Have A NPDES Storm Water Permit? [fj
When Pressure Washing Boats Coated With Ablative Paints, Is
Removed Material Collected To Prevent Releases To Water?
When Pressure Washing Boats Coated With Ablative Paints, Is
Removed Material Collected To Prevent Contamination Of
Land?
Are Blasting, Other Paint Preparation And Painting Activities
Contained Or Controlled To Prevent Abrasives, Paint Chips,
And Overspray From Being Released To The Water?
Are Blasting, Other Paint Preparation And Painting Activities
Contained Or Controlled To Prevent Abrasives, Paint Chips,
And Overspray From Being Released To Land?
Are Blasting, Other Paint Preparation And Painting Activities
Contained Or Controlled To Prevent Abrasives, Paint Chips,
And Overspray From Being Released To Protect Employees?
Are All Engine Fluids Promptly Transferred From Parts, Drip
Pans, Used Filters And Other Containers To Closed
Receptacles For Disposal Or Recycling?
Are Fuels, Solvents And Paints Stored In A Protected, Secure
Location, Away From Drains?
Are Fuels, Solvents And Paints Plainly Labeled?
Has The Marina Made Structural Changes To Minimize
Surface Water Runoff?
Has The Marina Made Structural Changes To Berming?
Has The Marina Made Structural Changes To Vegetation?
Has The Marina Made Structural Changes To Riprap?
Has The Marina Made Structural Changes To Drains?
Has The Marina Made Structural Changes To Placement Of
Filters In Drains?
2001 Data
Marinas
With a
Reported
Outcome
63
67
69
41
14
61
61
51
51
52
57
60
60
65
65
64
64
64
64
Average
Value for
Outcome
63.50%
52.20%
76.80%
2.40%
100.00%
59.02%
55.74%
82.35%
82.35%
90.40%
96.50%
95.00%
91.70%
69.20%
55.40%
51.60%
21.90%
25.00%
14.10%
2004 Data
Marinas
With a
Reported
Outcome
66
49
61
48
8
63
62
55
57
51
61
65
58
62
60
62
58
55
50
Average
Value for
Outcome
78.80%
73.50%
85.20%
35.40%
100.00%
63.49%
53.23%
81.82%
73.68%
92.20%
90.20%
80.00%
56.90%
71.00%
40.00%
51.60%
44.80%
29.10%
42.00%
Propensity Score Matching Analysis
Entities in
Treatment
Group
61
45
58
46
7
58
58
51
52
46
56
60
54
58
56
58
54
52
47
Matched
Controls
61
65
67
40
12
59
59
49
49
50
55
58
58
63
63
62
62
62
62
ATT
0.106
0.197
0.078
0.354
0
-0.019
-0.072
-0.023
-0.114
0
-0.065
-0.125
-0.368
0.047
-0.167
-0.012
0.212
-0.017
0.24
Standard
Error
0.087
0.096
0.074
0.077
0
0.096
0.098
0.082
0.087
0.062
0.05
0.06
0.079
0.087
0.096
0.097
0.09
0.088
0.088
t statistic
1.223
2.054
1.053
4.592
-
-0.193
-0.737
-0.280
-1.315
0.006
-1.29
-2.104
-4.632
0.532
-1.729
-0.126
2.367
-0.19
2.732
Outcome
Category
[a]
R
R
-
R
BP
R
R
R
R
R
R
R
R
BP
BP
BP
BP
BP
BP
D-3
-------
Outcome
Has The Facility Switched To Alternative Materials Or
Products To Reduce Toxicity Or Other Hazards To Health,
Safety, Or The Environment?
Has The Facility Switched To Safer Paint Stripping?
Has The Facility Switched To Safer Painting?
Has The Facility Switched To Safer MSB Odor Chemicals?
Has The Facility Switched To Dust Collection?
Has The Facility Switched To Phosphate-Free Cleaners?
Has The Facility Switched To Biodegradable Cleaners?
Has The Facility Switched To Safer Antifreeze?
Does The Facility Have A Sewage Pumpout System?
2001 Data
Marinas
With a
Reported
Outcome
66
41
52
57
50
53
62
60
65
Average
Value for
Outcome
89.40%
46.30%
73.10%
26.30%
66.00%
37.70%
88.70%
95.00%
69.20%
2004 Data
Marinas
With a
Reported
Outcome
57
40
51
44
51
52
55
53
69
Average
Value for
Outcome
87.70%
52.50%
62.70%
31.80%
78.40%
48.10%
72.70%
96.20%
62.30%
Propensity Score Matching Analysis
Entities in
Treatment
Group
53
39
48
42
48
50
51
50
64
Matched
Controls
64
39
50
55
48
51
60
58
63
ATT
-0.017
0.071
-0.116
0.034
0.141
0.098
-0.174
0.005
-0.039
Standard
Error
0.063
0.119
0.098
0.095
0.095
0.103
0.077
0.043
0.089
t statistic
-0.274
0.594
-1.188
0.356
1.479
0.956
-2.244
0.124
-0.439
Outcome
Category
[a]
BP
BP
BP
BP
BP
BP
BP
BP
BP
[a] Outcomes are classified as either "regulatory compliance" (R) or "best environmental practice" (BP).
[b] This outcome is a regulatory requirement if a facility's aggregate above ground storage is 1,320 gallons or more and/or a below ground storage tank has a capacity of 42,000 gallons
or more; this row includes only those facilities for which this outcome is a regulatory requirement
[c] If oil is stored in a smaller storage tank than the regulatory thresholds, then implementation of these outcomes is a best environmental practice; this row includes only those facilities
for which this outcome is a best environmental practice.
[d] This outcome was based on a four-point scale (where 4 was "Excellent" and 1 was "Poor"), as opposed to the rest of the outcomes in the table that all had Yes/No responses. Thus,
the "Average Value For Outcome" columns (for both the 2001 and 2004 data) for this outcome contain the actual average rating for the group (as opposed to the percentage of "Yes"
answers that is indicated for the other questions).
[e] This outcome is a regulatory requirement if a NPDES permit is required; this row includes only those facilities for which this outcome is a regulatory requirement.
[fj This outcome is a best environmental practice if a NPDES permit is not required; this row includes only those facilities for which this outcome is a best environmental practice.
D-4
-------
APPENDIX - additional figures
Figure D-l provides a graphical representation of the indicator outcomes for regulatory compliance highlighting changes between
2001 and 2004. In the figure, there are three qualitative categories for the indicators based on the percentage of facilities that attained
each:
• Green (good): More than 90 percent of facilities attained the outcome,
• Yellow (moderate): Between 50 and 90 percent of facilities attained the outcome, and
• Red (poor): Less than 50 percent of facilities attained the outcome.
Indicators are grouped first by their 2001 values and then show how the 2004 values were distributed. For example, in Figure D-l, we
see that there were eight indicators in the Good category in 2001.
D-5
-------
Good = 0
Moderate = 18
Good = 0
Moderate = 2
Moderate = 4
Poor = 0
2001 Good Total = 8
Poor = 0
2001 Moderate Total = 18
2001 Poor Total = 8
Figure D-l. 2004 Regulatory Compliance Indicators Assessed Qualitatively (Good, Moderate, and Poor) Grouped By 2001 Qualitative Assessments
Figure D-2 provides a graphical representation of the indicator outcomes for best practices highlighting changes between 2001 and 2004. As with the
regulatory compliance outcomes, we have created three qualitative categories for the indicators based on the percentage of facilities that attained each: Good
(more than 90 percent of facilities attained the outcome), Moderate (between 50 and 90 percent of facilities attained the outcome), and Poor (less than 50 percent
D-6
-------
of facilities attained the outcome). Indicators are first grouped by their 2001 values to show changes from 2001 to 2004. Among the BEP data, we see only one of
four indicators deteriorate from Good to Moderate. Among the 2001 categorized as Moderate, we see one improvement to Good, but two deteriorations to Poor.
Finally, among the Poor values in 2001, we see one improvement to Moderate.
Good = 0
Moderate = 1
Moderate = 1
Poor = 0
2001 Good Total = 4
2001 Moderate Total = 10
2001 Poor Total = 10
Figure D-2. 2004 Best Management Practice Indicators Assessed Qualitatively (Good, Moderate, and Poor) Grouped By 2001 Qualitative Assessment
D-7
-------
APPENDIX E: PROPENSITY SCORE MATCHING
METHOD DESCRIPTION
Propensity Score Matching (PSM) is a method that allows for comparison of outcomes between treatment and
control groups in cases where a randomized control trial (RCT) has not been used. An alternative approach to
making these comparisons would have been to use standard statistical hypothesis testing to test the differences
between two mean values. PSM represents a more rigorous approach to looking for program impacts since it
involves comparing matched units over time, thus mimicking a RCT. In fact, PSM is based on the premise that
the best method of determining the impact of a program is to use an RCT. RCTs, however, are not feasible in
non-experimental situations. In a RCT, a population (or sample from a population) is randomly divided into two
groups: an experimental group and a control group. An intervention (or treatment) is applied to the experimental
group, but not to the control group. The impact of the treatment is estimated by comparing outcomes for the
experimental and control groups. The following diagram reflects the basic measurement model employed in a
RCT:
Random Before T . After
A • T + +• Intervention
Assignment Intervention Intervention
->-No
Y,
The diagram depicts the two groups with the experimental group being denoted as Y and the control group as X.
Units are randomly assigned to each group from a larger group (U). The intervention/treatment is applied to the
experimental group while the control group receives no intervention/treatment. Each group is observed before
the intervention (subscript equals 1) and after the intervention (subscript equals 2) and the value of some
outcome is recorded at each observation point. Letting Y, and X, (/'=!, 2) be the measured values for the
outcomes, then the impact of the intervention can be measured as
(Y2-Y1)-(X2-X1)
A number of standard statistical tests can then be conducted to determine if the impact of the intervention, as
measured above, is significantly different than zero.
In a non-experimental setting, some of these conditions break down. One common issue is that the population in
non-experimental settings is not randomly divided between experimental and control groups. This is not an
issue for the New England Marina Initiative, however, since the 2001 and 2004 site visits were randomly
selected from the population of marinas and the marinas chosen for the customer satisfaction survey also will be
randomly selected.
A second issue is that we may not observe both pre- and post-intervention outcomes for the experimental group.
This is an issue for the New England Marina Initiative: we have post-intervention outcome data the marinas
visited in 2004, but no data on these groups prior to the intervention. Additionally, we have data for a control
group, i.e., marinas visited in 2001, but no follow-up data for that group. From the diagram above, we have the
values Y2 and Xi, but not the values YI or X2.
E-l
-------
One approach to deal with the missing outcome data is to focus on averages for each group rather than on
individual unit outcome measurements. Focusing on averages, however, has two important shortcomings. First,
it ignores the fact that we are missing data values for YI or X2.14 The fact that the groups are all randomly
selected reduces the impact of this problem to some degree. However, because the measurements are taken
three or more years apart, comparing averages does not provide an ideal way of estimating an impact. For
example, there might have been changes between 2001 and 2004 in the way marinas operate that are unrelated
to the program (intervention). If this is the case, then the value (X2 - Xi) will control for those changes. That is,
the value (X2 - Xi) represents the change between 2001 and 2004 in untreated units. Using average values
ignore the value (X2 - Xi).
To overcome the issue with using averages, we can use matching techniques. Matching involves pairing treated
and untreated units and then calculating the difference in outcomes between the matched pairs. This approach
has a long history in evaluation research. The simplest matching approach is to choose some fixed factors and
then match treated and untreated units based on those factors. For example, for the marinas, we could divide
them into "small" and "large" marinas and divide them based on state. We could then compare small marinas in
each state with one another and large in each state with one another. This still relies on averages, however, since
we are still grouping the units. A more complex approach would rely on matching treated units to untreated
units based on a statistical comparison on how well the units match to one another.
One issue, however, has to do with self-selection. That is, if the matching factors are correlated with treatments,
then the estimated impacts of treatment from matched pairs will be biased. In an important paper in matching,
Rosenbaum and Rubin (1983) identified conditions under which the bias can be controlled. Specifically, it is
necessary to match units based on a single (composite) "score" that accounts for the matching factors and is also
independent of treatment decisions.15 They also showed that a good score is one that reflects the probability of
treatment given an observed set of factors (e.g., the matching factors). Such a score is often referred to as the
"propensity score" and can be estimated using standard statistical techniques such a logistic regression analysis.
Propensity score values can be estimated for both treated and untreated units and the two groups can be matched
based on values of the propensity scores. In applying PSM, there are a number of choices that need to be made
(Caliendo and Kopeinig, 2005). We discuss some of those choices in the implementation section below.
There are some important things to understand about PSM, however. First, in PSM, there is no requirement that
a single treated unit be matched to a single untreated unit. In fact, in many cases it is more efficient (in a
statistical sense, which means a lower variance) to match one treated unit to more than one untreated unit.
Second, there is no requirement to match all treated units to at least one untreated unit. There may be some
treated units that do not have a good match. Third, one untreated unit can be matched to more than one treated
unit. That is, an untreated unit may be a good match for more than one treated unit. Finally, there must be some
overlap in the observed factors used for estimating the propensity score. For example, if all of the treated units
are "large" and all of the untreated are "small," then there is no overlap on size.
Method Implementation
In applying PSM to this evaluation, we will be estimating the impact of the program as a whole. That is, we will
be comparing the changes in outcomes between a baseline period (i.e., 2001) and a follow-up period (i.e.,
2004). The data available to us do not allow for comparing the impact of the various program components. The
outcomes will reflect both (self-reported) regulatory compliance and (self-reported) use of best environmental
14 In fact, the implicit assumption is that the average for Xi will approximate the averages for both YI and X2.
15 The use of a composite score values overcomes what is known as the "curse of dimensionality" issue. For example, if the matching
factors consist of n dichotomous variables, then there are 2" dimensions that need to be accounted for in the matching. For continuous
variables, the number of dimensions is more complex. The composite score reduces that number of dimensions to one: the values of
the composite variable.
E-2
-------
practices.16 We will look at changes between 2001 and 2004. In all cases, the "treatment group" will be marinas
visited in 2004 during the site visits and the "control group" will be marinas visited in 2001.
Implementing PSM for each analysis will require three main steps:
• Step 1—Estimate the propensity scores. In this first step, the analyst will estimate a logistic regression
model setting the dependent variable equal to one for members of the treatment group and zero for members
of the control group. The covariates in the logistic regression will be a set of fixed factors such as marina
size (e.g., number of boats docked) and location (state). Based on the resulting logistic regression model, we
will estimate propensity scores for each marina in the control and treatment groups.17
• Step 2—Match treatment group members with control group members using a matching algorithm. In order
to match treatment with control group members, the analyst will use the caliper matching method. This
method involves defining an acceptable interval in the propensity score (e.g., 0.01 percentage points) and
then matching each treatment group member to any control group member that has a propensity score within
the defined interval around the treatment group member's score. For example, if a treatment group member
has a propensity score of 20 percent and we are using a caliper of 0.1 percentage points, then any marina
with a propensity score between 19.9 percent and 20.1 percent is considered a match to the treatment group
marina. This approach to matching incorporates two other decisions:
o Allowing each treatment group member to be matched to more than one control group member.
o Allowing each control group member to be used as a control for more than one treatment group
member.
• Step 3—Calculate the "treatment" effect. In the final step, the analyst will calculate the impact of the
treatment. For the most part, this will be done by calculating the difference in the mean values between the
treatment and control groups and determining whether that difference is statistically significant using the
proper standard errors that take into account the matching used.
16 The set of regulatory compliance and best environmental practices outcomes can be found in Table B-l of Appendix B.
17 The propensity score for any marina is the probability of that marina being in the treatment group calculated using the estimated
logistic regression model and the marinas values for the covariates included in the model.
E-3
------- |