United States Office of Policy
Environmental Protection (1807T)
Agency
December 2012
EPA-100-K-12-012
Measuring the
Effectiveness of the
Ocean Dumping
Management Program
Final Report
December 15, 2012
-------
ACKNOWLEDGEMENTS
This evaluation of the Ocean Dumping Management Program was developed for the U.S.
Environmental Protection Agency's Office of Policy (OP) under Contract EP-W-10-002
between EPA and Industrial Economics, Inc. (IEc) of Cambridge, MA. Eastern Research
Group (ERG) of Lexington, MA assisted in the evaluation as a subcontractor to IEc.
Andy Schwarz, Tracy Dyke Redmond, and Josh Wolff represented IEc. Lou Nadeau and
Melanie Sands represented ERG.
Ryan Gross and Davie Nguyen provided important background information on the Ocean
Dumping Management Program. Matt Keene of OP's Evaluation Support Division
provided technical support and advice for the evaluation. Special thanks go to EPA staff
at Headquarters and the Regions who shared their perspectives and feedback to inform
the evaluation.
This report was developed under the Program Evaluation Competition, sponsored
annually by EPA's Office of Policy. Program Evaluation is one of the performance
management tools US EPA uses to assure itself, the public, and other interested
stakeholders that US EPA programs are protecting human health and the environment
effectively and efficiently. The information obtained through program evaluations can
shed light on whether US EPA programs are meeting their goals and objectives, provide
the evidence and road map needed to replicate successes, and identify those aspects of
US EPA programs needing improvement. To access copies of this or other EPA program
evaluations, please go to EPA's Evaluation Support Division website at
http://www.epa.gov/evaluate.
-------
TABLE OF CONTENTS
GLOSSARY
EXECUTIVE SUMMARY
CHAPTER 1 | INTRODUCTION AND BACKGROUND
Introduction 1-1
The Evaluation 1-2
Goal and Scope of the Evaluation 1-2
Evaluation Roadmap 1-3
Evaluation Questions 1-4
CHAPTER 2 | METHODOLOGY
Measures and Sources of Data 2-1
Literature Review 2-4
Pre-Interview Survey of EPA Staff 2-4
Interviews 2-4
Focus Groups 2-4
Analysis of Data 2-5
Overview 2-5
Data Storage 2-5
Interview Data Analysis 2-5
Survey Data Analysis 2-5
Document Review 2-6
Synthesis 2-6
Data Limitations 2-6
CHAPTERS | PROGRAM THEORY OF CHANGE
Ocean Dumping Management Program Logic Model 3-1
Findings 3-1
Program Processes and Activities 3-3
Headquarters 3-3
Regions 3-4
Program Assumptions 3-4
Program Information Flows and Feedback Mechanisms 3-5
Summary of Ocean Dumping Management Program in Practice 3-6
Allocation of Program Resources 3-10
Alignment of Partner Goals 3-11
-------
Conclusions 3-14
Recommendations 3-14
CHAPTER 4 | EMERGING TRENDS, OPPORTUNITIES, AND CHALLENGES
Findings 4-1
Historical Trends and Emerging Issues 4-1
Decreases in Ocean Dumping, Increased Focus on Beneficial Reuse 4-1
Advances in Science and Understanding of Contaminants 4-3
Changes in Program Resources 4-6
Staff Turnover 4-9
Trends and Emerging Issues Specific to EPA Regions 4-9
Obstacles and Delays 4-11
Opportunities for Improvement 4-12
Opportunity to Exercise Greater Coordination and Leadership from EPA HQ 4-13
Opportunity to Update Regulations and Guidance 4-14
Opportunity Related to London Protocol Ratification 4-14
Conclusions 4-15
Recommendations 4-16
CHAPTERS | RESOURCE ALLOCATION
Findings 5-1
Resources Required vs. Those That Can Be Reallocated 5-1
Opportunities to Improve Alignment of Resources with Intended Program Outcomes 5-1
Opportunities to Improve Alignment of Resources Across Regions 5-2
Opportunities, Risks, and Impacts of Resource Reallocation 5-3
Conclusions 5-3
Recommendations 5-4
CHAPTER 6 | PERFORMANCE MEASUREMENT
Findings 6-2
Current Measures 6-2
Strengths and Weaknesses of Current Measures 6-3
Potential Alternative measures 6-4
Outcome Measures 6-4
Output/Efficiency Measures 6-5
Conclusions 6-5
Recommendations 6-6
-------
APPENDICES:
APPENDIX A: SURVEY QUESTIONS
APPENDIX B: INTERVIEW GUIDES
APPENDIX C: FOCUS GROUP DISCUSSION GUIDE
APPENDIX D: FTE RESOURCES AND CONTRACT DOLLARS APPLIED TO THE
LOGIC MODEL
111
-------
GLOSSARY
Acronym Definition
CWA Clean Water Act
BSD U.S. EPA Evaluation and Support Division
FTE Full Time Equivalent
FY Fiscal Year
HQ Headquarters
MPRSA Marine Protection, Research & Sanctuaries Act
NGO Non-Governmental Organization
ORISE Oak Ridge Institute for Science and Education
OW U.S. EPA Office of Water
OWOW U.S. EPA Office of Wetlands, Oceans, and Watersheds
PCB Polychlorinated Biphenyl
PEC U.S. EPA Program Evaluation Competition
SINKEX U.S. Navy Sinking Exercise
USAGE U.S. Army Corps of Engineers
IV
-------
EXECUTIVE SUMMARY
The U.S. Environmental Protection Agency (EPA) Marine Pollution Control Branch,
within the Office of Water (OW), administers the Ocean Dumping Management
Program in coordination with each of EPA's seven coastal Regions (Regions 1, 2, 3, 4, 6,
9, and 10). The Ocean Dumping Management Program managers applied for and were
awarded resources to evaluate the program in order to provide information that will help
managers better align resources and activities with intended outcomes. This information
should set the stage for future program improvements by increasing awareness among
Ocean Dumping Management Program staff about how the program is working and
leverage points where the program could change its approach to be more effective.
Based on interviews with program staff and stakeholders, a survey of program staff, and
reviews of the literature and program documentation, the evaluation team answered four
primary evaluation questions and multiple sub-questions. These questions, and the
findings, are summarized below:
1. What is the Ocean Dumping Management Program's theory of change?
• What are the processes and activities that the Ocean Dumping Management
Program uses to achieve its outcomes? The EPA HQ Ocean Dumping Management
Program has five key processes/functions: 1) establishing environmental
criteria/guidance for implementing MPRSA; 2) reviewing, concurring, and issuing
ocean dumping permits; 3) designating ocean disposal sites; 4) establishing and
managing Site Management and Monitoring Plans; and 5) enforcing MPRSA. To
support these roles, the program seeks to undertake several activities, including: 1)
developing and updating legislation; 2) working with international organizations and
the State Department on international treaties and protocols; 3) developing policy on
emerging issues; 4) developing regulations and policies, and ensuring that existing
policies stay current; 5) issuing and updating guidance for implementing the
program; 6) addressing petitions, letters, and other inquiries; 7) issuing special or
general (non-dredge material) permits; 8) assisting Regions in reviewing and (where
appropriate) concurring on dredged material permits issued by USAGE; 9) assisting
Regions in conducting site monitoring and site designation; 10) assisting Regions and
U.S. Coast Guard with regulatory enforcement; 11) reviewing regional programs; 12)
coordinating across Regions; and 13) conducting strategic planning. Evaluation
findings suggest that the program is currently conducting most but not all of these
actions.
• What are assumptions that lead from activities/outputs to outcomes? The
program theory rests on assumptions that: 1) the regulations are current, 2) there is
adequate information to make program decisions, 3) there is clear communication
ES-1
-------
within EPA, 4) there are adequate program resources, and 5) program partners
cooperate with EPA. The evaluation finds that these assumptions do not hold true in
all cases.
• How are the program's resources allocated across the outcomes? The program's
FTEs are allocated to activities intended to lead to the following long-term outcomes
in descending priority order: environmentally acceptable conditions resulting from
disposal (46 percent); coordinated and effective Ocean Dumping Management
Program (29 percent); no legal vulnerability (15 percent); and maintaining an
international leadership role (10 percent). The program's contract dollars are
deployed in a similar manner across outcomes: Environmentally acceptable
conditions resulting from disposal (38 percent); maintaining an international
leadership role (31 percent); coordinated and effective Ocean Dumping Management
Program (22 percent); and no legal vulnerability (9 percent).
• What are the feedback mechanisms from outputs/outcomes to activity planning?
Information flows can serve as a feedback mechanism that helps program staff and
managers adapt their activities. Current information flows include: sharing
information from EPA Regions; interagency meetings; complaints from program
partners; Regional Dredging Team conversations; and public feedback. There are
opportunities to improve information flows, e.g., by conducting focused regular
meetings between the EPA Regions and HQ.
• At a high level, what are the goals of the non-EPA program partners? How do
these goals align with the goals of the Ocean Dumping Management Program?
For EPA Regions, the goals are to maintain environmentally acceptable conditions
resulting from disposal. For USAGE, the goals related to ocean dumping are to
maintain navigation and meet regulatory responsibilities for dredged material with a
focus on cost effectiveness. Other program partners include the EPA Clean Water
Act 404 Program, the Coast Guard, Department of Interior, Environmental Non-
Governmental Organizations, U.S. Fish and Wildlife Service, National Oceanic and
Atmospheric Administration, State Agencies, the U.S. Navy, and individual ports and
harbors. These partners have a wide array of goals, from natural resource protection
to dredging, regulating mineral rights, and vessel disposal. As a regulatory program,
EPA's goals do not align with all of its customers' and partners' goals, which is to be
expected. The one area of misalignment that seems to cause the most challenges for
the Ocean Dumping Management Program is the tension in goals between EPA and
USAGE.
2. How can the program deal with emerging issues, challenges, and
opportunities for national and regional implementation of the program?
• What are the emerging issues, challenges, and opportunities for the Ocean
Dumping Management Program? Are these the same for the national program
and the Regions? Historical trends and emerging issues common to all the EPA
Regions include: 1) decreases in ocean dumping nationwide, and an increased focus
on beneficial reuse, 2) advances in science and understanding of contaminants, 3)
ES-2
-------
changes in program resources, and 4) staff turnover. Emerging issues specific to
Regions include Gulf of Mexico restoration, major harbor and channel deepening
projects in the southeastern US, ensuring proper disposal of debris from the Japanese
tsunami on the west coast, ensuring proper disposal offish waste in Alaska and on
the west coast, and protecting sensitive near shore habitats (e.g., reefs) in the
Caribbean from effects of dredged material transport.
How have the environmental issues addressed by the program changed over the
past 20 years, and how have these changes been reflected in program design and
resource allocation? What changes in ocean dumping-related environmental
issues and associated goals can reasonably be expected over the next 10 years?
The environmental issues addressed by the program have indeed changed over the
last 20 years, particularly since ocean disposal of sewage sludge was prohibited in all
EPA Regions in 1992. Since that time, the focus of the program has shifted to
primarily regulating ocean disposal of dredging material, and preventing ocean
dumping of most other wastes. While we do not have budgetary information about
the program that stretches back as far as the 1990s, interviewees report that the
program has fewer resources now than it has had historically, and program staff say
that the value of the program is less appreciated in recent years than it has been
historically. Going forward, the Ocean Dumping Management Program is likely to
be asked to address an emerging set of issues, particularly related to emerging
contaminants and climate change.
What guidance and other assistance should be provided to Regions to assist
them in dealing with the emerging issues, challenges, and opportunities?
Program staff and stakeholders express concern that the regulations and guidance
documents have not kept pace with advances in science and therefore need to be
updated. Regions have requested a range of guidance to assist them with emerging
issues, challenges, and opportunities, as well as advances in science. Regions request
guidance and coordination on emerging issues such as contaminants (e.g., pyrethroids
and pharmaceuticals) and issues related to climate change. Additional requests
include updating the testing guidance (Greenbook); guidance on fish waste, vessel
disposal, site management, beneficial reuse and the Coastal Zone Management Act;
advanced maintenance dredging/overdepth/sampling; and guidance on the Water
Resources Development Act of 1992.
What innovations or improvements are needed to deal with the emerging issues,
challenges, and opportunities? Regions seek a centralized repository of checklists,
resource lists, and other resources to improve consistency of program
implementation. Staff praise the recent Ocean Dumping Coordinators meeting as an
opportunity to foster coordination across the Regions and seek creative means to
gather Ocean Dumping coordinators from across the country in focused
conversations. Regions seek support from HQ establishing consistent expectations
for USAGE Districts
ES-3
-------
• Is the program encountering obstacles or delays? If so, what are those
obstacles/delays and what can be done to overcome them? The program is
encountering obstacles, such as unclear program boundaries and priorities,
conflicting goals of some partners, and lack of coordination. These obstacles lead to
delays in activities such as updating guidance documents and reviewing regional
programs. The program may overcome some of its obstacles by more clearly
communicating its value, more clearly articulating its mission and priorities, and
fostering greater communication and consistency across Regions.
3. Given stable or reduced program resources, what are the opportunities for
improving the alignment between resource deployment and the program's
intended or desired outcomes?1
• Does the current resource allocation reflect the desired allocation across
program goals? If not, where should the emphasis be? Yes, the current resource
allocation across its activities does reflect the program's goals and priorities.
However, resource alignment is not currently optimized, and interviewees generally
agree that the program could do a better job of focusing its efforts, working more
efficiently, and being more proactive. Program managers could improve resource
alignment by confirming the program's priorities, articulating the minimum resources
needed to meet those priorities at both the HQ and regional level, and requesting
support from senior leadership to focus the program's limited resources on those
priorities.
• What resources are required/fixed in the current allocation? What resources
can be reallocated and under what conditions? It appears that the program's
current resources are required (and in fact may not be adequate to support the scope
of the program's work). Thus it does not appear that resources can be reallocated
away from the program without reducing its scope. However, it may be desirable to
reconsider the reallocation of resources across Regions to ensure equity given the
variations in scale of work to be accomplished.
• Of the resources that can be reallocated, what risks and/or opportunities does
their reallocation pose? Resource reallocation (whether within the program or
outside of the program) poses risks of taking time and effort away from program
work. Program staff and managers have not developed a shared view of the
minimum requirements that HQ and the Regions must fulfill, and the resources
needed to complete those tasks. Without such an agreement, it is difficult to pinpoint
opportune areas for resource reallocation.
1 Additional sub-questions, which are answered as part of the findings for sub-questions described above, include: Would
resources being reallocated be as effective in other places/roles within the Ocean Dumping Management Program as in
their current place/role? What would be the impacts of resource reallocation on the program's customers? Would those
impacts lead to feedback risks or opportunities for the program?
ES-4
-------
• Do the ultimate goals/targeted outcomes reflect the program's mission? Do the
Regions share these goals and outcomes? Yes, the program goals are consistent
with the program's mission, and the Regions share these goals.
4. How can the Ocean Dumping Management Program measure its success?
• Are the current measures understood throughout the program? Yes, the current
measure is understood, although it is not interpreted consistently.
• Are the current metrics valid measures for the program? A performance measure
is valid if it accurately represents the condition or phenomenon that it is purporting to
represent. The current measure would be valid if the key term "environmentally
acceptable conditions" were consistently defined and interpreted.
• Are there other (more valid or reliable) measures? If so, what are the pros/cons
of the other measures relative to the current ones? There are supplementary
outcome and output measures that could help measure program results. Key criteria
for assessing measures are the extent to which the program has control over the
measure and the extent to which the measure reflects program goals.
• What can the current or potentially new measures tell us about program
success? Supplementary measures could help assess the quality and efficiency of the
program's work.
In light of the findings above, the evaluation team concludes that the Ocean Dumping
program faces numerous challenges, including significant budget restructuring, and at
times operates in an inconsistent and somewhat reactive mode as new issues arise. We
offer the following recommendations.
1. Clarify and communicate the importance of the program, focusing on why
this program is essential to protecting ocean ecosystems. We suggest that
program managers articulate the importance of the program with regard to
managing dredge materials and encouraging beneficial reuse, and state the
benefits of the program in terms of environmental benefits (as well as
implementing regulations).
2. Seek to foster improved communication and partnership with USAGE. We
recommend that the Ocean Dumping Management Program work towards more
productive communication with USAGE by acknowledging differences in
mission and organizational constraints, and focusing on areas of shared
responsibility.
3. Update program guidance and use the London Protocol ratification process
as an opportunity to update the regulations. We believe the program will
need to identify approaches to take on the additional responsibilities that will
come with ratification. As a part of updating program guidance, program
managers at HQ should work to improve communications to Regions about how
to implement the regulations, standard procedures, and expectations for program
ES-5
-------
partners (e.g., USAGE Districts). HQ should seek to facilitate communication
and consistency across Regions.
4. Strengthen information flows to inform program adaptation, building on a
suite of performance measures that encompass both outcome and output
measures. We particularly recommend the program consider the following
measures: volume of contaminated material not allowed to be ocean dumped
(and of this amount, percentage that is used for beneficial uses); percentage of
monitored sites found to be in compliance; average number of years since ocean
disposal sites were monitored; and average number of years since Site
Management and Monitoring Plans have been reviewed and updated.
5. Assess resource needs: We suggest that HQ define the minimum requirements
that the Regions must fulfill. The program should then research what future
program activity levels (e.g., permit review, site designation activities) can be
anticipated in each Region, given trends in ocean dumping and port development.
The program can then analyze what resources will be required in each Region to
meet minimum program requirements given anticipated activity levels. Finally,
the program should consider resource reallocation across Regions if necessary to
ensure a balance between effort required and resources provided. HQ will also
need to assess its own minimum resource requirements to carry out its core
activities at the national level, given work on the London Protocol and the need
to update the regulations and provide additional guidance to Regions.
ES-6
-------
CHAPTER 1 | INTRODUCTION AND BACKGROUND
This chapter introduces the purpose and scope of the Ocean Dumping Management
Program evaluation. The chapter goes on to provide background on the program and its
origins and describe the evaluation questions and key audiences for the evaluation
findings.
INTRODUCTION
Enacted in 1972, the Marine Pollution, Research, and Sanctuaries Act (MPRSA)
regulates transportation for the purpose of dumping of all types of materials into ocean
waters and prevents or strictly limits the dumping into ocean waters of any material that
would adversely affect human health, welfare, or amenities; or the marine environment,
ecological systems, or economic potentialities.2 The MPRSA implements the
requirements of the London Convention, an international treaty governing ocean
dumping.3
Certain materials, such as high-level radioactive waste, medical waste, sewage sludge,
and industrial waste, are banned from ocean disposal. The primary source of material that
may be dumped in the ocean, with appropriate oversight and permitting, is navigational
dredging undertaken to remove sediment from navigation channels and vessel berthing
areas. Other items that may be disposed in the ocean include vessels that are no longer in
operation, fish wastes, and human remains.
EPA is responsible for designating ocean disposal sites (both for dredged and non-
dredged material), establishing and managing Site Management and Monitoring Plans for
all ocean disposal sites, and establishing environmental criteria for the evaluation of
permit applications. Though the U.S. Army Corps of Engineers (USAGE) issues ocean
dumping permits for dredged material (subject to EPA review and concurrence), EPA is
responsible for issuing ocean dumping permits (special, general, research, or emergency)
for all other substances.
EPA's Marine Pollution Control Branch, within the Office of Water (OW) in
Washington, B.C., administers the Ocean Dumping Management Program in
2Title I of the Marine Protection, Research, and Sanctuaries Act (MPRSA), also referred to as the Ocean Dumping Act,
generally prohibits (1) transportation of material from the United States for the purpose of ocean dumping; (2)
transportation of material from anywhere for the purpose of ocean dumping by U.S. agencies or U.S.-flagged vessels; (3)
dumping of material transported from outside the United States into the U.S. territorial sea. A permit is required to deviate
from these prohibitions.
3 The U.S. has signed but not yet ratified the 1996 London Protocol, which is designed to update and clarify the London
Convention. As a signatory, the United States is obliged not to take any action to defeat the 1996 Protocol's object and
purpose.
1-1
-------
coordination with each of EPA's seven coastal Regions (Regions 1, 2, 3, 4, 6, 9, and 10).
The Headquarters (HQ) program addresses national policy and programmatic issues,
leads guidance development in coordination with the Regions, issues permits under the
MPRSA, addresses cross-regional issues, and represents the program on/in intra-agency,
interagency, and international workgroups and forums. The HQ program, in coordination
with the regional offices, regularly addresses public and other agency inquiries related to
dumping or placement of substances in the ocean and addresses a variety of issues during
and following large-scale emergencies (e.g., Hurricane Katrina, Deepwater Horizon Oil
Spill). Regional ocean dumping programs generally focus on dredged material
disposal—including review and concurrence on USACE-issued permits, site
designations, and site management and monitoring activities—as well as vessel, fish
waste, burial at sea, and emergency issues.
The Evaluation
An evaluation of the Ocean Dumping Management Program's past, present, and future is
timely. MPRSA was enacted forty years ago and this is the first systematic evaluation of
the program outside of OW. Like many programs within the federal government, the
Ocean Dumping Management Program is operating within the context of diminishing
resources. To better understand the program's priorities and to improve management of
program resources, Ocean Dumping Management Program managers applied for and
were awarded evaluation resources through the EPA Program Evaluation Competition
(PEC). The PEC is an ongoing Agency effort overseen by the Evaluation and Support
Division (ESD), within the Office of Policy, that promotes the effective use and
integration of program evaluation throughout the Agency.
This evaluation is a collaborative effort between ESD and Ocean Dumping Management
Program staff, with contractor support. The overall evaluation design and objectives are
the result of collective discussion and planning by the evaluation team, comprising a lead
evaluator from ESD, one primary Ocean Dumping Management Program representative,
an Oak Ridge Institute for Science and Education (ORISE) Fellow from OW, and the
contracting team of Industrial Economics, Inc. (lEc) and its subcontractor Eastern
Research Group, Inc. (ERG). In the remainder of this report, we use the term "the
evaluators" or "we" to refer to the contractors who prepared this report under the
oversight and guidance of the EPA staff.
Goal and Scope of the Evaluation
The goal of the evaluation is to provide information to allow the program to align
resources and activities with intended outcomes. This information should set the stage
for future program improvements by increasing awareness among Ocean Dumping
Management Program staff4 about how the program is working and leverage points
where the program could change its approach to be more effective. The evaluation team
agreed on the following scoping decisions:
4 In general, we use the term "staff" to indicate both managers and staff. Where we specifically intend to indicate program
managers, we use that term rather than "staff."
1 -2
-------
• Focus: The evaluation will focus on the national Ocean Dumping Management
Program and how it operates as a whole, including the regional components.
• EPA Regions as Partners: The evaluation will consider EPA Regions as partners
of the national Ocean Dumping Management Program staff to the extent that
they: a) contribute to the development of priorities for the program, and b)
conduct activities that contribute to program outputs. Otherwise, the Regions are
HQ's customers in the process.
• Other Federal Agencies: Other Federal agencies (e.g., USAGE) involved in
implementing requirements under the MPRSA may provide input to the
evaluation, as appropriate, but will not be the focus of the evaluation.
Evaluation Roadmap
Exhibit 1-1 provides an overview of the evaluation process and how it will inform
program design. Starting at the top of the figure and proceeding clockwise, the evaluation
team first established an understanding of the Ocean Dumping Management Program's
mission, goals, and objectives and then revised a logic model that had been provided by
the program to reflect this information. Next, the team developed evaluation questions
based on the logic model designed to support the goals of the evaluation. The evaluation
questions then flowed into the team's discussion of the discrete measures that would be
most appropriate to address each question and the data that would be needed to develop
these measures. We also considered how and from what sources the data would be
gathered, and how they would be analyzed. The evaluation methodology, described in
Chapter 2, documents each of these elements of the planned evaluation process. The team
gathered data and analyzed findings in accordance with the methodology and prepared
this report to present findings, conclusions, and recommendations for potential
adaptations to the program and/or adjustments in how the program defines its work.
EXHIBIT 1-1. STEPS IN THE EVALUATION PROCESS
Adapt the Pi-.
Describe the Program
Communicate Results
for Program Use
\
Program
Evaluation
Cycle
Establish the Evaluation
Questions
I
Conduct the Evaluation
Define the Measures
and Data Sources
1 -3
-------
EVALUATION QUESTIONS
The evaluation team, in consultation with representatives from the Ocean Dumping
Management Program, identified four primary questions:
1. What is the Ocean Dumping Management Program's theory of change? The
purpose of this question is to better understand the various components of the
Ocean Dumping Management Program (inputs, outputs, and outcomes), their
relationships, and how they function in relation to the program's intended goals
and outcomes. Answering this question provides an opportunity to learn from
key partners, particularly EPA Regions. For example, with regard to ensuring
environmentally acceptable conditions resulting from disposal, the evaluation
will address how the program expects its regulations, policies, and enforcement
activities will help achieve the goal. The report will address where the program
is currently focusing efforts and resources and how that focus aligns with the
priorities of partners and stakeholders in the program. The findings will inform
the remainder of the evaluation by pointing to places in the program where
resource adjustment, improved communication, changes in information tracked,
or a realignment of program focus may be needed to obtain targeted goals.
Program staff may use the findings to communicate key aspects of their program
and assumptions about how the program works.
2. How can the program deal with emerging issues, challenges, and
opportunities for national and regional implementation of the program?
This question serves three purposes: 1) to describe how changes in science,
technology, and environmental issues have influenced the evolution of the
program's design and management over time, 2) to better understand issues and
relationships that influence the program's ability to meet its intended goals, and
3) to identify and explore aspects of the program (e.g., existing strengths or
positive relationships) that can be leveraged to improve program functioning and
resilience.
3. Given stable or reduced program resources, what are the opportunities for
improving the alignment between resource deployment and the program's
intended or desired outcomes? This question takes stock of the Ocean
Dumping Management Program's resources and how they are allocated in order
to determine whether or not this allocation reflects the program's goals and
priorities. This question delves into which program resources are fixed, which
resources may be reallocated, the structure of resource flows and delays, and the
implications of changing the resource allocation, should that be necessary to
achieve different program outcomes or due to decreases in program funding.
4. How can the program measure its success? The purpose of this question is to
understand how the program can assess and track the extent to which it is
meeting its goals. The question helps identify current program measures of
success, assess whether those measures are valid and/or easily understood by
staff and external stakeholders, and suggest new or additional measures that
might be beneficial in gauging success.
1 -4
-------
Exhibit 1-2 identifies a series of sub-questions that are designed to inform the primary
questions, along with the key audiences and communication mechanisms for the findings
for each primary question.
EXHIBIT 1-2. EVALUATION QUESTIONS AND THE KEY AUDIENCE AND COMMUNICATION
MECHANISMS FOR FINDINGS
EVALUATION QUESTIONS AND SUB-QUESTIONS
KEY AUDIENCES
COMMUNICATION
MECHANISMS FOR
FINDINGS
1 . What is the Ocean Dumping Management Program's theory of change?
a) What are the processes and activities that
the Ocean Dumping Management Program
uses to achieve its outcomes?
b) What are assumptions that lead from
activities /outputs to outcomes?
c) How are the program's resources allocated
across the outcomes?
d) What are the feedback mechanisms from
outputs/outcomes to activity planning?
e) At a high level, what are the goals of the
non-EPA program partners? How do these
goals align with the goals of the Ocean
Dumping Management Program?
Program Manager for
Ocean Dumping
Management Program
Director of the Oceans
and Coastal
Protection Division
Director and Deputy
Director and Director
of the Office of
Wetlands, Oceans and
Watersheds (OWOW)
Program logic model
and accompanying
narrative theory of
change. The
evaluation team will
document the
evolution of the logic
model over the course
of the evaluation.
2. How can the program deal with emerging issues, challenges, and opportunities for national and
regional implementation of the program?
a) What are the emerging issues, challenges,
and opportunities for the Ocean Dumping
Management Program? Are these the same
for the national program and the Regions?
b) How have the environmental issues
addressed by the program changed over
the past 20 years, and how have these
changes been reflected in program design
and resource allocation? What changes in
ocean dumping-related environmental
issues and associated goals can reasonably
be expected over the next 10 years?
c) What guidance and other assistance should
be provided to Regions to assist them in
dealing with the emerging issues,
challenges, and opportunities?
d) What innovations or improvements are
needed to deal with the emerging issues,
challenges, and opportunities?
e) Is the program encountering obstacles or
delays? If so, what are those
obstacles/delays and what can be done to
overcome them?
Same audiences as for
Evaluation Question 1 ,
and potentially,
partner agencies
(e.g., Coast Guard
and USACE)
Narrative in final
evaluation report
1 -5
-------
EVALUATION QUESTIONS AND SUB-QUESTIONS
KEY AUDIENCES
COMMUNICATION
MECHANISMS FOR
FINDINGS
3. Given stable or reduced program resources, what are the opportunities for improving the
alignment between resource deployment and the program's intended or desired outcomes?
a) Does the current resource allocation
reflect the desired allocation across
program goals? If not, where should the
emphasis be?
b) What resources are required /fixed in the
current allocation? What resources can be
reallocated and under what conditions?
c) Of the resources that can be reallocated,
what risks and/or opportunities does their
reallocation pose?
d) Would resources being reallocated be as
effective in other places/roles within the
Ocean Dumping Management Program as in
their current place/role?
e) What would be the impacts of resource
reallocation on the program's customers?
Would those impacts lead to feedback
risks or opportunities for the program?
f) Do the ultimate goals/targeted outcomes
reflect the program's mission? Do the
Regions share these goals and outcomes?
Same audiences as for
Evaluation Question 1
Narrative in the final
evaluation report
4. How can the Ocean Dumping Management Program measure its success?
a) Are the current measures understood
throughout the program?
b) Are the current metrics valid measures for
the program?
c) Are there other (more valid or reliable)
measures? If so, what are the pros/cons of
the other measures relative to the current
ones?
d) What can the current or potentially new
measures tell us about program success?
Same audiences as for
Evaluation Question 1
Narrative and
examples in the final
evaluation report
1 -6
-------
CHAPTER 2 | METHODOLOGY
This chapter describes how the evaluation team planned to answer the evaluation
questions, specifically: 1) the measures used and sources of data; 2) approaches to
analyzing data gathered; and 3) expected limitations of the analysis.
MEASURES AND SOURCES OF DATA
There are four key sources of information used in this evaluation:
1. Literature review
EXHIBIT 2-1. SEQUENCE
OF DATA COLLECTION
3. Interviews with:
2. Survey of program staff
4.
• EPA Ocean Dumping Management Program
HQ staff
• EPA staff from each of the seven Regions
participating in the program
• Stakeholders, including representatives from
EPA's Clean Water Act Section 404 (inland
disposal) Program, USAGE, an environmental
group that has engaged EPA's Ocean Dumping
Management Program in the past, and a former
program manager (now retired).
Focus groups with EPA Ocean Dumping
Management Program staff and managers.
Exhibit 2-1 shows the sequence of data collection used in
the evaluation. These sources of information will inform
measures that are used to answer the primary evaluation
questions and their sub-questions. Exhibit 2-2 shows the
measures that are associated with each evaluation
question and the source of those data.
Documentation Review
Survey of Program Staff
Program Staff Interviews
Regional Ocean
Dumping Coordinators
Meeting
Focus Groups
Additional Program Staff
Interviews
Stakeholder Interviews
2-1
-------
EXHIBIT 2-2. EVALUATION DESIGN MATRIX: MEASURES AND DATA SOURCES
EVALUATION QUESTIONS AND MEASURES [A]
DATA SOURCES
LITERATURE
REVIEW
SURVEY OF
PROGRAM
STAFF
INTERVIEWS WITH PROGRAM STAFF FROM...
EPA HQ
EPA REGIONS
STAKEHOLDERS
FOCUS
GROUPS
SUMMATIVE
FROM
FINDINGS1"1
1. What is the Ocean Dumping Management Program's theory of change?
Summary of the processes and activities that the program
uses to achieve its outcomes (1a)
Description of the assumptions that lead from
activities/outputs to outcomes (1b)
Quantification of the program's resources (Full Time
Equivalent (FTE) hours and contract dollars) allocated across
the outcomes (1c)
Description of the feedback mechanisms from
outputs/outcomes to activity planning (1d)
Description of the high-level goals of the non-EPA program
partners and how they align with the goals of the Ocean
Dumping Management Program (1e)
2. How can the program deal with emerging issues, challenges and opportunities for national and regional implementation of the program?
Description of the emerging issues, challenges, and
opportunities for the Ocean Dumping Management Program at
the national and regional levels (2a)
Description of trends in environmental issues addressed by
the program over the past 20 years and related changes in
program design and resource allocation over the same period
(2b)
Description of guidance/assistance needed to deal with the
emerging issues, challenges, and opportunities (2c)
Description of innovations or improvements are needed to
respond to emerging issues, challenges, and opportunities
(2d)
Description of obstacles or delays encountered by the
program and potential responses (2e)
mm.
3. Given stable or reduced program resources, what are the opportunities for improving the alignment between resource deployment and the program's intended or
desired outcomes?
Description of current resource allocation and emphasis (3a)
^ I
2-2
-------
EVALUATION QUESTIONS AND MEASURES [A]
DATA SOURCES
LITERATURE
REVIEW
SURVEY OF
PROGRAM
STAFF
INTERVIEWS WITH PROGRAM STAFF FROM...
EPA HQ
EPA REGIONS
STAKEHOLDERS
FOCUS
GROUPS
SUMMATIVE
FROM
FINDINGS1"1
Description of desired resource allocation (3a)
W///A
Description of required/fixed elements of the resource
allocation (3b)
Description of elements that can be reallocated under what
conditions (3c)
Risks and opportunities of reallocation (3c)
mm,
m
Description of the effectiveness of reallocating resources as
compared to current place (3d)
Impacts of reallocation on the Ocean Dumping Management
Program's customers, e.g., EPA regional staff and potentially
other stakeholders
Description of the alignment of targeted goals/outcomes with
program mission (3f)
Description of the whether or not targeted goals/outcomes
are shared by Regions (3f)
4. Given the answer to question #3, how can the Ocean Dumping Management Program measure its success?
Description of current program measures (4a)
mm.
Description of how program understands current program
measures (4a)
Description of the validity of current program measures (4b)
Potential alternative measures and/or suggested revisions to
current measures and their pros and cons (4c)
Description of how current or new measures gauge program
success (4d)
[A] Parenthetical references following the measures link the measure to the sub-questions listed in Exhibit 1 -2.
[B] Builds on findings from other evaluation questions
2-3
-------
Each of the four data collection methods is described in further detail below.
Literature Review
The evaluation team reviewed documents that are publicly available or made available by
program staff in order to: 1) better understand the Ocean Dumping Management Program
and its functioning, 2) identify key budgetary, technical, and managerial issues and trends
related to ocean dumping, and 3) provide context on relevant historical trends and events.
Pre-lnterview Survey of EPA Staff
Prior to interviews, the evaluation team distributed a brief online survey to 40 Ocean
Dumping Management Program staff at HQ and the Regions. The purpose of the survey
was to use staff time as efficiently as possible by collecting information for selected
questions in a standardized way, and to identify topics to address in greater detail in
subsequent interviews. The evaluation team received 13 responses (a 33 percent response
rate), representing HQ and six of the seven Regions involved in the program. The survey
gathered respondent opinions, perspectives, and feedback regarding emerging issues,
challenges, and opportunities for national and regional implementation of the program.
The evaluation team developed the survey questions with input from Ocean Dumping
Management Program HQ staff. Appendix A contains a copy of the survey questions.
Interviews
Following review of the survey results, the evaluation team conducted interviews with
EPA staff and individuals outside the agency. The interviews occurred in two sets: first,
the evaluation team interviewed EPA regional staff and a representative of EPA's Office
of Research and Development, which provides consultation to the Ocean Dumping
Management Program. We summarized the findings of these interviews at the Regional
Ocean Dumping Coordinators Meeting in August 2012. Following the meeting, the
evaluation team conducted a second set of interviews with EPA HQ staff, a representative
of EPA's Clean Water Act (CWA) Section 404 program, and external stakeholders
(including USAGE, an environmental group, and a former Ocean Dumping Management
Program employee). The evaluation team developed tailored interview guides for each
set of interviewees in consultation with Ocean Dumping Management Program HQ staff.
Exhibit 2-3 summarizes the interviews conducted. Appendix B contains the interview
guides used.
Focus Groups
The evaluation team conducted three one-hour focus groups with program staff at the
Regional Ocean Dumping Coordinators meeting in Washington, D.C., in August 2012 to
add depth and detail to the information learned through the interview and survey. The
discussions focused primarily on verifying and refining the theory of change, use of
program resources, effectiveness of HQ activities, and engagement with partners.
Appendix C includes the discussion guide for the focus groups.
2-4
-------
EXHIBIT 2-3. SUMMARY OF INTERVIEWS CONDUCTED
GROUP INTERVIEWED
EPA INTERVIEWEES
Regional staff
Office of Research and Development
HQ staff
CWA Section 404 staff
EXTERNAL INTERVIEWEES
USAGE
Environmental group (Clean Ocean Action)
Former Ocean Dumping Management Program
employee
NUMBER OF INTERVIEWS
CONDUCTED
6
1
2
1
1
1
1
ANALYSIS OF DATA
Overview
The general analytical approach for the evaluation included a content analysis of the
information gathered. In general, the analysis was qualitative, but where possible we
included numerical analysis of quantitative data points such as resource and FTE trends.
Data Storage
The evaluation team developed a master set of all notes in Microsoft Word to serve as the
foundation of the analysis. This file, along with the survey results, but excepting any
information deemed confidential by the interviewees, will be available to EPA upon
completion of the evaluation.
Interview Data Analysis
Given the manageable scope of data collected through the interviews, the evaluation team
did not code interview responses, but rather identified common themes and gathered
evidence from all data sources to address each evaluation question. The evaluation team
compiled quotes where appropriate to illustrate key points (quotes are not attributed). We
also noted where there were areas of consensus, and where there were differences in
perspective across different groups of interviewees (e.g., across different groups of
Regions, or between HQ and Regions).
Survey Data Analysis
The evaluation team compiled and analyzed survey data to look for trends and patterns
related to the Ocean Dumping Management Program. The survey results include the
percentage of respondents selecting specific responses or identifying particular themes.
2-5
-------
Document Review
The evaluators reviewed documents provided by the Ocean Dumping Management
Program and searched for publicly available literature relevant to this evaluation using
sources such as the EPA website, grey literature, and peer-reviewed journal articles. The
literature review helped us understand the Ocean Dumping Management Program, as well
as the historical context for the program, emerging issues, and resource trends.
Synthesis
To develop our conclusions and recommendations, the evaluation team synthesized
information across the data sources. We used multiple data sources to answer each
evaluation question and to verify apparent trends and patterns. Where different data
sources suggested different conclusions, we noted this and, when appropriate, searched
for additional data to confirm the analysis.
DATA LIMITATIONS
Throughout this study, the evaluation team sought to collect representative, objective, and
robust data. However, as with all program evaluations, data collection and analytical
limitations exist, and we make these transparent in our findings. We currently note the
following limitations to this evaluation methodology:
• Non-experimental research design: This evaluation is not suited to use of
comparison or control groups. However, the evaluation does involve
comparisons, particularly comparing information about the program over time
(e.g., current program activities and resource allocation, recollections about how
the program operated in the past, and expected future opportunities and
challenges).
• Limited information drawn from outside of EPA: The evaluation primarily
drew on information from within the Ocean Dumping Management Program
itself (e.g., program documentation, survey responses, and interview notes). The
evaluation team did gather some perspectives from outside EPA (e.g., through a
limited number of stakeholder interviews), but the evaluation does not
necessarily present a representative summary of viewpoints from stakeholders
outside EPA.
• Potential uncertainties in data on program resource allocation data: We
analyzed program resource quantities (FTE hours and contract dollars) allocated
across program outcomes based on data provided by program staff, including
work plans with staff allocations to specific program activities. If these work
plans and program staff input did not accurately reflect the activities that FTE
hours and contract dollars support, the accuracy of our results would be
compromised.
2-6
-------
CHAPTERS | PROGRAM THEORY OF CHANGE
In this chapter we consider how the Ocean Dumping Management Program is intended to
work, and how it works in practice, based on input from program staff and stakeholders.
The section begins with a brief description of the program logic model that we developed
in conjunction with program staff at the outset of the evaluation, and then describes our
findings about particular facets of the program theory and experience, including:
• Processes and activities that the program uses to achieve its outcomes;
• Assumptions about how the program's activities and outputs transition to
program outputs;
• Program information flows and feedback mechanisms;
• Allocation of program resources; and
• Alignment of non-EPA partner goals.
OCEAN DUMPING MANAGEMENT PROGRAM LOGIC MODEL
The program logic model presented in Exhibit 3-1, which was developed by the Ocean
Dumping Management Program and refined by the evaluation team, serves as a starting
point in the evaluation. It depicts key elements of the HQ program, including program
inputs, activities, outputs, customers, desired outcomes, and external factors that
influence results. The evaluation team used the information provided in the logic model
to clarify various aspects of the program's theory of change. The logic model also serves
as a template against which the program's actual experience can be compared, to learn if
there are aspects of the program that differ from the program design.
3-1
-------
EXHIBIT 3-1. OCEAN DUMPING MANAGEMENT PROGRAM LOGIC MODEL
Activities & Partners
Outputs
Customers
Short-term Outcomes
(Awareness)
Medium-term Outcomes
(Behavior)
Assist regions & USCG to
enforce regulations
Assistance w/ civil
penalties
USAGE and Permittees
/
US ACE & EPA aware of
monitoring requirements
and conditions at sites
/ \
Efficient and effective
evaluation of permit
proposals
Long-term Outcomes
(Conditions)
Develop and update
legislation (i.e. MPRSA)
Work w/ int'l orgs and
State Dept
Develop policy on
emerging issues (ie. CCS)
Develop regulations &
policies, keep current w/
Address petitions, letters.
and other inquiries
Issue special or general
(non-dredged material)
permits
Assist regions to review &
concur on dredged
material permits issues
>
\
\^
\
\
\
^
\»
/
Legislation package
(MPRSA Amendments)
International technical &
training documents
Annual report to IMO on
ocean dumping permits
Regulations & policies
Ocean dumping permits
»
\
V
^^»
\
\
\
\
\
— \—*
\ \
\\
US Congress
non-contracting parties
Gov't entities, regions.
regulated community,
env. groups, public
Regulated community.
env. groups, public
Ports and port- dependent
industries
»
>
\
\
^
/
/
Congress informed
regarding MPRSA and LP
EPA & State Dept.
communicate w/ int'l
conmunfty (LC/LP)
Stakeholders understand
role and requirements of
ODP policy
Stakeholders understand
& aware of requirements
for disposal
>
*
/
/
N.
\\,
\>
Congress amends MPRSA
and ratifies LP
waste
Stakeholders conform to
ODP regulations and
policies
Limit ocean disposal of
wastes
permit
acceptable ocean disposal
*
/
/
\
\
\
\
•
//
' /
/
int'l leadership role
acceptable
conditions resulting
from disposal
Develop guidance
documents
Assist regions in site
monitoring & designation
issues
Review regional
programs
Conduct monthly call &
semi-annual meeting for
OD coordinators
Strategic Planning
Guidance documents
Assistance with site
monitoring data, SMMPs,
site designation
Regional program review
reports
Coordinators' call notes,
meeting reports
EPA reports & planning
documents
4
7
/
Regional program offices
Division, office, OW
management, public
Increased regional
awareness of policy and
intent of policy
Mgmt aware of program
activities and direction
<
Guidance hip roves
•nplementation ofOD
management program
Consistent program
administration across
regions
Mgmt support of program
activities and budget
\
\
/
No legal
vulnerability
Coordinated &
effective OD mgmt
program
3-2
-------
Findings
PROGRAM PROCESSES AND ACTIVITIES
The program's goal of ensuring that material placed in the ocean does not negatively
impact the environment and/or human health through the implementation of MPRS A has
remained constant since the program's inception in 1972. However, since the program's
beginning, the types of materials being disposed of in the ocean have changed, shifting
primarily toward the disposal of dredged material. Over time, this shift in the types of
materials placed in the ocean has shaped the activities and emphasis of the Ocean
Dumping Management Program. The program serves five primary functions, including:
• Establishing environmental criteria/guidance for implementing MPRSA;
• Reviewing, concurring on, and issuing ocean dumping permits;5
• Designating ocean disposal sites;
• Establishing and managing Site Management and Monitoring Plans; and
• Enforcing MPRSA.
These functions are carried out by EPA HQ and Regions, as described below.
Headquarters
HQ interviewees conveyed that their core program responsibilities mirror those listed in
the Activity column of Exhibit 3-1 (e.g., dealing with statutory issues, providing guidance
to the Regions), however, funding and near-term priorities have shaped HQ activities. For
example, interviewees noted that the reduction in program funding has caused HQ to
operate in a "bare bones" fashion; staff have had to focus on addressing issues requiring
immediate attention, e.g., decommissioning the program's research and monitoring
vessel, the Bold.6 Interviewees report that some longer-term program initiatives, such as
those listed below, are not receiving adequate attention and resources:
• Updating the ocean disposal testing manual,
• Developing an amendments package for MPRSA,
• Dealing with Region-specific issues (e.g., fish waste in Region 10), and
• Conducting certain types of activities related to strategic planning activities, such
as regional reviews.
5 EPA is responsible for issuing ocean dumping permits (special, general, or emergency) for all material placed in the ocean
other than dredged material, which is permitted by the USAGE and subject to EPA review and concurrence.
5 According to one interviewee, approximately half of the Bold's activities supported monitoring related to ocean dumping.
In 2010, the Bold's survey missions included monitoring ocean dredged material disposal sites, coastal eutrophication and
toxicity assessments, monitoring ocean outfalls, assessing critical coral reef habitats, and monitoring hypoxia in the Gulf of
Mexico. Source: EPA's Ocean Survey Vessel Bold 2010 Annual Report,
http://water.epa.gov/type/oceb/assessmonitor/bold/reports.cfm
3-3
-------
Regions
EPA's seven coastal Regions (EPA Regions 1, 2, 3, 4, 6, 9, and 10) implement the Ocean
Dumping Management Program in conjunction with HQ. At the regional level,
interviewees conveyed that they primarily focus on site designation, monitoring,
compliance assistance, and stakeholder relationships. Although all the Regions work on
many aspects of Ocean Dumping—including evaluation of project proposals and scoping
documents, and development of and review of sampling/testing data— Regions
emphasize different activities depending on the needs and priorities of the Region. For
example, regional staff noted that they currently devote significant attention to the
following activities:
• Enforcement (Region 2);
• Dredging teams (Regions 1, 6);
• Beneficial use (Regions 6, 9, 10); and
• Disposal of fish waste (Region 10).
PROGRAM ASSUMPTIONS
There are several assumptions that underlie the program's theory about how activities
will lead to intended outcomes.
• Current Regulations: the program is operating with up-to-date regulations that
reflect current contaminants in material disposed of through ocean dumping.
• Adequate Information: the program has adequate information to establish
program criteria, assess the suitability of material proposed for ocean dumping,
and determine whether ocean dumping sites are meeting environmentally
acceptable conditions.
• Clear Communication: HQ provides Regions with sufficient information that
will enable them to implement the program consistently in accordance with
MPRSA, and program managers at the Office of Water understand the purpose
and function of the Ocean Dumping Management Program. For example, when
delivering program outputs, such as reports, legislative packages, and guidances,
the theory rests on the assumption that the HQ program clearly communicates
information to their customers and partners.
• Adequate Program Resources: there is sufficient funding and staff time to
support all of the activities necessary to achieve program goals.
• Program Partners Cooperate with EPA: program partners will cooperate, e.g.,
by sharing needed information, and thus help achieve program goals.
The evaluation examined the validity of these assumptions, and found that not all of them
hold true. For example, see the section "Summary of Ocean Dumping Management
Program in Practice" and Exhibit 3-3 for a discussion of examples where the assumptions
are not valid.
3-4
-------
PROGRAM INFORMATION FLOWS AND FEEDBACK MECHANISMS
According to systems theory, "missing feedback is one of the most common causes of
system malfunction. Adding or restoring information can be a powerful intervention" that
causes people to behave differently.7 Thus information flows can serve as a feedback
mechanism that helps program staff and managers adapt their activities. Regional
interviewees commented that they currently rely on the following sources of feedback to
help gather information about the Ocean Dumping Management program and its
functioning:
• Information and insights from Ocean Dumping Coordinators in the Regions;
• Interagency meetings;
• Complaints from program partners;
• Regional Dredging Team conversations; and
• Public feedback.
In addition, interviewees cited monthly conference calls for Ocean Dumping
Coordinators as a potentially useful feedback mechanism that currently lacks sufficient
focus and specificity to inform action. For example, instead of providing a brief
overview of regional updates, focus group participants suggest that the calls could focus
on specific topics or questions raised by the Regions.
Interviewees pointed out there is a need for more consistent transfer of information as to
what is working and not working in the program. For example, one interviewee said, "I
don't think we've really had time to do that. We've been in a total crisis mode for years.
The adaptation we've done is: when a crisis hits, we focus on it until the next crisis."
When survey respondents were asked to rate their level of agreement with the following
statement: "I am provided or can access information about program results that helps
inform my work," less than one-third agreed or strongly agreed with the statement (See
Exhibit 3-2).
7 Meadows, Donella. Leverage Points: Places to Intervene in a System. The Sustainability Institute, 1999.
3-5
-------
EXHIBIT 3-2. ACCESS TO INFORMATION THAT INFORMS WORK EFFORTS*
"I am provided or can access information about program
results that helps inform my work."
Agree/Strongly Agree
Disagree/Strongly Disagree
Neither Agree nor Disagree
Don't Know/Not Sure 0%
0% 5% 10% 15% 20% 25% 30% 35% 40% 45%
*ln Exhibit 3-2 and subsequent exhibits showing survey respondent agreement or disagreement
with a statement, agreement is shown in the lighter teal bar, and strong agreement (where it
occurs) is shown in the darker teal bar. Disagreement is shown in the lighter purple bar, and
strong disagreement (where it occurs) is shown with dark purple.
Summary of Ocean Dumping Management Program in Practice
We can assess how the program theory is working in practice by considering what HQ
and regional staff and stakeholders said about their experience with program
implementation. Exhibit 3-3 shows the program logic model, with three additional
features identified based on interview notes: 1) leverage points that strongly influence
program effectiveness, 2) process delays and need for improved processes, and 3)
processes that are not occurring according to interviewees. Each of these elements is
described below and illustrated in Exhibit 3-3.
Leverage Points
Systems theorists use the term "leverage points" to indicate "places within a complex
system.. .where a small shift in one thing can produce big changes in everything."8 We
identify three leverage points within the system of agencies and individuals responsible
for implementing MPRSA:
• The presence of current regulations seems to be a key leverage point that
strongly influences program effectiveness. Lack of current regulations drives a
feedback loop, whereby the USAGE does not consistently cooperate with EPA
and Regions do not have information to consistently implement MPRSA, which
undermines the credibility and effectiveness of the program. Updating the
regulations could lead to improved understanding and cooperation with USAGE
8 Meadows, Donella. Leverage Points: Places to Intervene in a System. The Sustainability Institute, 1999.
3-6
-------
and more consistent and efficient implementation of the program at the regional
level. Interviewees mentioned the following factors as impeding updates to the
regulations: concern about litigation particularly from environmental NGOs,
difficulty gaining agreement from other agencies (USAGE and NOAA) about
how to change the regulations, and concern that opening the regulations to
revision could lead to weakening of the regulations. In order to change the
current feedback loop, program managers will need to understand and address
the factors leading to the current situation.
• The presence of current guidance is another leverage point that influences
program effectiveness. As one interviewee said, "If regulatory reform is not
going to occur, then the program needs to get [its] guidance to the best place
possible so that people in the Regions can do the best job possible given the
circumstances." Lack of current guidance and training on key topics e.g., testing
guidance, fish waste, vessel disposal, and artificial reefs impedes consistent
program implementation and administration, which increases legal vulnerability
and reduces program efficiency. Interviewees mentioned the following factors
as impeding updates to the guidance: lack of updates to the regulations and
competing priorities. Updating the guidance would lead to more effective
program implementation, and could ultimately reduce resource demands by
requiring less HQ staff time to assist Regions on a case-by-case basis. One
interviewee said, "Guidance would save EPA and permittees time and money."
Another interviewee said, "Having strong consistent coordination and
implementation and good guidance would really help our program would help
new folks come on board. It would help us ensure we're being environmentally
protective across the board."
• Effective coordination across Regions is a third leverage point that influences
program effectiveness. This leverage point is shown in the logic model by the
activity of conducting a monthly call and semi-annual meeting for ocean
dumping coordinators. The program has not held regular in-person meetings in
recent years, which has reduced the Regions' ability to ensure consistent
program implementation. Interviewee experience at a recent Ocean Dumping
Coordinators meeting in Washington B.C. suggests that more regular focused
conversations between Regions and HQ can improve information flows.
Process Delays and Weed for Improved Processes
In Exhibit 3-3, yellow diamonds represent junctures where interviewees indicated that a
program process or activity is delayed or impeded. For example, HQ staff indicated that
their focus on certain near-term activities (e.g., resource reallocation) has shifted their
focus away from addressing some programmatic strategic planning issues, so a delay
symbol appears after the "Strategic planning" activity box. Interviewees also report
delays associated with developing policy on emerging issues, and assisting Regions in
site monitoring issues, which influence subsequent elements of the logic model. As
discussed in the text on leverage points, there are also delays associated with updating
guidance and a need for process improvement related to coordinating across Regions.
3-7
-------
Regional interviewees also commented that process delays can also stem from friction in
relationships with program partners. For example, although USAGE and EPA may be
"aware of monitoring requirements and conditions at sites," regional staff require
cooperation and adequate information from USAGE in order to conduct "efficient and
effective evaluation of permit proposals."
Processes not Occuring
The red stop sign symbols shown in Exhibit 3-3 indicate processes or activities that are
currently not being undertaken by the program. For example, interviewees conveyed that
regional reviews, conducted by HQ, have not been occurring. While HQ intends to
conduct the regional reviews and does allocate resources to this effort in its work plans,
other near term priorities have taken precedence. Regional personnel see the reviews as
an opportunity to communicate their concerns and feedback to HQ and generate
information that could help improve the program at the regional level. Therefore, the
absence of these reviews represents a missed opportunity to gather feedback on program
operations. In another example discussed above, regulations have not been updated, and
the fact that this process has not occurred is strongly influencing work with USAGE.
Examples of program processes not occurring, being delayed, or needing improvement
point to places in the program theory where the underlying assumptions do not hold true.
For example, the need for updated regulations and guidance suggests that the assumptions
of current regulations, adequate information, and clear communication do not fully hold
true. Instances of friction with USAGE reduce the extent to which the program has
adequate information and cooperation from program partners. Feedback from program
staff raises questions about whether the program has adequate resources.
3-8
-------
EXHIBIT 3-3. OCEAN DUMPING LOGIC MODEL REFLECTING PROGRAM LEVERAGE POINTS CONVEYED DURING INTERVIEWS
Act ivities& Partners
Outputs
Customers
policy
Develop and update
legislation (i.e. MPRSA)
Work w/ int'l orgs and
State Dept
Develop policy on
emerging issues (ie. CCS)
Develop regulations &
-o~
\
vs
^
Legislation package
(MPRSA Amendments)
International technical &
t rah rig documents
Annual report to IMO on
ocean dump rig permits
Regulations & policies
US Congress
Assist regions & USCG to
enforce regulations
Assistance w/ civil
penalties
Develop guidance
documents
Assist regions in site
monitoring & designation
issues
Review regional
programs
Conduct monthly call &
semi-annual meeting for
OD coordinators
Strategic Planning
-&•
-•*
-&*
-0-
Guidance documents (
Assistance with site
monitoring data, SMMPs,
site designation
Regional program review
reports
Coordinators' call notes, /
meeting reports
EPA reports & planning
documents
USAGE and Permittees
Regional
program offices
Leverage points that strongly influence program effectiveness
x
Response letters
Ocean dump rig permits
\ ,
\
\.
\v
Regulated community,
env. groups, public
Ports and port-dependent
industries
\
/
Stakeholders understand
& aware of requirements
for disposal
USACE & EPA aware of
monitoring requirements
and conditions at sites
Increased regional
awareness of policy and
intent of policy
Long-term Outcomes
(Conditions)
Division, office, OW
management, public
Mgmt aware of program
activities and drection
Mgmt support of program
activities and budget
Logic Model Assumptions include:
1) Current Regulations
2) Adequate Information
3) Clear Communication
4) Adequate Program Resources
5) Program Partners Cooperate with
EPA
3-9
-------
EXHIBIT 3-4.
ALLOCATION OF PROGRAM RESOURCES
To assess the current allocation of Ocean Dumping Management Program resources, the
evaluation team analyzed how resources in the program's fiscal year (FY) 2012 work
plan support the program's long-term goals.9 Exhibit 3-4 shows the percentage of Ocean
Dumping Management Program FTE hours and ORISE contract dollars10 that are
allocated to each activity in the FY 2012 work plan, based on the total number of FTE
hours and non-FTE hours (i.e. contract dollars). Based on this analysis, Exhibit 3-4
shows that the largest percentage of the program's FTE hours are allocated to activities
associated with developing policy on emerging issues (19 percent) and strategic planning
(16 percent). In terms of ORISE contract dollars, the largest percentage of these
resources is allocated to developing and updating legislation (24 percent) and developing
policy on emerging issues, e.g., carbon capture and storage (21 percent). However, it is
important to note that in practice staff do not conduct all of the activities that are allocated
staff resource in the work plan. For example, as noted earlier, regional reviews have not
occurred, even though they received an allocation of six percent of the FTE hours. Thus
intended allocation may differ somewhat from actual resource allocation.
ALLOCATION OF FTE HOURS AND CONTRACT DOLLARS APPLIED TO OCEAN
DUMPING ACTIVITIES FOR FISCAL YEAR 2012
Develop policy on emerging issues
Strategic Planning
Address petitions, letters, and other inquiries
Conduct monthly call & semi-annual meeting for OD...
Review regional programs
Assist regions in site monitoring & designation issues
Assist regions & USCG to enforce regulations
Assist regions to review & concur on dredged material...
Develop and update legislation
Develop guidance documents
Issue special or general (non-dredged material) permits
Develop regulations & policies, keep current w/ policy
Work w/ int'l orgs and State Dept
0% 6
0%
^™
^^
W
— •-
10%
* 10%
\
10%
7%
-5!°^
24%
0% 5% 10% 15% 20% 25%
• Allocation of FTE MRS Allocation of Contract Dollars
Program staff gave the evaluation team a copy of the FY2012 work plan.
10 The work allocation of ORISE fellows are considered non-FTE hours in the work plan and have been converted to contract
dollars for representation in the logic model. Contract costs associated with the OSV Bold are not included in this analysis.
3-10
-------
When the resources for specific activities are assigned to subsequent elements of the logic
model based on the connections in the model, we find that activities leading to
"environmentally acceptable conditions resulting from disposal" receive the largest
percentage of FTE hours (46%) and contract dollars (38%), while the long-term goal of
"no legal vulnerability" receives the fewest FTE hours (15%) and contract dollars (9%).n
Exhibit 3-5 shows the resource allocation for each of the four long-term program goals.
(For the details of this analysis, and a depiction of resources across the logic model, see
Appendix D.) As noted earlier, these percentages reflect intended allocations of effort,
and in practice some staff time appears to be diverted to other near term priorities.
EXHIBIT 3-5. RELATIVE PRIORITIZATION OF PROGRAM GOALS FOR FY 2012
Environmentally acceptable
conditions resulting from
disposal
Coordinated and effective
Ocean Dumping
management program
No legal vulnerability
Maintaining an international
leadership role
I Allocation of FTE
MRS
Allocation of
Contract Dollars
0% 10% 20% 30% 40% 50%
ALIGNMENT OF PARTNER GOALS
The Ocean Dumping Management Program has a variety of program partners whose
goals span from protection of natural resources to maintaining navigational waterways.
Exhibit 3-6 shows the program's key partners, which include federal, state, and local
entities, along with their goals related to ocean dumping. As noted in the introduction,
the evaluation team decided to consider EPA Regions as partners of the national Ocean
Dumping Management Program staff to the extent that they: a) contribute to the
development of priorities for the program, and b) conduct activities that contribute to
program outputs. EPA Regions are instrumental in achieving the program's goals.
1 Contract costs associated with the OSV Bold are not included in this analysis.
3-11
-------
EXHIBIT 3-6. OCEAN DUMPING PROGRAM PARTNERS AND GOALS RELATED TO OCEAN DUMPING
PROGRAM PARTNER
EPA Regions
USAGE
Clean Water Act 404 Program
Coast Guard
Department of Interior
Environmental NGOs
Fish and Wildlife Service
National Oceanic and Atmospheric
Administration
State Agencies
Navy
Ports and Harbors
PARTNER GOAL(S) RELATED TO OCEAN DUMPING
Maintaining environmentally acceptable
conditions resulting from disposal.
Maintaining navigation and meeting regulatory
responsibilities for dredged material with a
focus on cost effectiveness.
• Issuing permits for dredged material placed in
ocean waters (from baseline12 to three
nautical miles only) for a purpose other than
dumping, i.e., for the purpose of raising the
bottom elevation (e.g., beneficial reuse)
• Regulating discharge of dredged material to
US waters, including wetlands
Surveillance and enforcement of dumping sites
Overseeing issues pertaining to mineral rights,
carbon capture and storage, and platform/rig
reefing and abandonment.
Natural resource quality and protection
Ensure consistency of state water quality
standards/requirements for dumping that occurs
in state waters
Vessel Disposal (SINKEX)
Materials being dredged and/or placed in these
locations
To assess how the goals of partners outside EPA align with the goals of EPA's Ocean
Dumping Management Program, the evaluation team asked program staff to rate their
agreement with the following statement during the pre-interview survey: "The goals of
the non-EPA program partners align with the goals of the Ocean Dumping program." No
respondents agreed with the statement, and over three-quarters of respondents disagreed
or strongly disagreed with the statement (See Exhibit 3-7).
12 The United States baseline is the mean lower low water line along the coast, as shown on official United States nautical
charts. The baseline is drawn across river mouths, the opening of bays, and along the outer points of complex coastlines.
3-12
-------
EXHIBIT 3-7. ALIGNMENT OF GOALS WITH NON-EPA PROGRAM PARNTERS
"The goals of the non-EPA program partners align with the
goals of the Ocean Dumping program."
Agree/Strongly Agree 0 Vo
Disagree/Strongly Disagree
Neither Agree nor Disagree
Don't Know/Not Sure
69%
8%
4-
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
Interviewees specifically noted a disconnect with the goals of USAGE. USACE's goal is
to maintain navigational waterways while meeting regulatory responsibilities in a cost-
effective manner and with minimal delays. At times, this focus conflicts with the Ocean
Dumping Management Program's focus on ensuring adequate testing, monitoring, and
review. This has led to friction between EPA and USAGE in some Regions and Districts.
For example, in EPA Region 6, the program struggles to obtain program-related data and
information from USAGE.
Aside from USAGE, interviewees pointed out occasional overlaps in responsibilities and
potential tension with other program partners. For example, while the Ocean Dumping
Management Program, along with USAGE, oversee the disposal of dredged material, the
CWA 404 program also becomes involved when dredged material is placed in the ocean
within 0-3 miles from the baseline for a purpose other than disposal (i.e., raising the
bottom elevation). Where both MPRSA (ocean dumping) and CWA 404 programs
intersect, some interviewees questioned which program should be providing guidance and
authority. Interviewees indicated that at HQ, there is a clear delineation between the
CWA 404 and Ocean Dumping Management Program staff, but in some Regions the
same individuals may work in both programs. In another example, one interviewee noted
that U.S. Fish and Wildlife Service has shifted their regional relationship from one of
proactive collaboration to an increasing level of litigation.
3-13
-------
Conclusions
The program's activities and outputs appear well structured to lead to desired outcomes,
and the program's intended resource prioritization is appropriate and consistent with
EPA's mission, since environmental protection is the top priority. Nevertheless, several
assumptions that underlie the program theory do not hold true, and it will be important for
the program to address these challenges.
In order to operate effectively the program will need to address key leverage points (e.g.,
updating regulations and guidance, and ensuring coordination across Regions).
While some misalignment amongst agency and organizational missions is typical, tension
with USAGE is posing a serious challenge for the Ocean Dumping Management
Program, even though some Regions have a positive working relationship with USAGE.
Ensuring a more consistently productive relationship with USAGE appears important to
helping the program achieve its goals.
Recommendations
The evaluation team recommends that the Ocean Dumping Management Program:
• Update program regulations and/or guidance to improve consistency of program
implementation and potentially improve relationships with USAGE;
• Work towards more productive communication with USAGE by acknowledging
differences in mission and organizational constraints, and focusing on areas of
shared responsibility;
• Clarify and communicate to senior EPA management the importance of the
program, focusing on why this program is essential to protecting ocean
ecosystems; and
• Consider rebranding the program with a new name and new focused outreach
approaches designed to communicate to key audiences (e.g., co-regulators at
federal and state levels, permittees, and environmental groups).
3-14
-------
CHAPTER 4 | EMERGING TRENDS, OPPORTUNITIES, AND
CHALLENGES
The Ocean Dumping Management Program has encountered significant changes over the
past two decades, and faces emerging trends, opportunities, and challenges. This section
describes our findings related to: 1) historical trends and emerging issues, 2) obstacles
and delays, and 3) opportunities for improvement. At the end of the section we present
our conclusions and recommendations.
FINDINGS
HISTORICAL TRENDS AND EMERGING ISSUES
We identified four historical trends important to understanding the context of the Ocean
Dumping Management Program: 1) decreases in ocean dumping, and an increased focus
on beneficial reuse, 2) advances in science and understanding of contaminants, 3)
changes in program resources, and 4) staff turnover. Though there is some variation,
these trends are common across all of the Regions. After considering common historical
trends and emerging issues, we point out trends and issues that are specific to particular
Regions.
Decreases in Ocean Dumping, Increased Focus on Beneficial Reuse
Since the 1972 passage of MPRSA, ocean dumping of many types of substances has been
phased out (with the notable exception of material from navigational dredging, which is
discussed below). In particular, sewage sludge, which was dumped in the ocean by many
cities starting in the 1920s, was outlawed by the Ocean Dumping Ban Act, which
prohibited all ocean dumping of industrial waste and municipal sludge after December
31, 1991. The final city to cease dumping, New York City, ended the practice on June 30,
1992. On that date, the Deputy Administrator for EPA's Region 2 stated that "EPA is
committed to ... ensuring] that this problem is not just moved from the ocean to land,
but that long-term, land-based disposal options are implemented. We will also continue to
encourage the use of sludge as a benefit to the environment."13
The theme of limiting ocean dumping and instead finding new uses for materials
previously dumped in the ocean has continued, with a new focus on dredged material.
Dredged material from navigation channels is now the predominant substance disposed of
through ocean dumping, and managing this disposal has occupied a substantial portion of
the Ocean Dumping Management Program's time and resources. Between 1998 and
2009, there were 651 dredged material permits issued by USAGE following EPA review
13 U.S. EPA Region 2 Press Release, "EPA Declares End of Ocean Dumping as New York City Ceases; EPA Committed to Long-
Term Beneficial Alternatives, "June 30, 1992, http://www.epa.gov/aboutepa/history/topics/mprsa/03.html
4-1
-------
and concurrence. During this same period, an EPA general permit enabled the disposal of
50 vessels, seven emergency permits were issued by EPA, and two fish waste permits
were issued (and reissued) by EPA.14 EPA has also issued permits to the U.S. Navy and
the National Science Foundation for the transportation of target vessels and ice piers in
Antarctica, respectively.
Exhibit 4-1 shows that on a national level, there was a pronounced decline in ocean
dumping of dredged material starting in the early 2000s, with a record low in 2007,
although in recent years the national trend has started to reverse. The amount of dredged
material dumped at sea depends not only on permitting decisions by the Ocean Dumping
Management Program and USAGE, but also on the amount of dredging occurring, the
contamination of dredged material, and the funds available for land disposal or beneficial
reuse.
Dredging trends vary widely by Region and fluctuate frequently. Regions 4 and 6 have
concurred on permits for the greatest volume of ocean dumping in most years of the
program's history. Ports in these EPA Regions are currently undergoing major harbor
and channel deepening projects as they seek to accommodate post-Panamax vessels, a
very large class of ship that will begin transporting goods internationally in coming
years.15 The USAGE has identified the Southeast and Gulf coast ports as top candidates
for economically justified port expansion/dredging projects.16 One interviewee noted,
"We have a number of ports that want to deepen for the post-Panamax vessels - [these
projects] put a strain on the program, [because they require] a lot of testing to review a lot
of material."
M National Ocean Disposal Activities Excel file, dated 9-11-12, provided by EPA Ocean Dumping Staff. There are also other
activities covered by EPA permits not listed in the spreadsheet. A burial at sea permit was issued in 1977; EPA does not
report individual burial at sea activities. The National Ocean Disposal Activities Excel file does not list activities related to
transportation for target practice permits in the time period described.
15 http://www.pancanal.com/common/maritime/advisories/2009/a-02-2009.pdf
16 Institute for Water Resources, U.S. Army Corps of Engineers, "U.S. Port and Inland Waterways Modernization: Preparing for
Post-Panamax Vessels: Report Summary." June 20, 2012. http://www.iwr.usace.army.mil/index.php/us-port-and-inland-
waterways-moderinzation-strategy
4-2
-------
EXHIBIT 4-1. OCEAN DUMPING OF DREDGED MATERIAL
120,000,000
100,000,000
80,000,000
60,000,000
40,000,000
20,000,000
1976 1978 1980 1982 19841986 1988 19901992 1994 19961998 2000 2002 2004 2006 2008 2010
Source: USAGE. 2011. Ocean Disposal Database, http://el.erdc.usace.armv.mil/0cean Dumping/amount by all districts.asp
EPA has encouraged beneficial reuse or land disposal rather than ocean dumping where
feasible or necessary to protect marine habitats. USAGE reports that currently "about 20
to 30 percent of port and waterway dredged material is used for habitat creation and other
beneficial use."17 Program staff note the connection between climate change, sea level
rise, and beneficial reuse. For example, one survey respondent identified "climate
change" as one of the top two emerging trends for the Ocean Dumping Management
Program and said, "More and more, we are reviewing projects where sediments are being
proposed for beneficial reuse, as protective measures for repair of dikes and providing
sandy sediments for beaches and other coastal areas for protection of infrastructure."
Another respondent identified "alternatives to ocean disposal, especially in light of
climate change and sea level rise" as one of the top two emerging trends. These responses
highlight the extent to which EPA and its partners are increasingly considering beneficial
reuse as an alternative to ocean dumping.
Advances in Science and Understanding of Contaminants
Over the last two decades, EPA and USAGE have jointly developed and updated several
guidance documents for implementing various aspects of MPRSA. These include
document:
17 Institute for Water Resources, U.S. Army Corps of Engineers, "U.S. Port and Inland Waterways Modernization: Preparing for
Post-Panamax Vessels: Report Summary." June 20, 2012. http://www.iwr.usace.army.mil/index.php/us-port-and-inland-
waterways-modernization-strategy
4-3
-------
• For evaluating alternative disposal methods for dredged material: Evaluating
Environmental Effects of Dredged Material Management Alternatives — A
Technical Framework18 issued jointly by EPA and USAGE in 1992 and revised
in 2004. This document provides a general technical framework for evaluating
the environmental acceptability of dredged material management alternatives,
including open-water disposal (including ocean dumping), confined disposal (this
includes disposal on nearshore or upland locations), and beneficial uses. The
framework augments other existing program guidance, and is intended to foster
enhanced consistency and coordination in USACE/USEPA decisions regarding
dredged material management.
• For assessing the suitability of dredged material for ocean disposal:
Evaluation of Dredged Material Proposed for Ocean Disposal Testing Manual,19
or the Green Book, which EPA issued jointly with USAGE in February 1991.
The manual contains technical guidance for determining the suitability of
dredged material for ocean disposal through chemical, physical, and biological
evaluations. The Green Book updated the prior testing manual, the Ecological
Evaluation of Proposed Discharge of Dredged Material into Ocean Waters,
published in 1977. The 1991 update introduced atiered testing approach that
sequentially increases the amount of information collected to fully consider
biological impacts.20
• For preparing ocean dredged material disposal site management plans:
Guidance Document for Development of Site Management Plans for Ocean
Dredged Material Disposal Sites developed jointly by EPA and USAGE in 1996.
This document lays out a recommended framework for site management plan
development and content.
EPA and USAGE have provided several updates to these guidance documents. For
example, in the 1990s, EPA made changes to its bioaccumulation testing procedures that
had a substantial impact on the amount of material that could be placed in a major ocean
dumping site off of the Port of New York and New Jersey. These changes resulted in re-
designating the location as a remediation site that would accept only uncontaminated
sediment.21 Between 2000 and 2003, EPA tightened its standards for substances
contaminated with polychlorinated biphenyls, or PCBs, permitted for ocean dumping in
the same EPA Region. These changes in guidance and standards were the subject of (and
some say a product of) a series of lawsuits initiated by Clean Ocean Action and other
environmental advocacy groups pushing for more rigorous bioaccumulation testing
procedures, and a lawsuit from the United States Gypsum Company arguing for EPA to
http://water.epa.gov/type/oceb/oceandumping/dredgedmaterial/upload/2004_08_20_oceans_regulatory_ dumpdredged_
f ramework_techf ramework. pdf
19 http://water.epa.gov/type/oceb/oceandumping/dredgedmaterial/upload/gbook.pdf
20 Burroughs, Richard and Christine Santora, "Disposal of Contaminated Sediment from The Port of New York," University of
Rhode Island Transportation Center, June 2004.
21 Ibid.
4-4
-------
maintain its current PCB standards for the purpose of its permit application. The complex
history of these lawsuits and related negotiations, public involvement process, and
involvement of Congressional delegations and Vice President Gore is beyond the scope
of the current document, but is described in existing literature.22
The most recent update made to the ocean dumping guidance was the 2004 framework
for evaluating alternative disposal methods for dredged material. The underlying
legislation (MPRSA) was last updated in 1992 by the Water Resources Development Act
of 1992 (Public Law 102-580).23 As noted earlier, in FY2012, the Ocean Dumping
Management Program prioritized activities including strategic planning and developing
policy on emerging issues; in contrast the activity of developing guidance documents was
the focus for only three percent of total FTE hours (see Exhibit 3-4).
Overall, program staff and stakeholders expressed concern that the regulations and
guidance documents have not kept pace with advances in science and therefore need to be
updated. Several program staff mentioned outdated regulations, new contaminants, and
the need to revise testing and assessment methods as one of the top two emerging trends
for the Ocean Dumping program. For example, one survey respondent discussed
"emerging contaminants," explaining that "Pyrethroids are coming to the forefront in
various watersheds ... we are beginning to look into this class of pesticides as a source of
acute and chronic toxicity, but we will soon need guidance from HQ and ORD [the EPA
Office of Research and Development]. Similarly, pharmaceuticals may be implicated in
contributing to chronic toxicity in sediments." Another respondent said that a top
challenge is "sediment testing improvements (including ocean dumping regulation
revisions): how/when to incorporate emerging contaminants, combining the Ocean and
Inland Testing Manuals, etc." For example, one interviewee said, "We know so much
more about contamination and carcinogens nowadays, but we don't sample for them
because the 1970's regulations don't [cover] them."
A new class of substances related to climate change that may be proposed for ocean
dumping, including sub-seabed carbon sequestration and geo-engineering (e.g., ocean
fertilization), constitutes an important emerging issue that will likely receive increased
public attention overtime, particularly in light of their potential for environmental
impacts. For example, Canada's environmental ministry is currently investigating an
incident involving the Haida Salmon Restoration Corporation and of a California
businessman who reportedly dumped iron dust approximately 200 nautical miles of the
coast of British Columbia.24 The iron sparked the growth of plankton, which was
intended to aid the recovery of the local salmon fishery for the native Haida as well as to
sequester carbon, because plankton absorbs carbon dioxide and settles deep in the ocean
22 For a detailed history, see Burroughs, Richard and Christine Santora, "Disposal of Contaminated Sediment from The Port of
New York," University of Rhode Island Transportation Center, June 2004. Additional analysis of the case is included in
Brown, N. and R.W. Knecht. 1998. "The New York/New Jersey Harbor dredging conflict." In J.R. Waldman, W.C. Nieder
(eds.), Final Reports of the Tibor T. Polgar Fellowship Program, 1997. Hudson River Foundation.
23 http://www.epa.gov/owow/oceans/ndt/publications/pdf71996_smmp_guidance.pdf
24 Fountain, Henry, "A Rouge Climate Experiment has Ocean Experts Outraged." New York Times, section A1, Friday October
19, 2012.
4-5
-------
when it dies.25 This issue was raised at the 29 October -2 November 2012 London
Convention/London Protocol meeting where the Parties to the London
Convention/London Protocol issued a statement of concern regarding this ocean
fertilization activity. The EPA Ocean Dumping Management Program could be called
upon to prevent or enforce against similar activities off the U.S. coast in the future.
Changes in Program Resources
As funding for OWOW and the Office of Water overall have increased, funding for the
Marine Pollution Control Branch, which includes the Ocean Dumping Management
Program, has declined 14 percent in the last six years (Exhibit 4-2). During the same
period, while the number of HQ OWOW staff FTEs allocated to the Ocean Dumping
Management Program has more than doubled, the overall FTE allocated to the HQ
program remains relatively small (less than 4 FTE) (Exhibit 4-3). There is broad
consensus among regional and HQ program staff interviewed that they are being asked to
do "more with less," and that they have fewer resources to do their jobs. Ten of 13
survey respondents cited reduced or insufficient funding as a top emerging issue or
challenge for the program. One survey respondent said a top challenge for the program is
"limited resources and a continually growing portfolio (a broad array of issues)."
Program staff have been actively discussing EPA's disinvestment in the Ocean Survey
Vessel Bold and its implications for the Ocean Dumping Management Program. Ten of
13 survey respondents identified the loss of the Bold or maintaining sufficient funding for
monitoring as one of the top two emerging issues or challenges for the program.
Interviewees from several regional offices, particularly those on the east coast, were
concerned about the impact of the loss of the Bold on monitoring. One interviewee
explained that the vessel was the program's most important monitoring tool and its only
solid asset. Interviewees worried that the loss of the ship would open the door to future
budget cuts. As one interviewee explained, "One of the things about having a ship is that
you can't cut [the budget] below $5 million. [But] once you get rid of it you can cut it
whenever you want." Another interviewee remarked, "We need to be developing a
strategy for that [monitoring] money [to replace the Bold] so we don't lose it, so it's
predictable, [and] so we can plan." Other interviewees expressed concern about the
monitoring work the Bold used to do for programs other than the Ocean Dumping
Management Program (e.g., related to corals and Total Maximum Daily Load
monitoring), which will no longer be supported. On the other hand, some interviewees
thought that the disinvestment from the Bold could bring more equity in regional
monitoring resources. One interviewee said "With the east coast having the majority use
of the vessel, their approach to monitoring [has been] really different than ours.... But
now there's an opportunity to have all the Regions look at how ocean disposal sites are
managed and monitored in a consistent manner."
25 Fountain, Henry, "A Rouge Climate Experiment has Ocean Experts Outraged." New York Times, section A1, Friday October
19, 2012.
4-6
-------
Program managers note that budget cutting exercises take time in and of themselves,
which could be better directed toward substantive planning for the program. For
example, one interviewee said, "We have spent a lot of time on budgeting exercises for
the last year and a half... with all of those resources we could have done things that
would have been more important. Rather than working on the budget level allocation, we
should be working on planning [for the future]... for example how to meet required
monitoring levels [and] how to even out resources between the Regions."
Program staff report they are concerned that they are less involved in resource decisions
than they have been in the past. One interviewee said, "Before the last two or so years,
program staff across Regions had a bottom-up style of communication that fostered team
decision-making and a stewardship approach to work. For example, if a budget cut had to
made, we would all talk it out until we could come to some understanding of how best to
proceed. Now, we have managers at the next level above us making decisions that we
don't necessarily agree with, and they inform us of those decision through rather poor
messaging. ... There is a lack of transparency." One manager from the Office of Water
reports that the Agency as a whole is under severe budget constraints, and that the Ocean
Dumping Management Program needs to clearly communicate what it does and the value
of the program to the EPA mission.
4-7
-------
EXHIBIT 4-2. RESOURCE TRENDS WITHIN THE OFFICE OF WATER (DOLLARS IN THOUSANDS)'
PROGRAM
- RELATED GOAL(S)
Office of Water
- Clean and Safe Water/
Protecting America's Waters
Office of Wetlands, Oceans,
and Watersheds
Protect Water Quality
Marine Pollution
PERCENT
CHANGE OVER
SIX YEAR
PERIOD
21%
55%
(14%)
FY2008
ACTUAL
BUDGET
$3,119,201.2
$1,658,310.4
$13,430.4
FY2009
ACTUAL
BUDGET
$8,887,323.4
$5,538,892.8
$13,064.7
FY2010
ACTUAL
BUDGET
$4,989,963.6
$3,375,542.5
$9,783.7
FY2011
ACTUAL
BUDGET
$5,085,863.7
$3,553,462.7
$15,570.5
FY2012
ENACTED
BUDGET
$4,094,452.5
$2,798,913.5
$12,898.0
FY2013
PRESIDENT'S
BUDGET
$3,782,228.0
$2,565,462.0
$11,587.0
T Source: EPA Congressional Justifications, available at http://www.epa.gov/planandbudget/archive.html. Includes contract costs associated with the OSV Bold.
EXHIBIT 4-3. RESOURCE TRENDS WITHIN THE OFFICE OF WATER (ANNUALIZED FULL-TIME EQUIVALENTS)'
PROGRAM
RELATED GOAL(S)
Office of Water
- Clean and Safe Water/
Protecting America's Waters
Marine Pollution
EPA HQ Ocean
Dumping Management
Program*
OWOW FIE
PERCENT
CHANGE OVER
SIX YEAR
PERIOD
21 .4%
2.1%
128%
FY2008
ACTUAL
BUDGET
2,815.1
42.8
1.71
FY2009
ACTUAL
BUDGET
2,868.3
44
1.30
FY2010
ACTUAL
BUDGET
3,471.3
41.5
1.00
FY2011
ACTUAL
BUDGET
3,510.3
44.5
2.5
FY2012
ENACTED
BUDGET
3,423.6
43.7
3.6
FY2013
BUDGET
3,418.9
43.7
3.9
Source: EPA Congressional Justifications, available at http://www.epa.gov/planandbudget/archive.html
* Source: Work plans for FY 08 through FY 13 provided by Ocean Dumping Management Program staff. Where possible, values taken from subsequent year's work plan (to
more closely account for FTEs actually used, not just planned).
4-8
-------
Staff Turnover
The Ocean Dumping Management Program has experienced considerable staff turnover
in recent years. One interviewee noted that three of the seven Ocean Dumping Regions
have lost staff with long tenures over the last seven years. Another interviewee noted that
"It's hard to know what to do and how to do it in this program unless you've been in it a
long time. We need checklists... and other documentation to get people up to speed
quickly." Loss of institutional knowledge about the Ocean Dumping Management
Program's processes and procedures may lead to inefficiencies in permitting, monitoring,
and other activities.
Trends and Emerging Issues Specific to EPA Regions
In addition to common historical trends and emerging issues cited by respondents, several
issues were cited by only one or two Regions as a high priority in their jurisdiction.
These issues include:
• Restoring the Gulf of Mexico, including post British Petroleum Deepwater
Horizon work in the Gulf of Mexico related to Ocean Dredged Material Disposal
Sites and harbors;
• Managing major harbor and channel deepening projects in the southeastern
US. As noted earlier, ports in this area are expected to undergo major expansion
in coming years to accommodate post-Panamax vessels;
• Ensuring proper disposal of debris from the Japanese tsunami on the west
coast, and avoiding introduction of invasive species on the west coast;
• Ensuring proper disposal of fish waste in Alaska and on the west coast; and
• Protecting sensitive near shore habitats (e.g., reefs) in the Caribbean from
effects of dredged material transport.
Survey respondents are relatively evenly divided over whether HQ and Regions face the
same emerging issues and challenges: just over half (54 percent) of survey respondents
report that EPA HQ and Regions face the same emerging issues, and a slightly smaller
share of survey respondents (46 percent) agree that EPA HQ and Regions face the same
challenges (see Exhibits 4-4 and 4-5). The survey responses do not suggest any notable
patterns of agreement or disagreement based on the Region of the country or the tenure of
staff responding.
4-9
-------
EXHIBIT 4-4. SHARED HEADQUARTERS AND REGIONAL EMERGING ISSUES
"Headquarters and regional staff face the same emerging
issues."
Agree/Strongly Agree
Disagree/Strongly Disagree
Neither Agree nor Disagree 0'/0
Don't Know/Not Sure
8%
0% 10% 20% 30% 40% 50% 60%
EXHIBIT 4-5. SHARED HEADQUARTERS AND REGIONAL CHALLENGES
"Headquarters and regional staff face the same program
challenges."
Agree/Strongly Agree
Disagree/Strongly Disagree
Neither Agree nor Disagree 8%
Don't Know/Not Sure
0% 10% 20% 30% 40% 50%
4-10
-------
OBSTACLES AND DELAYS
Findings from this evaluation suggest that the Ocean Dumping Management Program has
faced obstacles and delays in recent years. Eighty-four percent of program staff surveyed
agree or strongly with the statement that "Over the past three years the Ocean Dumping
Program has encountered significant obstacles and/or delays to accomplishing its
mission" (Exhibit 4-6). Obstacles cited include:
• Lack of adequate time and resources;
• Lack of clear priorities and "mission creep;"
• Limited senior management direction/feedback;
• Administrative "fire drills;"
• Loss of the Bold and associated increases in administration time to find other
vessels to serve this function;
• Lack of funding for monitoring Ocean Dredged Material Disposal Sites so effects
to marine species are not adequately understood;
• USAGE disagreement or inflexibility and lack of communication with some
USAGE Districts;
• Lack of cooperation between federal and state agencies, despite best efforts of
Regional Dredging Team; and
• Threat of lawsuits.
EXHIBIT 4-6. ENCOUNTERING OBSTACLES AND DELAYS
"Over the past three years the Ocean Dumping program has
encountered significant obstacles and/or delays to
accomplishing its mission."
Agree/Strongly Agree
Disagree/Strongly Disagree 0'/0
Neutral
Don't Know/Not Sure 0 Vo
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
4-11
-------
OPPORTUNITIES FOR IMPROVEMENT
Program managers seeking to improve the Ocean Dumping Management Program will
need to anticipate emerging issues, challenges, and opportunities. Survey results suggest
that to date, the Ocean Dumping Management Program has done a better job of
anticipating emerging issues compared to anticipating challenges and opportunities, but
in general the program has room to improve in both of these areas. Specifically, only 31
percent of respondents agreed or strongly agreed with the statement "Over the past three
years, the Ocean Dumping program as a whole has done a good job of anticipating
emerging issues" (roughly two-thirds of respondents disagreed with this statement or did
not express an opinion) (see Exhibit 4-7). Only 15 percent of respondents agreed with
the statement "Over the past three years, the Ocean Dumping program as a whole has
done a good job of anticipating challenges and opportunities," and no respondents
strongly agreed (see Exhibit 4-8). These findings suggest that the Ocean Dumping
program needs a renewed focus on strategic planning, anticipating emerging challenges,
and seizing potential opportunities. In the following sections, we highlight three areas of
opportunity for the Ocean Dumping program to focus on improvements: 1) exercising
greater coordination and leadership from EPA HQ, 2) updating regulations and guidance,
and 3) supporting London Protocol ratification.
EXHIBIT 4-7. ANTICIPATING EMERGING ISSUES
"Over the past three years the Ocean Dumping program as a
whole has done a good job of anticipating emerging issues."
Agree/Strongly Agree
Disagree/Strongly Disagree
Neither Agree nor Disagree
Don't Know/Not Sure 8%
0% 5% 10% 15% 20% 25% 30% 35%
4-12
-------
EXHIBIT 4-8. ANTICIPATING EMERGING CHALLENGES AND OPPORTUNITIES
"Over the past three years the Ocean Dumping program as a
whole has done a good job of anticipating challenges and
opportunities."
Agree/Strongly Agree
Disagree/Strongly Disagree
Neither Agree nor Disagree
Don't Know/Not Sure
0% 10% 20% 30% 40% 50% 60%
Opportunity to Exercise Greater Coordination and Leadership from EPA HQ
Interviewees and focus group participants suggest that, particularly in light of the recent
staff turnover, EPA HQ could provide more leadership, facilitation, and coordination for
the Regions. One survey respondent reported that a major challenge for the program was
the "lack of consistency and standard operating procedures between Regions." Regions
seek a centralized repository of identified checklists, resource lists, and other resources.
Participants in the Ocean Dumping Coordinators meeting acknowledged that the creation
of training documentation may represent a large investment of time and effort in the short
term but is a necessary step to address staff turnover and foster the long term resilience of
the program.
Staff praised EPA HQ's recent convening of an Ocean Dumping Coordinators meeting in
Washington, D.C. as an opportunity to foster coordination across the Regions.
Participants at the meeting acknowledged that regular in-person meetings might not be
possible due to resource constraints, but suggested that EPA HQ seek creative means to
gather Ocean Dumping coordinators from across the country in focused conversations
(e.g., through facilitated conference calls and/or meetings using online document sharing
and participation software such as GotoMeeting, WebEx, or Microsoft 365). Meeting
participants also acknowledged that EPA HQ does hold regular conference calls, but
pointed out that they tend to be an opportunity for updates, rather than focusing on a
particular topic of concern.
4-13
-------
In addition to coordination across Regions, staff seek greater leadership from EPA HQ.
During the strengths, weaknesses, opportunities, and threats analysis at the Ocean
Dumping Coordinators meeting, participants identified an absence of leadership as a key
threat or weakness for the Ocean Dumping Management Program. One interviewee said,
"There is a national consistency issue: [we need to] get HQ on the same page [and get]
information coming down from the top telling [USAGE] Districts what they need to do."
Another interviewee reported, "We have five or six voices telling you how to do things.
[There is] unclear program direction [and a] lack of a unified voice from leadership."
Opportunity to Update Regulations and Guidance
Although there is broad-based agreement among Ocean Dumping Management Program
staff and program stakeholders that it is important to update the regulations and program
guidance, several barriers make this task difficult. First, the regulations themselves are
quite detailed and need to be at least partially updated, if the program is to fully update
the technical guidance. Moreover, stakeholders do not agree on how to update the
regulations. One interviewee explained that the program's technical guidance cannot be
changed because it was codified into the original regulations. Another challenge is that
current resource constraints and demands on program staff time have taken focus away
from efforts to update the regulations and guidance. As noted earlier, the program has
updated its testing guidance in the past, which was the subject of litigation. One of the
four long term outcomes that the program says it is trying to achieve is "no legal
vulnerability." One interviewee suggested that the program's emphasis on avoiding
litigation has "stymied program creativity." Another interviewee remarked that "It would
save everyone a lot of work to update the regulations because they are not working," but
that the "risk of changing the regulations is to EPA, not USAGE," presumably because
once open for negotiation the regulations could become less protective of the marine
environment.
The fact that the Ocean Dumping Management Program has not updated the guidance in
recent years is straining relations with USAGE and making program implementation
more challenging for the Regions. The Ocean Dumping Management Program is
planning to update the MPRSA regulations in conjunction with the London Protocol
Ratification process, which could have the added benefit of potentially improving
relations with USAGE. The London Protocol Ratification process is described in the next
section.
Opportunity Related to London Protocol Ratification
The London Convention, which the U.S. ratified in 1975, covers the deliberate disposal
of wastes or other matter from vessels, aircraft, platforms, and other man-made structures
at sea. In 1992, the 80 Parties to the London Convention began a comprehensive review
of the Convention, resulting in a new treaty called the 1996 Protocol.26 The London
Protocol, which the U.S. has signed but not yet ratified, updates and improves upon the
London Convention. Currently, some Parties to the London Protocol are interested in
6 http://water.epa.gov/type/oceb/oceandumping/dredgedmaterial/londonconvention.cfm
4-14
-------
developing a binding regulation to address marine geoengineering activities, including
ocean fertilization, through an amendment to the London Protocol.
The United States signed the 1996 London Protocol in 1998. Since then, the United
States has been working on the ratification process via implementing legislation
(amendments to the MPRSA) and seeking necessary action by the Senate. In 2007, the
White House submitted the London Protocol to the U.S. Senate for its advice and
consent. The London Protocol is on the Obama Administration's treaty priority list. The
proposed implementing legislation submitted to Congress in 2008 is undergoing review
prior to re-submittal to Congress. The Ocean Dumping Management Program is currently
reviewing the MPRSA legislation package submitted to Congress in 2008 to align with
the requirements of the London Protocol. One interviewee noted that, "Updating MPRSA
would allow us .. .to fix a few things to be more efficient for how we implement the
Ocean Dumping Management Program." For example, one interviewee explained that
"right now [the program] can only issue an emergency permit if it's a threat to human
health. We don't have the authority to issue permits to deal with an emergency if the
threat is to environmental health."
If the MPRSA is revised, the Ocean Dumping Management Program will also have an
opportunity to adjust the scope of the regulation. For example, the program may be
called upon to help mitigate emerging issues like ocean acidification and climate change
(e.g., ocean fertilization). Interviewees disagreed about whether the program should
expand its scope to address emerging ocean problems, such as acidification, sea level
rise, and carbon capture. Some program staff felt that the program should expand its
purview beyond dumping to address these issues. For example, one regional interviewee
suggested that EPA is likely "missing out" on helping with issues such as "ocean
acidification, invasive species, and marine debris." A survey respondent remarked "[The
program] need[s] to expand [the] scope of [the] program beyond just ocean dumping; we
need to be looking at other ocean issues, and there are a lot of them." Other staff felt that
expanding the scope of the program would be a distraction for a program with a narrowly
focused role and a lack of resources. Some interviewees expressed skepticism about the
viability of carbon sequestration in deep ocean waters. Considering which of these issues
to include in its scope will be a major strategic decision for the Ocean Dumping
Management Program going forward.
CONCLUSIONS
The Ocean Dumping Management Program has been challenged to clearly articulate the
scope and benefits of the program, given shifts in the types of materials proposed for
ocean disposal, multiple agencies involved in ocean dumping permitting, and reduced
public attention to environmental and human health risks of ocean disposal.
The changing nature of contaminants and advances in monitoring and testing
technologies require updates to EPA requirements and guidance. While the Ocean
Dumping Management Program has jointly developed and updated several guidance
4-15
-------
documents for implementing various aspects of MPRSA with USAGE, the differences in
perspective between the two agencies, multiple program priorities, and limits to EPA
staff resources have constrained EPA's ability to keep pace with needed updates.
The Ocean Dumping Management Program is working with a very small staff at HQ, and
is challenged to provide effective coordination, leadership, and guidance to the Regions.
Staff turnover and associated loss of institutional knowledge, insufficient guidance and
documentation, limited opportunities for coordination across Regions, and varying levels
of cooperation from USAGE have hampered consistent implementation of the program
across the country.
RECOMMENDATIONS
The evaluation team recommends that the Ocean Dumping Management Program:
• Update program guidance and use the London Protocol ratification process as an
opportunity to update MPRSA;
• Identify approaches to take on the additional responsibilities that will come with
ratification;
• As a part of updating program guidance on regulations, work to improve
communications to Regions about how to implement the regulation, standard
procedures, and expectations for program partners (e.g., USAGE Districts);
• Foster communication and consistency across Regions; and
• Seek to foster improved communication and partnership with USAGE.
4-16
-------
CHAPTER 5 | RESOURCE ALLOCATION
Given current resource trends and a broadening portfolio of work, the Ocean Dumping
Management Program seeks to identify opportunities for improving the alignment
between resource deployment and the program's intended or desired outcomes. We
group our findings into four main topics: 1) resources that are required vs. those that can
be reallocated, 2) opportunities to improve alignment of resources with intended program
outcomes, 3) opportunities to improve alignment of resources across Regions, and 4)
opportunities, risks, and impacts of resource reallocation. Conclusions and
recommendations follow the findings.
FINDINGS
RESOURCES REQUIRED VS. THOSE THAT CAN BE REALLOCATED
Most individuals we interviewed believe that the program is operating with minimal
resources and that there is no opportunity to be gained by moving resources out of the
program. For example, one interviewee said, "This is a program that in my view never
had any 'fat.' This program is running on empty. It's really hard for me to think what
they could focus their resources on better; they're only doing the absolute core essential
elements of the program because that's all they can do." Another interviewee said, "Can
the program be cut more and still meet the regulatory requirements? There are certain
aspects of [the program] that really aren't discretionary; when EPA looks at
disinvestment, [it should] look at whether this is something some other agency is doing,
[and whether the Ocean Dumping Management Program] has authorities that no one else
has. [Resource reductions are] cutting into the bones of the agency we're supposed to
keep alive." Interviewees point out that the Ocean Dumping Management Program has
authority for enforcement, permits (specifically concurring on dredged material ocean
dumping permits and issuing ocean dumping permits for all other substances), site
management, and site monitoring, and that EPA cannot meet these responsibilities with
fewer resources. Only one interviewee (outside of EPA) suggested that the program had
resources to spare: this person said "EPA's 35-person staff is still the same size as it was
back when everything (not just dredged material) was dumped in the ocean." Program
FTE data shown in Exhibit 4-3 suggests that this size staff more likely relates to the
Marine Pollution Control Branch as a whole, not just the Ocean Dumping Management
Program.
OPPORTUNITIES TO IMPROVE ALIGNMENT OF RESOURCES WITH INTENDED
PROGRAM OUTCOMES
While not explicitly addressing opportunities to improve resource alignment,
interviewees did suggest situations where resource alignment is not currently optimized.
5-1
-------
For example, interviewees generally agree that the program could do a better job of
focusing its efforts and working more efficiently. Some interviewees suggested that
program staff were trying to work on too many disparate issues and that they needed to
remain focused on core regulatory requirements. For example, one regional staff person
said, "We have suffered from mission creep ... HQ requires Regions to help enforce
ocean dumping [requirements] but also help with the policy for the oceans27 ... But we
only get FTEs for the dumping portion. This [mismatch] in resource allocation and
duties presents confusion in the program."
Some interviewees suggested that the program is operating in a reactive mode because it
is so under-resourced that it does not have the systems and procedures in place to
anticipate and manage workloads. One person said, "We're just trying to hold the key
things together and deal with administrative [requirements], political directives,
emergencies, and prepare for international meetings [related to the London
Protocol]... .We respond to emergencies well, but what happens is a lot of other stuff
languishes... .It would be great just to finish the guidance on the testing manual, but
we've made slow progress. We don't have the resources to get over the hump."
Interviewees also noted that lack of resources and attention to documenting procedures
and ensuring consistency between Regions (e.g., establishing consistent site monitoring
frequencies) leads to inefficiencies in work. The lack of effective manuals and guidance
leads to increased time and resources needed to bring new staff up to speed when they
replace experienced staff. This has been a particular challenge in Regions that have
historically experienced management challenges and/or significant staff turnover. For
example, one Region is working to create electronic filing systems, standard operating
procedures, updated research, and a new understanding with USAGE about what is
required under the regulations. This renewed effort has required an investment of
resources not only within the Region but also at EPA HQ.
OPPORTUNITIES TO IMPROVE ALIGNMENT OF RESOURCES ACROSS REGIONS
Several interviewees suggested changes to how resources are allocated across Regions,
though there was no clear consensus on how to improve this allocation. Two
interviewees said that resources should not be allocated equally between the Regions, but
rather should be based on the number of ocean dumping sites within the Regions.
However, another interviewee cautioned that this sort of pro-rated allocation had been
tried in the past, and that it had led to some Regions that received relatively more funds
redirecting funds away from ocean dumping to other activities.
A few interviewees suggested that there is a minimum level of resources required for
monitoring, and that each Region should be allocated sufficient resources to conduct
required monitoring. One interviewee (in the east) said, "What we've had to do is bare
bones monitoring. We have adapted how we are monitoring sites by reducing the
number of samples and replicates, and instead of doing everything the regulations say, we
27 The interviewee most likely was referring to the National Ocean Policy described at
http://www.whitehouse.gov/administration/eop/oceans/policy
5-2
-------
have pulled back to the absolute minimum where we could still get useful information.
At some point, data is so minimal it is not useful." On the other hand, another
interviewee said, "the east coast has had the [Bold] vessel available and has had an
incentive to keep it busy, so they've done extra work, while we can't even do the basics
on the west coast. There needs to be a nationally recognized minimum monitoring
policy." Interviewees point out the importance of monitoring to protect human health
and the environment. For example, one regional representative noted that monitoring is
important to understand how activities during disposal affect marine environment.
Another regional representative said, "Site monitoring is a feedback mechanism.
[Monitoring allows us to] look for trends to see whether testing and management is
adequate; without the monitoring results, we wouldn't be able to make changes in how
materials are placed [in the ocean to protect the marine environment.]" Another
interviewee pointed out that ensuring adequate monitoring frequency is necessary to
ensure that EPA is meeting its regulatory obligations and that the Agency is not
vulnerable from a legal perspective. Until the minimum requirements for monitoring are
agreed on and Regions determine how they will conduct monitoring without the Bold
vessel, it remains unclear what are the minimum dollar resources Regions will need to
contract out adequate monitoring.
With regard to allocation of staff across Regions, the current allocation is based on the
historical number of ocean dumping sites in each Region. One interviewee argued this
no longer makes sense. However, this interviewee said "it doesn't matter because
Regions allocate staff the way they want to anyway... .and HQ does the same thing.... a
lot of staff who have nothing to do with ocean dumping are officially 'Ocean Dumping
FTEs.'" One interviewee said that each Region should have a minimum of two staff
persons, to allow for backup/redundancy in case one person leaves. In a group discussion
among program staff, participants suggested that the Program assess the FTE allocation
across regional offices and revisit the workload model from the 1990s.
OPPORTUNITIES, RISKS, AND IMPACTS OF RESOURCE REALLOCATION
As noted earlier, the Ocean Dumping program has recently undergone a major resource
reallocation, namely disinvestment from the ocean monitoring vessel, the Bold. While
several Regions lamented the loss of the vessel, others felt it would lead to an appropriate
reallocation of resources. In addition, interviewees pointed out that the act of
disinvestment itself has taken resources and has been a considerable distraction for the
program in recent years. Some interviewees expressed concern that disinvestment from
the Bold will actually end up costing EPA more resources in the long term.
CONCLUSIONS
The findings suggest that the Ocean Dumping Management Program does have potential
to improve alignment of its resources with intended outcomes. We conclude that the
program managers could help align resources by confirming the program's priorities,
5-3
-------
articulating the minimum resources needed to meet those priorities at both the HQ and
regional level, and requesting support from senior leadership to focus the program's
limited resources on those priorities. Findings from interviews suggest that the HQ
program is vitally important to coordinate, guide, and ensure consistency across Regions.
In addition, HQ is essential to representing the Agency in cross-agency efforts, e.g., the
interagency workgroup on the London Convention. In our opinion, these areas of activity
should be the primary focus of the Ocean Dumping Management Program.
Whenever resources are shifted, there are likely to be "winners" and "losers," and it is
important to be sure that the change provides enough benefits to be worth the friction and
distraction of implementing the change. Program staff and managers have not developed
a shared view of the minimum requirements that HQ and the Regions must fulfill, and the
resources needed to complete those tasks. Without such an agreement, it is difficult to
pinpoint opportune areas for resource reallocation.
It is not clear that the program has adequate resources to address its core functions, take
on new emerging issues, and cope with the transaction costs associated with budget
restructuring. To operate with existing resources, the program may need to prioritize its
activities with a focus on core program functions (e.g., providing guidance to the
Regions), while devoting less attention to EPA initiatives that are important but not
directly related to the program's mission (e.g., administrative work associated with
issuing monitoring grants in lieu of the Bold, and coordinating with other agencies on
carbon capture and storage). On the other hand, in order to address emerging issues (e.g.,
related to climate change) the program may need more resources.
RECOMMENDATIONS
The evaluation team recommends that the Ocean Dumping Management Program take
the following steps:
• Assess resource allocations for Regions going forward: Define the minimum
requirements that the Regions must fulfill. Research what future program
activity levels (e.g., permit review, site designation activities) can be anticipated
in each Region, given trends in ocean dumping and port development. Analyze
what resources will be required in each Region to meet minimum program
requirements given anticipated activity levels. Finally, consider resource
reallocation across Regions if necessary to ensure a balance between effort
required and resources provided.
• Assess minimum resource requirements for the HQ program: given work on
the London Protocol and the need to update the regulations and provide
additional guidance to Regions, determine what resources will be needed to
adequately carry out core program activities.
5-4
-------
CHAPTER 6 | PERFORMANCE MEASURES
Performance measures provide essential feedback to assess program performance and to
inform ongoing program learning and evaluation. There are a number of factors which
should inform the design of performance measures for any program. These factors
include, for example, the goals of the program, the aspects of the program over which
program managers exert some control or influence, and the availability of reliable and
meaningful data. Programs may develop a
suite of performance measures that include
both output and outcome measures intended
to provide a comprehensive picture of
program results. Not all aspects of a
program can be measured; the best measures
provide an indicator of program
effectiveness (See Exhibit 6-1 for additional
characteristics of effective program
measures.)
The Ocean Dumping Management Program
seeks to measure its success in a complex
environment. First, the program is trying to
achieve goals that have been articulated in
several different ways and the definition of
the goals is currently evolving.
Traditionally, the program has articulated its
goals in terms of implementing the MPRSA
and the London Convention. The FY13
Congressional Justification describes the
program's major areas of effort as
"Designating, monitoring, and managing
ocean dumping sites and implementing
provisions of the National Dredging Policy." As part of the draft "EPA Operational
Priorities for FY13 Ocean and Coastal Protection" the Ocean Dumping Management
Program is helping define an operational priority for ocean dumping, which addresses
monitoring of ocean disposal sites to ensure protection of environmental health,
appropriate testing of material to be disposed, and encouraging beneficial reuse of
materials. The program is also considering a process of engaging the users of ocean
disposal sites in financing site designation and management costs.
EXHIBIT 6-1. CHARACTERISTICS OF
EFFECTIVE PERFORMANCE MEASURES
Data are:
• Available
• Measurable
• Verifiable
Measures are:
• Timely and actionable
Responsive to the needs of
stakeholders (e.g., staff, managers,
public, flinders)
Comprehensible and credible to
stakeholders
Motivating
Comparable across jurisdictions
and programs, and over time
6-1
-------
Program staff say they are working to minimize the amount of substances placed into the
ocean and to ensure that any matter that is dumped is tested appropriately, meets
applicable standards, and is managed and monitored appropriately. However, the
program has no direct influence over the amount of material that is dumped; this is
largely dependent on dredging projects overseen by USAGE and influenced by economic
conditions and priorities, as well as the amount of sedimentation in navigation channels.
While the Ocean Dumping Management Program seeks to encourage beneficial reuse
where possible, this approach is often logistically or economically infeasible.28 In the
remainder of this chapter we present our findings regarding the current performance
measures in use, strengths and weaknesses of the current measures, and potential alternate
measures, along with our conclusions and recommendations.
FINDINGS
CURRENT MEASURES
Overall, the Ocean Dumping program focuses on measuring outcomes, namely the
percentage of Ocean Dumping sites which meet specified conditions. Specifically, the
Ocean Dumping program has a strategic measure which states that, "By 2015, 95 percent
of active dredged material Ocean Dumping sites, as determined by 3-year average, will
have achieved environmentally acceptable conditions (as reflected in each site's
management plan and measured through onsite monitoring programs)."29 (See Exhibit 6-2
for a description of performance on this measure). In addition, the Ocean Dumping
program tracks the "number of active dredged material Ocean Dumping sites that are
monitored in the reporting year."30 The FY 13 Congressional Justification states that
"During FY 2013, the EPA will collect environmental data from several offshore areas
for use in the designation of... dredged material disposal sites and .. .monitor, as required,
the 67 active dredged material ocean disposal sites." In addition to documented measures,
program staff shared that they track additional measures and are working on developing
new measures. For example, one interviewee described an output measure under
development that would track the number of projects and suitability determinations
processed each year. Another interviewee said the program tracks the timeframes of how
often site monitoring is conducted and when site management plans are updated, to track
whether the site management plans are reviewed and revised every 10 years as required.
28 EPA Operational Priorities for FY13 Ocean And Coastal Protection, Draft August 27, 2012
29 Source FY 2011 - 2015 EPA Strategic Plan, http://www.epa.gov/planandbudget/strategicplan.html. The universe of active
dredged material ocean disposal sites was 60 in 2005, 65 in 2009, and is currently 67 sites.
30 EPA Office of Water, Fiscal Year 2013 National Water Program Guidance, Appendix A Measures Summary, April 2012.
6-2
-------
EXHIBIT 6-2. PERFORAMANCE ON OCEAN DUMPING STRATEGIC PLAN MEASURE
MEASURE
Target
Actual*
Percent of Active Dredged Material Ocean Dumping Sites
that Will Have Achieved Environmentally Acceptable
Conditions (as Reflected in Each Site's Management Plan).
FY08
95
99
FY09
98
99
FY10
98
90.1
FY11
98
93
FY12
95
FY13
95
UNITS
Percent
sites
*Due to variability in the universe of sites, results vary from year to year (e.g., between 85 percent
and 99 percent). While this much variability is not expected every year, the results are expected to
have some change each year.)
Explanation of Results: Gulfport Western Ocean Dredged Material Disposal Sites (ODMDS) has exceeded
the minimum depth limitation. The Miami ODMDS has elevated PCB levels. In addition, multiple Gulf of
Mexico ODMDS likely do not meet environmentally acceptable conditions due to the Deep Water
Horizon Oil Spill and need to be evaluated.
Source: FY 13 Congressional Justification, page 853-854
STRENGTHS AND WEAKNESSES OF CURRENT MEASURES
Interviews with Ocean Dumping Management Program staff suggest that while the
current strategic plan measure is understood in theory, different Regions have different
definitions of what constitutes "environmentally acceptable conditions," and thus the
Regions do not use the measure in a consistent way. One interviewee stated that program
staff have discussed what the term "environmentally acceptable conditions" means, but
have reached no consensus. Another interviewee said that by definition, only sites that
had achieved environmental acceptable conditions would be approved, and therefore by
definition the measure should always be 100 percent. Another regional interviewee said
that one state in the Region would not sign off on some site management plans (in an
effort to make sure all dredge material was used for beneficial reuse) without the state's
agreement the Region could not say the sites had achieved environmentally acceptable
conditions. These incompatible perspectives mean that the values reported for the
strategic plan measure are not consistent or meaningful.
A review of performance on the strategic plan measure by Region shows that all but two
Regions rate themselves as having 100 percent of their active dredged material ocean
dumping sites that achieve environmentally acceptable conditions.31 For the two Regions
that miss the target, one interviewee notes that one Region understands the issues at their
sites (e.g., elevated PCB levels), and the other Region believes that environmental
conditions at the ocean dumping sites are not adequately characterized. While the current
measure does flag potential problems in these two Regions, it does not distinguish
31 EPA Office of Water, Fiscal Year 2013 National Water Program Guidance, Appendix E Detailed Measures Appendix, April
2012, page 12.
6-3
-------
between the two different types of problems these Regions face. More importantly, it
suggests that performance has been maximized in all the other Regions, which interviews
suggest is not the case. Overall, definitions underlying the performance measures reflect
different Regions' conceptions of the measure more than variations in performance.
In addition to concerns about inconsistency in interpreting the strategic plan measure,
interviewees express concern that the measure does not capture the quality of the work
the program does, or the broad range of program activities conducted. Overall, it does
not appear that the current measure provides feedback for staff about the effectiveness of
their efforts, because staff feel they have little control over the measure.
POTENTIAL ALTERNATIVE MEASURES
Interviewees and focus group participants suggested a number of options for developing
alternative performance measures, and both HQ and the Regions informally track some
data in addition to the Strategic Plan Measure. Possible alternate measures for the
program include outcome and output/efficiency measures, as described below.
Outcome Measures
Effective performance measures should track progress toward a goal (in addition to
meeting the criteria described earlier). For the Ocean Dumping Management Program,
goals include site designation and monitoring to ensure that substances dumped in the
ocean do not have a negative impact on human health or the environment. Another
objective is to increase the beneficial use of dredge material. Potential measures related
to these goals include:
• Volume of material dumped at ocean dumping sites. Since one overall goal
of the program is to minimize ocean dumping, this measure would indicate the
environmental benefit of the program, if the volume of matter disposed of
through ocean dumping declines overtime.
• Volume and percentage of contaminated material proposed for ocean
disposal that is not allowed to be ocean dumped (e.g., sent for land disposal
instead). This measure speaks directly to the environmental benefit of the
program, since it would show the extent to which the program prevents the
dumping of contaminated matter.
• Percentage of monitored sites found to be in compliance. This measure would
document whether sites are in compliance.
• Percentage of projects where material is diverted to beneficial reuse rather
than ocean dumping. This measure would show whether the program is
making progress in encouraging beneficial reuse for uncontaminated dredging
material, which is a goal of the program.
All of the outcome measures described here may be strongly influenced by factors
outside the program's control. Thus while these measures are important to
understanding progress toward the program's goals, they must be interpreted carefully,
and changes in an outcome measure should not be assumed to be cause by the Ocean
6-4
-------
Dumping Management Program. Evaluation is a useful tool that can assess contribution
of a program toward observed outcomes.32
Output/Efficiency Measures
In addition to outcome measures, the Ocean Dumping Management Program may wish to
also include output or efficiency measures, which it can more directly control. Several
options are listed below.
• Average number (and range) of years since site monitoring has occurred.
This measure would document whether the program is meeting the expectation of
monitoring all sites within the past 10 years, which is a key program activity.
Note that some sites need to be monitored more frequently than every 10 years.
In order for the measure to be comparable and meaningful, it would be necessary
to define what constitutes adequate monitoring.
• Average number (and range) of years since Site Management and
Monitoring Plans (SMMPs) that have been reviewed and updated. This
measure would document whether the program is keeping site management plans
up to date and meeting the requirements that SMMPs are reviewed and revised at
least every 10 years.
• Average time to complete specific EPA activities (e.g., leading to site
designation, suitability determinations, or permit reviews). This type of
measure would track program efficiency, and managers would seek to reduce the
measure over time to demonstrate improved efficiency. This type of measure
would need to be balanced with an adequate review of quality of EPA's work, in
order to ensure there is no perverse incentive to rush the work.
• Count of EPA activities (e.g., inquiries EPA responds to, permits reviewed,
etc.). This type of measure would not indicate program performance, but rather
would provide context for other measures by indicating the extent to which there
are ocean dumping activities that are potentially subject to EPA oversight and the
volume of work EPA is completing.
CONCLUSIONS
Overall, the current measure is not consistently interpreted by the Regions and does not
provide particularly useful feedback for understanding program effectiveness. A valid
performance measure is one that accurately represents the condition or phenomenon that
it is purporting to represent. The current measure would be valid if the key term
"environmentally acceptable conditions" were consistently defined.
32 In some cases, programs can attribute outcomes to a program, but this requires a higher standard of evidence, and is
difficult to do in the context of government program evaluations.
6-5
-------
The Ocean Dumping Management Program could improve information flows by
developing a suite of performance measures that reflect both outcomes related to the
programs goals and outputs over which the program has direct control.
RECOMMENDATIONS
The evaluation team recommends that the Ocean Dumping Management Program:
• Strengthen information flows to inform program adaptation, building on a suite
of performance measures that encompass both outcome and output measures; and
• Consider the following measures: volume of contaminated material not allowed
to be ocean dumped (and of this amount, percentage that is used for beneficial
uses); percentage of monitored sites found to be in compliance; average number
of years since ocean disposal sites were monitored; and average number of years
since Site Management and Monitoring Plans have been reviewed and updated.
6-6
------- |