Solutions-Driven Research Pilot
Problem Formulation Workshop:
Report and Evaluation
-------
Solutions-Driven Research Pilot
Problem Formulation Workshop
Report and Evaluation
National Health and Environmental Effects Research Laboratory
Office of Research and Development
U.S. Environmental Protection Agency
Atlantic Ecology Division
Narragansett, Rhode Island 02882
-------
Notice and Disclaimer
This Problem Formulation Workshop was held to gather input about research needs, challenges, and
opportunities in nutrient management from a range of stakeholders working in Three Bays, on Cape
Cod, and elsewhere. The statements and information captured in this report reflect the individual expert
views and opinions of the workshop attendees and the summary observations and recommendations of
an Agency evaluation team and the Organizing Committee. They do not represent positions of the U.S.
Environmental Protection Agency. While participants investigated research needs around several types
of non-traditional interventions, this is not an endorsement of any particular solution to nutrient
management and the report does not substitute for CWA or EPA regulations, nor is it a regulation itself.
Thus, it cannot impose legally binding requirements on EPA, states, territories, tribes, or the regulated
community and might not apply to a particular situation or circumstance.
This document was subjected to the Agency's peer and administrative review and approved for
publication as an EPA document. Mention of trade names or commercial products does not constitute
endorsement or recommendation for use. This is a contribution to the EPA Office of Research and
Development's Safe and Sustainable Water Resources Research Program.
The appropriate citation for this report is:
Twichell JH, Mulvaney KK, Hubbell B, Erban LE, Berry W, Chintala MM, Crocker Z, Gleason TR,
Horsley S, Munns, Jr. WR, Rea AW, Smith SN, Soto Reyes S. 2019. "Solutions-Driven
Research Pilot Problem Formulation Workshop: Report and Evaluation." U.S. Environmental
Protection Agency, Office of Research and Development, National Health and
Environmental Effects Laboratory, Atlantic Ecology Division, Narragansett, Rl.
EPA/600/R-19/107.
This document can be downloaded from:
www.epa.gov/water-research/water-research-publications-science-inventorv
iii
-------
Table of Contents
Notice and Disclaimer iii
Table of Contents iv
List of Figures vi
List of Tables vii
List of Acronyms viii
Foreword ix
Authors and Contributors x
Acknowledgements xi
Executive Summary xii
1. Introduction 1
1.1 Background 1
1.2 Purpose of this Report 2
2. Day One Workshop Synopsis 5
2.1 Goals of the Problem Formulation Workshop 5
2.2 Day One Workshop Design and Planning 6
2.3 Day One Workshop Overview 8
3. Workshop Knowledge Gap Findings 13
4. Day One Workshop Evaluation 19
4.1 Purpose of this Evaluation 19
4.2 Evaluation Methods 20
4.3 Evaluation Results and Discussion 21
4.3.1 Outputs-Participants Engaged 21
4.3.2 Short-term Outcomes 22
4.3.2.1 Benefits to Workshop Participants 23
4.3.2.2 Benefits to the EPA 26
4.4 Lessons Learned 27
4.4.1 Lessons for Workshop Design 27
4.4.2 Utility of Evaluation to Increase Program Impacts 28
5. Day Two Meeting Synopsis 30
5.1 Day Two Meeting Overview 30
5.2 Systems Mapping Exercise 31
iv
-------
6. Next Steps 34
7. Conclusion 38
References 40
Appendix A Sample invitation letter to participants 43
Appendix B Participant agenda 46
Appendix C Handouts for the morning breakout session 48
Appendix D Blank grid posters for the morning breakout sessions 56
Appendix E Blank prioritization posters for the morning breakout sessions 58
Appendix F Blank grid posters for the afternoon breakout sessions 59
Appendix G Blank prioritization posters for the afternoon breakout sessions 61
Appendix H Facilitator agenda 62
Appendix I Notetaker guidance and templates 68
Appendix J Full meeting summary shared with participants 84
Appendix K Consensus Building Institute Day One DRAFT meeting summary 94
Appendix L Pre-workshop questionnaires 105
Appendix M Post-workshop (exit) questionnaires 106
Appendix N Detailed systems map 108
v
-------
List of Figures
Figure 1. Map of impaired watersheds on Cape Cod, including Three Bays watershed 1
Figure 2. Illustration of a solutions-driven research cycle 3
Figure 3. Basic model of outcomes for Three Bays nutrients project 8
Figure 4. Examples of posters used in the morning breakout sessions 10
Figure 5. Example of posters used in the afternoon breakout sessions 11
Figure 6. Nine key themes identified as priorities during the afternoon breakout sessions 12
Figure 7. Breakdown of participants in the Problem Formulation Workshop by role 15
Figure 8. Map of Day One of the Problem Formulation Workshop 32
Figure 9. Example dialogue map from an afternoon breakout session 33
Figure 10. Sample stakeholder map of workshop participants and other stakeholders 36
Figure 11. Sample stakeholder map of specific potential engagement techniques 36
Figure 12. Problem Formulation Workshop solutions-driven research cycle 38
vi
-------
List of Tables
Table 1. Program Goals of the Problem Formulation Workshop 5
Table 2. List of expected outputs with notes on activity structure 6
Table 3. Expected short-term outcomes 7
Table 4. Groups represented at the Problem Formulation Workshop 16
Table 5. Themes used to group research needs posed by participants in breakout groups 17
Table 6. Major themes from responses relating to the top three challenges listed by participants in exit
surveys 20
Table 7. Responses about participants' overall experience in the workshop 23
Table 8. Major themes from responses to "Overall, what did you get out of today's workshop?" and
"What was the best part of the day for you?" 24
Table 9. Responses about participants' experience in activities in the workshop 25
Table 10. Goals for ORD and BCWC following Day One (includes Day Two and beyond) 30
Table 11. Participant responses to the question, "How would you like to receive information about
research efforts going forward?" 34
Table 12. Participant recommendations of roles not represented at the workshop that they believe
should be engaged going forward 35
vii
-------
List of Acronyms
BCWC: Barnstable Clean Water Coalition
CBI: Consensus Building Institute
CEC: Contaminant of emerging concern
COMM district: Centerville-Osterville-Marstons Mills district
EPA: United States Environmental Protection Agency
l/A: innovative and alternative septic system
MassDEP: Massachusetts Department of Environmental Protection
MassDER: Massachusetts Department of Environmental Restoration
MBL: Marine Biological Laboratory
MEP: Massachusetts Estuaries Program
N: Nitrogen
NGO: Non-governmental organization
ORD: United Stated Environmental Protection Agency Office of Research and Development
PRB: Permeable reactive barrier
SMAST: University of Massachusett- Dartmouth School for Marine Science and Technology
TMDL: Total maximum daily load
TNC: The Nature Conservancy
Watershed MVP: Watershed Multi-Variant Planner
-------
Foreword
The U.S. Environmental Protection Agency (EPA) is charged by Congress with protecting the Nation's
land, air, and water resources. Under a mandate of national environmental laws, the Agency strives to
formulate and implement actions that balance between human activities and the ability of natural
systems to support and nurture life. To meet this mandate, EPA's Office of Research and Development
(ORD) is solving environmental problems and building the scientific knowledge base necessary to
manage ecological resources, understand how pollutants affect human and environmental health, and
prevent or reduce environmental risks in the future.
ORD provides the strong scientific and technical foundation the Agency relies on to fulfill its statutory
obligations and address the most pressing environmental and public health challenges. By working
closely with external stakeholders throughout research planning and implementation, ORD ensures a
collaborative, transparent, and highly coordinated research program. ORD's water research program
delivers the data and information that Agency program and regional offices need, while also providing
resources that help state and local agencies, tribes, and communities inform actions to protect their
environment, safeguard public health, and support a robust economy.
ORD has emphasized solutions-driven research in the next set Strategic Research Action Plans.
Solutions-driven research focuses on stakeholder needs and involvement and spans from problem
formulation, research execution, all the way through evaluation of relevancy, usefulness, and timeliness
of deliverables. By working closely with partners throughout the research cycle, ORD also ensures that
the results - and translation of those results - are meaningful to inform water-related decisions and
activities.
This report is for EPA scientists and management as well as other agencies and organizations for use in
developing solutions-driven research efforts that closely involve stakeholders throughout the design,
planning, implementation, and dissemination of research efforts. It documents the methods and
approaches used in the first stage of the research planning process and problem formulation for one
project focused on nutrient reduction in Three Bays, Cape Cod, Massachusetts. This report also assesses
those approaches in: 1) achieving effective stakeholder engagement through building trust; 2) providing
an inclusive and deliberative environment for exchanging ideas and information; and 3) building capacity
for stakeholders to be involved in the research that can affect their lives and communities. The goal of
this document and assessment is to add to the literature on good practices for stakeholder-engaged
research and to identify and share lessons to inform future research efforts.
Suzanne van Drunick, National Program Director
Safe and Sustainable Water Resources Research Program
US EPA Office of Research and Development
ix
-------
Authors and Contributors
This report was prepared by an Evaluation Team and Organizing Committee to document the evaluation
of the Nutrients Solutions-Driven Research Pilot Problem Formulation Workshop.
Evaluation Team
Julia H. Twichell, Principal Author
National Health and Environmental Effects
Research Laboratory
Office of Research and Development
U.S. EPA
Narragansett, Rhode Island
Kate K. Mulvaney
National Health and Environmental Effects
Research Laboratory
Office of Research and Development
U.S. EPA
Narragansett, Rhode Island
Bryan Hubbell
Immediate Office of the Assistant Administrator
Office of Research and Development
U.S. EPA
Research Triangle Park, North Carolina
Laura E. Erban
National Health and Environmental Effects
Research Laboratory
Office of Research and Development
U.S. EPA
Narragansett, Rhode Island
Organizing Committee
Walter Berry
National Health and Environmental Effects
Research Laboratory
Office of Research and Development
U.S. EPA
Narragansett, Rhode Island
Marnita M. Chintala
National Health and Environmental Effects
Research Laboratory
Office of Research and Development
U.S. EPA
Narragansett, Rhode Island
Zenas Crocker
Barnstable Clean Water Coalition
Three Bays Preservation
Osterville, Massachusetts
Timothy R. Gleason
National Health and Environmental Effects
Research Laboratory
Office of Research and Development
U.S. EPA
Narragansett, Rhode Island
Scott Horsley
Independent Consultant
Barnstable, Massachusetts
Wayne R. Munns, Jr.
National Health and Environmental Effects
Research Laboratory
Office of Research and Development
U.S. EPA
Narragansett, Rhode Island
Anne W. Rea
Safe and Sustainable Water Resources Research
Program
Office of Research and Development
U.S. EPA
Research Triangle Park, North Carolina
Sofia Soto Reyes
Consensus Building Institute
Cambridge, Massachusetts
Stacie Nicole Smith
Consensus Building Institute
Cambridge, Massachusetts
x
-------
Acknowledgements
The participants in the workshop provided their time and a wealth of useful information and insights.
Notetakers Heather Rockwell and Casey Dannhauser of the Barnstable Clean Water Coalition
(Barnstable, Massachusetts) and Hale Thurston of the National Risk Management Research Laboratory
in EPA's Office of Research and Development (Cincinnati, Ohio) provided thorough documentation of
the workshop. Brenda Rashleigh of the National Health and Environmental Effects Research Laboratory
in EPA's Office of Research and Development (Narragansett, Rhode Island) and Caroline Ridley and Scot
Hagerthey of the National Center for Environmental Assessment in EPA's Office of Research and
Development (Research Triangle Park, North Carolina and Washington, D.C., respectively) provided
guidance and feedback during the planning and implementation of the workshop.
-------
Executive Summary
The U.S. Environmental Protection Agency Office of Research and Development (ORD) collaborated with
Barnstable Clean Water Coalition (BCWC) to implement a Problem Formulation Workshop. The two-day
workshop aimed to engage key stakeholders and experts to identify research needs and initiate
coordination of a scientifically robust research effort focused on nutrient reduction in Three Bays, Cape
Cod, Massachusetts. This report presents a synopsis, formal evaluation, and lessons learned from the
workshop.
The first (full) day workshop included nearly sixty scientists, decision makers, and other key stakeholders
working on nutrient reduction around Three Bays and elsewhere on Cape Cod. The goal was to identify
and prioritize key knowledge gaps and partnership needs related to possible solutions for mitigation of
nonpoint nutrient pollution in the Three Bays watershed. The workshop consisted of a series of
structured small group and plenary discussions focused on capturing and clarifying research needs
around (a) non-traditional nutrient-reduction interventions and (b) watershed-level conditions and
impacts. The second (half) day comprised of ORD and BCWC action planning to debrief the workshop,
review initial insights from the stakeholders, and identify next steps to continue forward-looking
discussions.
This report provides: (1) synthesis and analysis of information outputs co-produced by participants and
organizers during the workshop and (2) an impact analysis of benefits to the EPA and to participating
scientists, decision makers, and other stakeholders from the workshop. A number of key lessons were
learned from the process:
Key lessons from the Problem Formulation Workshop
Clearly defined outputs and outcomes improved planning, workshop
design, impact assessment, and identification of lessons learned.
Structured activities and detailed facilitator guidance enabled
participants and organizers to co-produce higher quality data.
Facilitator and observer training improved comprehensive
documentation of the information produced in the workshop.
Content and document analysis condensed transferable and operable
information outputs from the raw data that could be useful to
participants, the EPA, and other stakeholders.
Application of program evaluation and social science techniques added
structure and value to the design process, implementation, and
information processing following the workshop.
xii
-------
The information outputs of the workshop included a series of one- or two-page tables, shared both
within the EPA, BCWC, and with participants, that compile the research needs identified in the
structured discussions. These were organized thematically across discussion groups. Major themes
included:
Research needs identified in the Problem Formulation Workshop
calls for pilot projects on Cape Cod
monitoring, permitting and funding support
technology characterization and performance assessment
behavioral change and social acceptance research
a centralized online hub for information sharing.
Overall, the Problem Formulation Workshop was an effective platform for ORD to engage with a diverse
group of stakeholders before developing detailed research plans have been developed. Participants felt
the workshop was constructive and a number of research areas were identified that can be
implemented by researchers at ORD or partner organizations. The workshop outputs provided not only
insights related to research gaps, but also helped identify participants with similar research interests and
expertise for partnering with ORD and each other in future efforts. Participants benefited through the
opportunity to network with colleagues, experts, and decision makers. They reported satisfaction that
the workshop reflected broad, multidisciplinary participation, and both EPA personnel and participants
described the day as a valuable learning opportunity about the big picture of opportunities, challenges,
needs, and local and regional efforts around nutrient management in Three Bays and elsewhere on Cape
Cod. The workshop was a robust early step toward identifying research activities to address key
scientific issues that will inform stakeholders' long-term goals to significantly reduce nutrient loading to
surface waters and coastal embayments and achieve community and regulatory water quality goals in
Three Bays and Cape Cod, and to transfer solutions and lessons learned to other areas facing similar
challenges. Similar problem formulation workshops may be of use for EPA scientists and management,
as well as other agencies and organizations, for use in developing solutions-driven research efforts that
actively engage stakeholders throughout research or program efforts.
-------
1. Introduction
1.1 Background
Nonpoint source nutrient pollution is one of the most widespread, costly, and challenging
environmental problems in the United States. Nutrient pollution on Cape Cod, Massachusetts is no
exception. More than 30 of the 53 watersheds on Cape Cod have Total Maximum Daily Load (TMDLs) set
by the state of Massachusetts under the Clean Water Act to address nitrogen impairment in coastal
embayments. This includes Three Bays, an estuarine system in Barnstable, Massachusetts and the area
of focus for the program evaluated in this report (Figure 1).
SANDWICH
BARNSTABLE
MASHPEE
Watershed Impairment Status (2017)
Impaired
Three Bays (Impaired)
BARNSTABLE
BOSTON
FALMOUTH
Nantucket Sound
Nantucket Sound
Figure 1. Right: Impaired watersheds on Cape Cod (solid yellow), including Three Bays in Barnstable,
Massachusetts (turquoise stripes). Left: Three Bays watershed (solid turquoise) overlaid with town
boundaries. Data sourced from Cape Cod Commission GIS Hub (2017).
Nonpoint source nutrient loading on Cape Cod and specifically to Three Bays is a unique challenge
compared to other parts of the country as it is primarily sourced from groundwater flow impacted by
use of septic systems. It is estimated that 65 percent of nitrogen loading on Cape Cod stems from
wastewater, with the majority of that wastewater loading from aging or outdated septic systems
(Howes et al., 2003). As a result, nutrient loading has negatively impacted groundwater supplies and is
expected to continue to cause increased impacts to surface water and groundwater quality, thereby
affecting communities and the environment.
1
-------
In 2015, the Cape Cod Commission updated Cape Cod's 1978 Area Wide Water Quality Management
Plan (208 Plan Update), which requires municipalities to significantly increase nutrient reduction efforts
for Cape Cod water bodies (Cape Cod Commission 2015). In order to meet nutrient reduction
requirements and environmental protection goals, municipal efforts must mitigate new nitrogen inputs
in wastewater and also must address the legacy nutrient pollution still remaining in surface and
groundwaters. Cape Cod municipalities do not yet have active programs that are adequately equipped
to reduce nonpoint source and legacy nutrient pollution. Further, similar to many other embayments on
Cape Cod, the nutrient problem in Three Bays spans municipal boundaries, with the majority of the
Three Bays watershed area comprising the town of Barnstable, but also extending into the towns of
Sandwich and Mashpee (Figure 1). The 208 Plan Update outlines actions for coordinated, watershed-
scale solutions to address these jurisdictional concerns, primarily through the development of
watershed permits (Cape Cod Commission 2015). Establishing and implementing such programs to meet
the watershed permits or through other efforts to mitigate nitrogen pollution will be difficult and costly
for municipalities.
Both traditional sewering and wastewater treatment plants, as well as a number of non-traditional
interventions, are being considered to help meet nutrient reduction requirements and environmental
protection goals. Expanding and updating traditional sewering for source control is one option, albeit
very expensive and with considerable time delay for improvements in some areas. On Cape Cod, a
number of alternative interventions for nitrogen mitigation are gaining traction. Innovative and
alternative (l/A) septic systems can provide source reduction benefits. Permeable reactive barriers
(PRBs) and fertigation wells can provide site remediation or watershed intervention benefits. Habitat-
and landscape-scale interventions such as shellfish, aquaculture, and stream, lake, and wetland
restoration are also viable options. Research is needed to better understand which nutrient reduction
interventions will most effectively remove nutrients from the system and on what time scale, which are
most cost effective, and which are most socially acceptable. Research, development of watershed-scale
approaches, and implementation will likely require coordinated efforts among scientists and local, state,
and federal-level decision makers. Some research has been conducted by ORD on Cape Cod and in Three
Bays related to these needs such as decision-making analyses (Martin et al. 2019) and system
framework description (Merrill et al. 2018) and valuation of ecosystem service benefits of coastal
recreation (Merrill et al. in review, Mulvaney et al. in review), but stakeholder engagement and
collaboration has highlighted the need for addressing furthering this work and investigating additional
research needs.
1.2 Purpose of this Report
The Problem Formulation Workshop, jointly hosted by the U.S. Environmental Protection Agency (EPA)
Office of Research and Development (ORD) and Barnstable Clean Water Coalition (BCWC), was an early
step in ORD's work in a pilot project to apply principles of solutions-driven research (also referred to as
translational research or translational science). The ultimate goal of the pilot project is to design and
implement research to inform watershed-scale solutions to nutrient pollution in Three Bays that are
2
-------
transferable to other areas on Cape Cod and in other regions facing similar challenges within the United
States.
Translational science is the study of how research can better lead to improved health and well-being
outcomes for individuals and communities. It includes engagement by researchers with their
stakeholders to identify and prioritize needs and to design, conduct, and disseminate research that
better informs solutions (National Institutes of Health 2018). ORD's concept of solutions-driven research
builds on the successful model from the National Center for the Advancement of Translational Science.
The model centers around stakeholder needs and involvement and emphasizes a cycle of discovery,
application, and evaluation (Figure 2).
,\oPNe\w Approac/,,
Y Environment and J
/\ Public Healthy "N.
Training and
Application
Stakeholder
Involvement
Tool
Development,
Field Research
y & Synthesis y
'e°^/ a;en|EA3
Figure 2. Illustration of a solutions-driven research cycle, adapted from the National
Institutes of Health Translational Science Spectrum (National Institutes of Health 2018).
The two-day Problem Formulation Workshop was the official launch of the nutrients solutions-driven
research pilot. In the first day of the workshop, scientists; local, state, and federal decision makers; and
other key stakeholders gathered to identify research needs and priorities for resolving nutrient
impairment in the Three Bays estuary. The second (half) day comprised of ORD and BCWC action
planning to debrief the Problem Formulation Workshop, review initial insights from the workshop
participants, and initiate forward-looking discussions. Careful documentation of the workshop enabled
dissemination of the synthesized information outputs back to participants. These key research needs
and priorities, gathered from the participants, were also useful for evaluating the impact of the
workshop and will be applied for developing approaches in the next phases of research planning.
Benefits of the workshop for participants and other stakeholders will be assessed when the research
plans are implemented and results are provided. The extensive stakeholder input during problem
formulation, and that is anticipated moving forward, will guide the project plan development and inform
continued dialogue throughout implementation of the plan.
3
-------
This report encapsulates both a meeting synopsis and a formal evaluation of the Problem Formulation
Workshop. Section Two provides a workshop synopsis that describes the planning, design, and
implementation of Day One of the Problem Formulation Workshop. Section Three provides the findings
from the workshop in terms of research gaps related to the potential nitrogen removal technologies
which includes the compilation and analysis of the key information outputs distilled via document and
content analysis of the workshop discussions. Section 4 comprises an evaluation of Day One. This
includes the investigation of the key benefits to participants and to ORD resulting from the workshop,
and lessons learned about effective workshop design and about the value of program evaluation for
increasing workshop and other types of programs' impacts. Program evaluation was important as it
provided valuable structure to planning, design, and implementation and, as a result, facilitated better
planning, enabled the production of higher quality data, and helped organizers compile
recommendations for future similar efforts. Section Five provides an overview of the second (half) day
of the workshop summarizing the debrief discussions between EPA and BCWC and a review of the
systems mapping exercise that was piloted. As the Problem Formulation Workshop is one piece of a
longer, solutions-driven research initiative, the report concludes with a discussion of potential next
steps (Section Six) and overall conclusions (Section Seven).
4
-------
2. Day One Workshop Synopsis
ORD teamed with BCWC to initiate the first broad, multidisciplinary effort since the 208 Plan Update to
make progress toward developing a coordinated and scientifically robust approach for tackling nutrient
reduction in Three Bays. Jointly designed by ORD and BCWC, Day One of the Problem Formulation
Workshop hosted scientists, decision makers, and other key stakeholders working on nutrient reduction
around Three Bays and elsewhere on Cape Cod.
2.1 Goals of the Problem Formulation Workshop
Through the broader solutions-driven research pilot study, ORD aims to identify effective ways to
implement solutions-driven nutrient reduction research efforts in ORD's next four-year research cycle
that 1) build trust, dialogue, and collaboration with stakeholders; 2) complement and translate to
current efforts in Three Bays, Cape Cod, and elsewhere; 3) help inform nutrient reduction requirements
and environmental protection goals; and 4) enhance capacity-building for solutions-driven research
approaches and improve the usability of ORD science. To that end, the Problem Formulation Workshop
was designed to serve as the initial phase in which ORD worked in collaboration with key stakeholders to
identify and prioritize nutrient management research needs as well as to foster partnerships and
innovative approaches to address those needs.
Organizers produced a set of initiative goals for the Problem Formulation Workshop that were shared
with participants prior to the workshop in the invitation letter and participant agenda (Table 1; Appendix
A, B). Setting goals helped elucidate what organizers intended for the ultimate impacts of the workshop.
These goals were thus integral for designing the structure of the workshop and deciding who should
participate in order to produce those impacts. Defined goals also enabled evaluation of whether the
program ultimately produced those impacts.
Table 1. Program Goals of the Problem Formulation Workshop
1) Clarify the knowledge gaps and relationship/partnership needs related to the efficacy
and social acceptance of potential nitrogen removal technologies and the
environmental health of the watershed as a whole.
2) Identify the potential research contributions and approaches for key workshop partners
to improve our understanding of the identified key knowledge gaps.
3) Build relationships across scientists, decision makers, and other stakeholders, and
identify additional needs and approaches for fostering partnerships and networks
within and beyond the watershed.
4) Explore how the work in Three Bays watershed can be transferred to other watersheds
facing similar challenges.
5
-------
2.2 Day One Workshop Design and Planning
Evaluators generated expected outputs and short-term outcomes for the workshop, which were refined
during workshop planning in consultation with the organizing committee and the contracted facilitators
from the Consensus Building Institute (CBI) (Table 2). Outputs are what was produced. These include the
overall workshop, the specific activities, and the products generated before, during, and after the
workshop by organizers and by participants. The information outputs produced by organizers and
participants during the workshop were defined specifically to meet the goal of using this workshop to
clarify the key knowledge gaps, potential research approaches, and relationship/partnership needs
around potential nutrient removal interventions and their impacts on the watershed as a whole.
Table 2. List of expected outputs with notes on activity structure (activities were structured
around extracting those information outputs)
Morning Breakout Groups -four breakout groups, same outputs for each intervention-focused group
Activity: Refining the Knowns and Unknowns of the Efficacy and Social Acceptance of Nitrogen
Removal Technologies
¦ Output: List of knowns and unknowns (review handout (Appendix C), blank flipchart)
Activity: Intervention-Focused Research Needs Identification
¦ Output: List of research needs developed around the identified unknowns
¦ Optional: organized in four categories: Monitoring, Modeling, Social Science, Other
(Appendix D)
Activity: Group Prioritization of Research Needs
¦ Output: List of the three to five highest priority research needs in the next two to five years
(Appendix E), distinguished by multi-voting or by group consensus
Afternoon Breakout Groups -four breakout groups, same outputs for each group
Activity: Watershed-Level Research Needs Identification
¦ Output: Each participant's three to five key needs/questions (written on sticky notes)
organized by four categories (Monitoring, Modeling, Social, Other - Appendix F)
Activity: Group Prioritization
¦ Output: List of the three to five highest priority research needs in the next two to five years
(Appendix G), distinguished by multi-voting or by group consensus
Plenary Discussion - small groups (two to three participants) followed by plenary report out
Activity: "Taking it Forward" discussion
¦ Output: Each participant group records (on sticky notes) and reports out one or two
actions/next steps based on priority research needs, linked with potential
organizations/roles where possible
Once outputs were defined, organizers refined the structure of the workshop so those outputs could be
extracted by facilitators. Organizers iteratively adjusted the program agenda away from an open-ended
discussion format toward a structured sequence of activities conducted in multiple breakout sessions. A
detailed facilitator agenda was created to streamline and unify those activities across groups (Appendix
H). In addition, a series of posters were created to guide facilitators and participants through production
6
-------
of information outputs from each activity and to serve as a visual for participants (Appendices D-G).
Creating this clear structure enabled participants and facilitators to produce explicit information outputs
in a format that was useful and meaningful as well as relatively standardized across the four breakout
groups (see Section 2.3 for a description of how the breakout groups were organized). Furthermore,
clearly defining the outputs improved communication and planning among organizers, the contracted
facilitators, and the breakout session facilitators.
Expected short-term outcomes were also defined prior to the workshop. Outcomes are the benefits to
organizers and participants resulting from the outputs produced. Short-term outcomes were limited to
outcomes that organizers anticipated could be accomplished by the end of Day One of the workshop.
These included initiated and improved communication, better understood research needs, and positive
engagement with participants (Table 3). Meeting the ultimate goals of the workshop (Table 1) was made
possible by first achieving these benefits for stakeholders and for organizers. Evaluators worked with
organizers to refine these expected short-term outcomes so that they aligned with the goals of the
workshop.
Table 3. Expected short-term outcomes
¦ Communications are initiated or strengthened between workshop participants (e.g., between
local, state, and federal representatives and/or between experts/scientists and decision makers).
¦ Participants feel heard.
¦ Participants feel hopeful and galvanized about research directions identified during the session.
¦ Participants feel that research needs are better understood and more clearly defined.
¦ Participants feel that research directions are useful to stakeholders.
¦ Participants brainstormed, organized, and documented ideas during the workshop that can serve
as a grounding framework for constructing a research program plan.
¦ Key stakeholders feel encouraged that progress will be made toward meeting their organizations'
research needs.
An initial, high-level model of the linkages between short-, medium-, and long-term outcomes was
produced to help evaluators and organizers visualize how the workshop would tie into the overall
solutions-driven research pilot effort (Figure 3). This model is designed to be a precursor to a more
refined and detailed logic model that will be iteratively adjusted by organizers as part of the longer-term
pilot process. Logic models describe critical linkages among inputs, outputs, and short- and long-term
outcomes (McLaughlin and Jordan 1999, University of Wisconsin 2017). For example, some of the
observed short-term outcomes to participants in the workshop, such as increased trust among
stakeholders and improved understanding of the goals, processes, and challenges in Three Bays (see
Section 3.3.2), would be added in to increase the level of detail and expand the scope of the model.
Further, new short-term goals and outcomes corresponding to next steps after the workshop will need
to be added, to map how organizers navigate from the Problem Formulation Workshop to the broad
medium-term outcomes mapped below (medium-term outcomes in green).
7
-------
Gaps and constraints
(unknowns and knowns)
identified
Comprehensive approach to
nitrogen management
developed and implemented
with some guidance from the
research plan
Research produces models,
tools, strategies for use in
Three Bays and other areas
Research questions and
needs identified; possible
research collaborators
identified
Research communicated with
decision makers (and to other
watersheds)
Three Bays TMDL met/ Water
quality goals in Three Bays
watershed are achieved
Research/action plan
developed between ORD &
other researchers
Multidisciplinary research
across partners implemented
to address research gaps.
Other watersheds on the
Cape and elsewhere also
implement nutrient
mitigation efforts informed by
the research plan
Day One Workshop:
Short-term outcomes
Medium-term
outcomes
Long-term outcomes
Figure 3. Basic model showing progression from short- to medium- to long-term outcomes for Three
Bays nutrients project.
2.3 Day One Workshop Overview
The Problem Formulation Workshop was hosted by ORD and BCWC on October 30 at the Marine
Biological Laboratory, in Woods Hole, Massachusetts (Appendix B). The purpose of the workshop was to
identify key knowledge gaps and research needs for the sustainable management of nutrients in the
Three Bays watershed, as a model for Cape Cod and other areas facing similar challenges elsewhere in
the United States. Stakeholders from relevant towns, research institutions, and regulatory agencies
were invited for the day to identify the potential research needs, contributions, and approaches for
meeting nutrient management needs and environmental protection goals in Three Bays and to foster
partnerships and networks within and beyond the watershed.
The morning began with welcome and introductory remarks from ORD's National Program Director for
the Safe and Sustainable Water Resources Program, Suzanne van Drunick. She introduced ORD's
Nutrients Solutions-Driven Research pilot project focused on innovative, solutions-based research for
nutrient management and the role of the pilot project within the broader research portfolio outlined in
ORD's Strategic Research Planning. She also mentioned EPA's role during the workshop, emphasizing
that it was to listen to better understand to the issues of highest importance to the
stakeholders/partners and the specific research needs identified to inform our understanding of these
issues.
8
-------
Next, Bryan Hubbell, ORD's Senior Advisor on Social Science, presented in more detail on solutions-
driven research. He defined the solutions-driven science model as one that works to "help us, help you".
He stated that ORD is committed to producing research that specifically addresses "real world problems
in a real-world context" allowing for moving towards solutions that are implementable with partners. He
elaborated that this model is centered on stakeholder involvement, with a desire to engage with
stakeholders from the beginning to the end of a process (i.e., beginning with developing research plans
through the delivery of results).
Stacie Smith, Senior Mediator from the Consensus Building Institute (CBI) and meeting facilitator, then
provided an overview of the day and also reviewed ground rules and logistical items. She introduced
Zenas Crocker, Executive Director of the Barnstable Clean Water Coalition (BCWC), who presented on
the "Vision for Three Bays." He gave an overview of the nutrient management challenges and recent
management efforts in Three Bays, and personally thanked many of the participants by name for their
contributions. He noted the potential for Three Bays watershed to serve as a national model for the use
of non-traditional nitrogen reduction interventions in meeting nutrient management requirements and
environmental protection goals.
Following the plenary presentations, the group broke into four pre-assigned small groups for concurrent
discussions. Participants with specialized expertise were pre-assigned to groups based primarily on their
familiarity with the subject matter. Colleagues from the same organization were separated, if possible,
to allow for broader representation across the groups. Anyone without specialized expertise was
randomly assigned to a group. Each of the breakout groups was led by one of four additional facilitators,
three from ORD and one Associate from CBI. Facilitators directed breakout group discussions using a
focus group format, guided by a detailed stepwise facilitator agenda which increased synchrony across
groups. Facilitators used multi-voting to manage group prioritization where needed (Tague 2005).
Two forms of documentation were used to record, in detail, all discussions in the workshop. Facilitators
elicited and documented key discussion points using a series of prepared flipcharts and posters (Figures
4-5, Appendices D-G). These posters also served to visually track the discussion so that participants felt
heard and that their input was valued. A dedicated notetaker was assigned to each breakout group to
record the flow of conversation. They were directed to capture information in participants' own words
as much as possible. Notetakers were trained in advance in the use of structured templates (Appendix I)
that guided their notetaking through different activities in each discussion. Both the facilitators' and
notetakers' records enabled later analysis and distillation of information co-produced from the
workshop. The rigorous and structured documentation of each activity in the workshop was designed to
increase the quality of data outputs.
The goal of the first breakout group discussion was to identify key priority technical/scientific questions
that need to be answered regarding different non-traditional nutrient removal interventions in order to
move forward with piloting and implementing those interventions in Three Bays and elsewhere on Cape
Cod. These breakout discussions focused around four types of nutrient reduction interventions. These
included (1) shellfish and aquaculture, (2) permeable reactive barriers (PRBs) and fertigation wells, (3)
innovative and alternative septic systems (l/A systems), and (4) habitat restorations as cranberry bog,
stream, and pond restorations.
9
-------
The group began by identifying and discussing what is known and unknown around each technology.
Discussion was prompted by a handout(s) that organizers produced in advance, detailing an initial list of
knowns and unknowns (Appendix C). Participants were directed to read through the handout(s) and
comment on whether its contents were accurate, suggest additional knowns or unknowns, or refine the
list. The facilitator flip-charted key discussion points. Next, participants generated a list of research
needs. These were recorded by the facilitator on large gridded posters designed to organize discussion
into four topics (1) monitoring, (2) modeling, (3) social science, and (4) other (Figure 4L; Appendix D).
Finally, participants used multi-voting or verbal consensus to identify priority research needs. These
were listed separately by the facilitator on a new poster (Figure 4R; Appendix E).
¦l
F
Monitoring
i illlLlVi vvy
'VWMtavrA fit brotWi
^
TftW ^ MliwrJC -fo WW <* ^iu,ld ^
i Sun vji-
\- W ^
Modeling
rvwMiMj (Mttirtrvt toftytfunia >
t«4UWC-ty\Xc4*tMiKr of Hvn oih'al
' JKSfW h Rita*
p4|MwW^ WcwitVi \ xitid
F SociaF Science
' Wtf/'vfyU L> Uctl (Wfr.r ?
' tkWvi "\fl\us ^ teas
't> Y v( Il'. H VCSftl
¦mfcvodttk K|
| '
Other
fTWWit titatrvrt Au boo.-,
in KSWMvcft?
to 3
COStef do h( »\dfa\iSo
CMate ^
^ CattxA -
Shellfish & Aquaculture Breakout Group
Priority Researctj Needs
I Need cc* *' 5.^5*+- uW '<>
I f I.
eed3: iM " ^ kU*"
eed 2
eed 4:
Need 5:!/ M
Need 6:
^ riywl
Figure 4. Examples of posters used in the morning breakout sessions. Posters were titled and color-
coded by each group's intervention topics. (L) the three-foot by five-foot "Research Needs
Identification" gridded poster. (R) The "Priority Research Needs" poster.
The groups were charged with answering the following questions:
1) Did we get the knowns/unknowns right? If not, what did we get wrong?
2) Have we identified the key questions about this intervention related to modeling, monitoring,
regulatory issues, economic/financing issues, and social acceptability? Are there other barriers
we need to overcome or opportunities that we can draw on?
3) What are key research needs/questions?
4) What are the highest priority research needs and questions we should seek to answer in the next
two to five years?
After the breakout session, the workshop participants reconvened in a plenary session where each
breakout group reported on the three to five top priority needs identified in their session to the full
group, with time for discussion and questions.
After lunch, a second set of breakout sessions commenced. Two to four participants from each morning
breakout group were separated into four new groups, with the intention that a few voices in each
afternoon group represented each of the morning breakout groups. Participants were charged with
10
-------
focusing on key watershed-wide questions and systems-ievel research needs. Each breakout group was
asked to focus on the question, "Above and beyond intervention-specific questions, what are the
broader systems-level key research needs/questions for the watershed as a whole related to modeling,
monitoring, regulatory issues, economic/financing issues, and social acceptability?"
Participants used new blank posters designed in the same grid format from the morning breakout
sessions (Figure 5, Appendix F). Unique to the afternoon breakout sessions, participants were provided
with several blank, large sticky notes. They were asked individually to write out three to four watershed-
level research needs and post them within the grid on the blank poster.
Afternoon Breakout Group^
Monitoring _
Modeling
Monitoring
Social
Social Science
Figure 5. Examples of posters used in the afternoon breakout sessions. Posters were titled and color-
coded by each group's intervention topics. In the afternoon sessions, participants posted research needs
on large sticky notes (sticky note colors had no significance). Some groups then opted to (L) use multi-
voting on priority research needs using sticky dots (sticky dot colors had no significance) or (R) group
and prioritize research needs using their own agreed-upon organization system.
They were then asked to refine, add, and/or streamline these ideas into a list of priority watershed-level
research needs, using verbal consensus or multi-voting (Figure 5; Tague 2005). These priority research
needs were then recorded on a blank poster (similar to Figure 4R, Appendix G). After the breakout
session, groups reported on the priority research needs to the full group, with time for discussion and
questions.
The day closed with a plenary discussion. The CBI facilitator and organizers roughly synthesized the
priority research needs from the afternoon breakout sessions and posted the list as a discussion prompt
(Figure 6). Participants formed small groups (two or three people) and brainstormed two possible
actions or solutions each for two prompts from the list. They were directed to narrow their
consideration of actions to those that are salient, legitimate, credible, and attainable (Cash et al. 2003).
They were also directed to identify people or organizations that might be best situated to carry out
those actions. This exercise did not constitute a commitment to fulfill those actions. Small groups
reported out to the full group. The aim of this activity was to identify what might be done collectively to
move forward and to provide organizers with a concrete sense of the types of actions participants hope
to see and whom they believed the key roles might be going forward.
11
-------
Key Afternoon Breakout Discussion Themes
1. Pilot: Identify sites, put projects in the ground, monitor them
2. Funding: What are the sources, innovative, opportunities, etc.?
3. Monitor: Coastal, freshwater (and interventions)
4. Social Science: Identify the right incentives/engagement for citizen action
and education
5. Permitting: Ident fy opportunities for changes and coordination
6. Prioritization: Understand cost/kg for changes of different options
7. Prepare for change: Impact on environmental loading and on effectiveness
of interventions due to climate change, contaminants of emerging concern
(CECs), second order effects, increased development/load
8. Scaling up: What else do we need to know to move scale?
9. Centralizing information and data sharing
Figure 6. Drawing from report-outs after the afternoon breakout sessions, the planning team
identified nine key themes that emerged as priorities and posted this slide for participants.
One week following the workshop, organizers sent personalized thank you letters via email to
participants. One month after the workshop, a draft meeting summary of the workshop was
disseminated to participants (Appendix J). This included the synthesized notes from the meeting
organized thematically in tables, a list of attendees, and the workshop agenda (Appendix B). Participants
were invited to clarify or add any key points from discussion and to indicate if they would like to
continue to be involved with the project. No comments were received. The Consensus Building also
provided a separate meeting summary for Day One that has been incorporated into this evaluation
report (Appendix K).
12
-------
3. Workshop Knowledge Gap Findings
One primary objective for the Problem Formulation Workshop was to "clarify the key knowledge gaps
and relationship/partnership needs related to potential nitrogen removal technologies and the
watershed as a whole." To meet this objective, content analysis of notetakers' notes, posters and
flipcharts produced by facilitators, and sticky notes from participant' activities was undertaken (details
on the collection of the notes, etc. is in Section 2.3). For each breakout group, a one- or two-page table
was produced, listing the research needs distilled from discussions and grouped by emergent themes
(Table 4; Appendix J). Observed themes were created by the evaluators, not by participants. These
themes are relatively high level and are not presumed equal in weight or importance. In many cases,
research needs identified by participants span multiple themes, although they are assigned to only one
dominant theme for simplification purposes.
Table 4. Themes used to group research needs posed by participants in breakout groups. "X" denotes
which themes emerged in the different breakout sessions. The "Watershed (Afternoon Breakout)"
column condenses themes relating to information from all four afternoon breakout groups.
Breakout Grou
p
Research Needs Themes
Shellfish and
aquaculture
PRBsand
fertigation
wells
Restoration
l/A systems
Watershed
(Afternoon
Breakout)
Data Needs
X
Data Sharing/Transferability
X
Data Validation
X
Economics/Market
X
X
Financing
X
X
X
Implementation
X
X
X
Modeling
X
X
X
X
Monitoring
X
X
X
X
X
Pilot/Case Study
X
X
X
X
X
Public Health
X
Regulatory Structure/Permitting
X
X
X
X
Scalability
X
X
X
Secondary Effects
X
X
X
X
Siting
X
X
X
Social Acceptance & Behavior
Change
X
X
X
X
X
Technology Performance &
X
X
X
X
Effectiveness/Compare Alternatives
Watershed-Level Needs
X
Monitoring and Pilot/Case Study were dominant, related discussion themes in all four morning breakout
groups and in the afternoon discussions (Table 4). Many participants vocalized that the time was right to
begin implementing pilot interventions in order to closely monitor those pilots and to better understand
13
-------
the merits, characteristics, and requirements of different alternatives: 'time to put interventions in the
system and monitor.'1 Organizers agreed in the Day Two workshop debrief that participants' call for
pilots was the loudest, unified message they heard throughout Day One. An organizer reported, for
example, that they heard participants saying, 'There's a need for implementation, and to get stuff in the
ground and tested. We're ready to get some stuff cooking.' Another recapped, 'The research/science is
where it's needed to be, implementation is next. This is where the challenges are now.'
Technology Performance & Effectiveness/Compare Alternatives was another primary discussion theme.
Many participants noted the need to compare intervention alternatives for performance and cost
effectiveness. In particular, they advocated for the use of cost per kilogram of nitrogen removed as a
standardized metric, in addition to nitrogen reduction measurements. Several participants also indicated
with comments such as, 'The Cape needs an effective case study,' that pilot studies were needed to help
inform decisions on the most effective approaches to use moving forward. Ideally, these pilots would
function as examples, close to home, that could serve to increase trust among practitioners and
communities that non-traditional interventions could be used to meet nutrient reduction goals.
Towns expressed that there is interest in pursuing non-traditional interventions, but that they would not
be ready to move forward with implementation until they had more information about the
characteristics of different options, and what works best and in what conditions. One municipal
participant commented, 'As the ultimate consumer, we're waiting for "this is good, this is the product,
the characteristics where it is most effective...." We want a data paper done, to best make choices.' This
participant also said, 'There is money for known solutionswe need confidence. We're not there yet....
My community isn't resistant, but they want to do the right thing.' Another municipal participant added,
'Funding can flow if we nail down the priorities...' Managing town staff's, officials', and residents'
uncertainty around alternative nutrient reduction interventions will be important moving forward. It
was also noted by some that towns should not be responsible for shouldering the costs of testing and
piloting interventions.
Siting, piloting, and monitoring are cost- and time-intensive. Some participants noted the lag time
before information from new pilots could be made available and advocated for concurrent work in the
time period while pilots were going in. For example, many participants mentioned the opportunity to
work with partners who are already piloting similar projects. They suggested that monitoring efforts
could be intensified around those projects, and efforts to compile and share information could be
augmented. The message was, 'don't reinvent the wheel.' Efforts to investigate how to translate
information from success stories outside Three Bays and Cape Cod in order to 'demonstrate success
before making the jump to general use,' were advocated as an immediate step forward.
The discussion topic, Social Acceptance & Behavior Change, was another common theme across all
groups. Some pointed out that a lack of social and political will for non-traditional interventions poses a
key barrier to implementation. Others commented about the need to change negative risk perceptions
about different technologies and limited social acceptance of new methodologies. Participants
suggested at different times the need for study of behavioral incentives, investigation of policy and
1 Single quotation marks indicate quotes recorded by notetakers and therefore may be slightly paraphrased.
14
-------
regulatory instruments to motivate segments of the public, identification of social barriers, and/or
education and outreach about a variety of topics related to nutrient management and alternative
technologies. Some argued that a co-benefit of implementing pilots would be to 'use interventions to
increase public engagement, leading to buy in/ for example through establishment of living labs or other
educational installations.
Discussions of permitting challenges arose many times over the course of the day. The use of the term
"permitting" is representative of a number of different policy challenges in the use of alternative
technologies for meeting the ultimate goal of nitrogen reduction. First, each of the technologies will
need local and sometimes state or federal permits for their installation and use. For example, the use of
innovative and alternative septic systems may need building permits as well as permitting and
inspections from the town's health division. Another example, cranberry bog restoration, may require
wetland permits from the town's Conservation Commission and permission from the Massachusetts
Department of Environmental Restoration (MassDEP), the Army Corps of Engineers, or others. In
addition to the permitting for installation and use, there are also permitting challenges related to
receiving credit for the use of alternative technologies to meet TMDL loading reduction for a Watershed
Permit. Watershed permits are a recent effort of the MassDEP to manage nitrogen on Cape Cod. They
are permits filed at a watershed-, rather than town-, level and allow for the incorporation of alternative
technologies, but with strict monitoring and planning requirements. Thus far, only one watershed had
navigated the Watershed Permit process (Pleasant Bay; MADEP 2018).
Permitting needs were raised relating to navigating, updating, or streamlining the permitting processes,
building in credits for nutrient removal technologies (and quantifying nutrient reduction for developing
credit) to the Watershed Permit, and building in flexibility for the use of alternative technologies. Some
suggested that permitting and regulatory structures from other locations on and beyond Cape Cod
might be appropriate for use in Three Bays. Ultimately, identifying the critical permitting needs was
considered a priority in the near term. One breakout group, for example, concluded that developing
permitting for the interventions was needed prior to implementing initial pilot interventions.
Various researchers and organizations have spent more than a decade developing the TMDL models for
Cape Cod estuaries, including Three Bays (often referred to as the MEP models; Howes et al. 2003, Cape
Cod Commission 2015). In the meeting, there was rough consensus on the use of those TMDL models as
the appropriate guide moving forward. There was specific discussion of the desire to not "re-litigate" the
MEP models, and instead, to move forward in trying to meet those predicted load reductions. Modeling
related to the timing and siting of interventions was, however, identified as a need. Participants
discussed the need for a focus on site characterization, including groundwater characteristics and the
advantages and possible configurations for siting multiple types of interventions.
In an open-ended question in the exit questionnaires, participants were asked to identify their top three
challenges for nutrient management in Three Bays. The top challenges listed were funding (63 percent),
permitting (56 percent), and gaining social acceptance (50 percent) (Table 5). This suggests that the
majority of participants who responded to this question felt these factors were fundamental obstacles
to progress in nutrient management.
15
-------
Table 5. Major themes from responses relating to the top three challenges listed by participants in exit
surveys, and their suggested information and/or research needs (N = 32).
Response theme to "What are the top three challenges for
nutrient management in Three Bays?"
Percent of related
responses
Funding
Permitting
Gaining social acceptance/buy-in and changing negative risk
perceptions/inertia/lack of social will/getting moving
Regulatory and interagency coordination; data sharing; develop a
coherent plan
Monitoring/monitoring plan
Getting pilots in the ground/implementation
Prioritization/effectiveness of alternatives/estimates of cost/kg of
nitrogen removed
Incorporating scientific findings/support
Scaling up the use of technologies
Modeling (hydrology, climate change, development)
Economic impacts
63
56
50
28
25
25
19
9
9
6
3
A strong emphasis on piloting interventions was noted throughout workshop discussions and the core
challenges of funding, permitting, social acceptance, and monitoring are also heavily related to the
implementation of pilot projects. These other factors limit not only overall progress in nutrient
management, but also the implementation and monitoring of initial pilots. While it was clear in the
workshop that pilot studies around various interventions are a desired next step, concurrent efforts to
reduce funding, permitting, and social barriers are also likely needed in the short term, both as piloting
begins and to enable scalability of these non-traditional interventions.
In the final, plenary discussion, participants proposed explicit actions or next steps and identified actors
who might be appropriate to play an implementation role. It was emphasized that the objective of the
discussion was not that all of the proposed actions must be achieved. Rather, proposing next steps and
actions helped to clarify, in concrete terms, what participants thought solutions might look like. A
variety of actions and next steps were proposed by participants (Appendix J). Eight of the 20 small
groups proposed pilot projects, typically associated with monitoring activities. A number of specific
locations for new pilots were suggested. A variety of monitoring and public participation efforts were
suggested in conjunction with pilot proposals. Advised monitoring efforts included building knowledge
about baseline data beyond just the sentinel sites developed for the MEP TMDLs; monitoring for
performance effectiveness (nitrogen removed and cost per kilogram nitrogen removed); developing
monitoring plans for the technologies1; and identifying and/or formalizing monitoring parameters and
requirements related to siting and permitting interventions. Participants also anticipated the need for
social acceptance and behavior change research, such as compiling lessons about social acceptance from
1 Note that many of the technologies already have preliminary monitoring protocols that have been developed by
the Cape Cod Commission (Cape Cod Commission 2016). It is unclear whether all participants were aware of this.
16
-------
pilot studies, providing education opportunities, developing incentives, and identifying funding sources
for social acceptance and behavior change research and incentives.
A number of groups suggested actions that were related to compiling data and lessons about the use
and effectiveness of non-traditional interventions from existing pilot research efforts beyond Three Bays
and Cape Cod. One group suggested looking at 'Chesapeake Bay and other places to learn from.' They
argued that compiling research on the social acceptance of different technologies was needed prior to
piloting because 'there would be no sense in piloting a technology that no one would use.' Another
group proposed, 'In order to scale up, we need to provide evidence of the success of promising
practices. We need to compile findings and practices from around the region and share them [here in
Three Bays].' A third group proposed the need for funding to increase monitoring of existing pilots
outside of Cape Cod. They said, 'we need to reach out and tie data in' and also that 'there are a lot of
monitoring programs for existing pilots' but that those programs were not extensive enough because
they were underfunded.
In all discussions about pilot projects, timing was an underlying factor. A sense of impatience to get
started was evident among many. Participants urged an immediate start for putting pilot interventions
in the ground, saying, 'It's time to do something' and 'We just want to get started.' On the other hand, it
was less clear how long it would take to extract meaningful results from pilots. Some brought up the
issue that there will likely be a lag time between pilot implementation and positive, visible results. They
raised the need to learn how best to maintain public interest in funding and supporting alternative
interventions without immediately visible results. The implications for policy making and community
decision making resulting from the issue of lag time from pilot projects are still unclear.
Participants also raised the need to more broadly share information about funding availability,
partnership opportunities, monitoring data, as well as information about technologies, ongoing
research, and pilot projects. This was an important piece identified that could facilitate transferability
and scalability of research and pilot projects and other concurrent efforts in the area. Several suggested
that an information hub or database was needed and pointed to an opportunity to expand on Cape Cod
Commission's Technologies Matrix web tool (Cape Cod Commission 2018). Others suggested compiling
information from outside of Cape Cod to characterize alternative interventions using data about
performance effectiveness, economic impacts, avoided costs, public health impacts, and social impacts.
Five participants requested in exit questionnaires that a website, data sharing, and broad
communications be a focus following the workshop. Developing a data and communications hub could
be one way to meet the long-term workshop goal that work in Three Bays watershed can be transferred
to other watersheds facing similar challenges.
Participants offered few concrete actions related to navigating financing and permitting obstacles, but
many identified these as fundamental barriers to meeting nutrient management requirements.
Participants strongly emphasized the need to calculate the cost of different interventions (specifically
using the metric: cost per kilogram of nitrogen removed); however, several participants also pointed out
that there will be a lot of uncertainty in these metrics. Until better metrics can be produced from the
pilots, one group of participants proposed compiling a list of funding sources and opportunities (e.g.,
federal and state grants) and/or developing innovative financing strategies, such as private investment.
17
-------
Eight participants requested in exit questionnaires that "funding opportunities" be addressed after the
workshop. Relating to permitting, participants indicated the need for support in navigating the present
complex local and state permitting processes. The group also supported investigating the development
of the MassDEP watershed permit, although scaling up pilots to the levels required for the permit may
take some time. For both financing and permitting, a distinction emerged between shorter-term "must
haves" and longer-term "wants." Identifying and addressing funding and critical permitting needs were
"must haves" that are likely more attainable in the shorter term. The cost per kilogram nitrogen
removed metric and the watershed permit were longer-term "wants" and the timeline for achieving
them was uncertain.
Many participants suggested potential roles for their proposed actions, although some did not. Notably,
few participants suggested roles for ORD. This suggests that participants may not be sure what ORD's
role should or can be going forward, in particular as ORD has just begun to pilot the solutions-driven
science approach with increasing stakeholder involvement. This leaves ORD with an opportunity to
identify where ORD researchers might contribute productively, as well as where partners and other
stakeholders can contribute. It also opens an opportunity to dialogue further with stakeholders about
the kinds of research and support ORD can offer, so that in the future, participants will be more aware
of the ways that ORD can be a resource to them and to open new avenues for partnership and two-way
communication.
Overall, the workshop produced rich information outputs including a broadly representative set of the
identified data, knowledge, resource, and regulatory constraints and needs around nutrient
management in Three Bays that will be useful for informing the development of research and action
plans by ORD and partners. Among the stakeholders present in the workshop, there was a burgeoning
sense of impatience to determine whether alternative interventions could help them meet nutrient
management requirements and environmental protection goals. This was tempered by a disinclination
to implement non-traditional interventions due to a range of real and perceived barriers, including
uncertainty around performance, costs, financing, permitting, and social acceptance as well as a lack of
local success stories. A multifaceted strategy that concurrently addresses both shorter- and longer-term
identified needs is likely needed to reduce all perceived and actual implementation barriers in parallel.
18
-------
4. Day One Workshop Evaluation
4.1 Purpose of this Evaluation
This program evaluation of the Problem Formulation Workshop is the first piece of a broader evaluation
of ORD's solutions-driven nutrients management research pilot. This report evaluates the short-term
outputs and outcomes from the workshop to assess what was accomplished by the end of the first day
of the workshop and includes initial investigation of program impacts and how the workshop met ORD's
program goals. It also includes lessons learned from program planning and implementation as well as
from the evaluation itself. Because the workshop is part of a larger pilot project, the ability to reflect on
the production of outputs and outcomes and the lessons learned from this initial workshop will be
important as organizers proceed with planning and implementing other elements of the solutions-driven
research process. A longer-term evaluation of the overall pilot project will include this report but will
primarily focus on all research, communication, and engagement activities occurring after the Problem
Formulation Workshop.
Outputs and outcomes evaluation is useful for facilitating program design and for increasing the
effectiveness of elements of both the program itself (e.g., this workshop) and the overall process around
the program. Evaluation methodologies guide organizers to think critically and concretely about what
they aim to accomplish. Defining concrete outputs for the program (e.g., the information and data they
aim to collect) and expected outcomes (i.e., benefits to participants and organizers) is a valuable
exercise that helps organizers structure the program and identify activities and strategies that will best
facilitate production of those elements. These methodologies facilitate better planning and the
production of higher quality data.
An outputs evaluation also synthesizes and analyzes information outputs, to produce useful and
actionable information for program organizers and participants. This report focuses primarily on the
data outputs that were collectively produced by organizers and participants during the workshop. Those
outputs will be used to inform ORD's longer-term research planning. The report presents evaluation of
whether program activities produced the expected outputs as planned as well as analysis of the
information produced, including a detailed synthesis of discussion from each workshop activity and
identification of themes and commonalities across discussions.
The outcomes evaluation focused on short-term outcomes, i.e., how participating scientists, decision
makers, and other stakeholders benefited by the end of the workshop. This piece of the evaluation
investigated two questions: (1) Was the program constructive for participants tackling and researching
nutrient reduction in Three Bays and on Cape Cod? and (2) What was the workshop's impact on
participating stakeholders and for the EPA? Analysis of outcomes entailed exploration of trust and buy-
in to the research goals by both participants and ORD, as well as identifying research needs critical
opportunities and barriers to effective nutrient management that can shape both ORD's research efforts
and the complementary efforts of stakeholders.
19
-------
4.2 Evaluation Methods
During initial formulation of the workshop concept, organizers drafted program goals, a skeleton
agenda, and a workshop problem statement. In the planning phase, evaluators used these initial
materials to define specific expected outputs and short-term outcomes for the workshop (Table 2, Table
3). These guided the design of the workshop, including the selection of targeted activities that would
best facilitate production of information outputs and outcomes. These defined outputs and outcomes
also functioned as tangible measures by which the program could be evaluated.
As described in Section 2.3, each breakout group was assigned a facilitator and a notetaker. Facilitators
followed a stepwise facilitator agenda so that activities and findings were structured comparably across
groups. The facilitators used a series of pre-made flipcharts and posters to elicit and document key
information outputs from each discussion. Templates were created for notetakers (Appendix I). These
were designed to capture information outputs from the series of activities in each breakout session.
Notetakers were trained in advance and provided with detailed guidance. They were directed to
document the flow of the conversations from the various perspectives represented in each breakout
group, recording participants' own words as much as possible to supplement facilitators' summarized
notetaking. The notetakers also conducted a high-level form of participant observation (Spradley 2016).
They reported on group dynamics, defined in this context as "mood, energy-level, emotions,
communication quality, receptivity, disagreements, understanding, finding common ground, problem-
solving, 'aha' moments, sticking points, or rabbit holes" (Appendix I).
All recorded notes, flipcharts, and posters were transcribed. Document analysis was applied to capture
emergent themes from participant discussions and sort information according to those themes (Patton
2002, Bowen 2009). Limitations to this methodology can include potential for biases in both the
documentation and the interpretation process. Notetakers and facilitators interpret participants' words
through their own personal lenses when documenting what they hear. Independent documentation by
facilitators and notetakers provided a mechanism for reducing bias via triangulation between the two
types of records. Notetakers' documentation of participants' direct quotes was used to cross-check the
summarized and filtered content produced by facilitators and add layers of detail. To account for
interpretation bias by evaluators, all original transcribed notes as well as the synthesized outputs and
thematic classification resulting from document analysis were reviewed by facilitators, notetakers, and
participants from EPA ORD's Atlantic Ecology Division to verify that the outputs reflected what
happened during the workshop. The outputs and thematic classification were also sent to all of the
participants four weeks after the meeting for their review.
Pre- and post-workshop evaluation questionnaires distributed by BCWC were used to measure benefits
to participants (Appendices L,M). Questions were designed around the expected short-term outcomes
defined prior to the workshop (Table 3). Questionnaires included Likert scales to rate activities (range: 1-
5; 1 = disagree strongly and 5 = agree strongly) and a series of open-ended questions to evaluate
participants' knowledge, attitudes, and experiences both before and after the workshop. Exit
questionnaires also provided a means for all participants to be heard equally, as group discussions can
sometimes be dominated by louder voices. Additional open-ended questions were included to collect
20
-------
information about critical needs, opportunities, and barriers around nutrient management.
Questionnaires were developed based on those previously tested in program evaluation (Twichell et al.
2018b, US EPA 2017). Content analysis was applied to the qualitative data to quantify the frequency of
occurrences of key thematic elements (Krippendorf 1989).
4.3 Evaluation Results and Discussion
4.3.1 Outputs-Participants Engaged
In addition to the research needs outputs discussed in Section 3, participant engagement was a notable
output of the problem formulation workshop. Fifty-nine scientists, decision makers, and other
stakeholders participated in the workshop. The participants were identified through past participation in
meetings, research, and work by ORD and BCWC on Cape Cod and in Three Bays. Within that group,
fifteen EPA ORD scientists and managers were also involved in planning, facilitating, observing, and/or
supporting the workshop. Participants included local, county, state, and federal representatives and
comprised of academics, public officials, regional coordinators, technology experts, nonprofit
representatives, and others (Figure 7; Table 6).
©¦ Academic
¦ Consulting
¦ County
¦ EPA non-ORD
¦ EPA ORD
¦ Federal (non-EPA)
¦ NGO
¦ Regional Planning Organization
¦ State
¦ State (non-MA}
Town
¦ Town (not Barnstable)
Figure 7. Breakdown of participants in the Problem Formulation Workshop by role.
21
-------
Table 6. Groups represented at the Problem Formulation Workshop.
¦ Association to Preserve Cape Cod (APCC)
¦ Barnstable Clean Water Coalition (BCWC)
¦ Cape Cod Commission (CCC)
¦ Cape Cod Cooperative Extension (CCCE)
¦ Florida Department of Health, Division of Disease Control and Health Protection
¦ Consulting
¦ Marine Biological Laboratory (MBL)
¦ Martha's Vineyard Commission (MVC)
¦ Massachusetts Alternative Septic System Test Center (MASSTC)
¦ Massachusetts Department of Environmental Protection (MassDEP)
¦ Massachusetts Department of Fish and Game, Division of Ecological Restoration
(MassDER)
¦ Quantified Ventures
¦ The Nature Conservancy (TNC)
¦ Town of Barnstable
¦ Town of Falmouth
¦ Town of Yarmouth
¦ U.S. Environmental Protection Agency (U.S. EPA) Office of Research and Development
(ORD)
¦ U.S. EPA Region 1 (New England)
¦ U.S. EPA Region 2 (New York, New Jersey, Puerto Rico, US Virgin Islands)
¦ U.S. EPA Region 4 (Southeast)
¦ University of Massachusetts Amherst (UMass Amherst)
¦ University of Massachusetts School of Marine Science & Technology (SMAST)
¦ United States Geological Survey New England Water Science Center (New England WSC)
¦ Woods Hole Oceanographic Institute (WHOI)
¦ Woods Hole Research Center (WHRC)
The plurality of participants was from EPA ORD and EPA Regions 1, 2, and 4, as the workshop required
staffing with enough facilitators and notetakers to adequately extract and document information
outputs. EPA's high participation also reflected the intention that the workshop serve as a listening
opportunity for EPA to more effectively shape research planning around stakeholders' needs.
4.3.2 Short-term Outcomes
This section of the evaluation focuses on short-term outcomes, i.e., the benefits to stakeholders and to
organizers and the EPA by the end of the day. It includes evaluation of whether the workshop was
constructive for participants and organizers as well as initial assessment of the program's impact. Pre-
and post-workshop questionnaires and participant observation were used to evaluate outcomes. Out of
the fifty-nine total attendees, thirteen were EPA and partners on the organizing team. Organizers were
not asked to fill out evaluation questionnaires. Twenty-nine of the forty-six workshop participants (forty
non-EPA participants, and six EPA scientists who were invited as participants) responded to the pre-
workshop questionnaire and thirty-six responded to the post-workshop questionnaire.
22
-------
4.3.2.1 Benefits to Workshop Participants
Overall, participants were satisfied with the workshop. A majority of participants who filled out an exit
survey agreed or agreed strongly that the workshop was constructive (83 percent, median Likert
response = 4 (Agree)) and achieved its goals (80 percent, median Likert response = 4 (Agree)) (Table 7).
Participants were satisfied that the workshop reflected broad, multidisciplinary participation (94 percent
agreed or agreed strongly, median Likert response = 4 (Agree)). Ninety-one percent agreed that all
voices were heard over the course of the day (median Likert response = 5 (Agree strongly)), which
suggests that the strong majority of participants also felt their own voices were heard and that activities
were structured in such a way that effectively enabled inclusive discussion. Organizers felt that 'People
were interested in the issues,' 'People were engaged, were trying,' and 'People were more open than
expected.' One organizer said they had 'expected more disagreement [between stakeholders]' and
observed that 'there was a willingness to move difficult issues' that was surprising.
Table 7. Responses about participants' overall experiences in the workshop
(Likert scale: 1-5; 1 = "Disagree strongly" and 5 = "Agree strongly").
Percent
satisfied*
Median
Mode
Response
Range
N
Overall, the workshop was
constructive
83
Agree
(4)
Agree
(4)
3-5
31
The workshop achieved its
goals
80
Agree
(4)
Agree
(4)
3-5
30
The workshop reflected broad,
multidisciplinary participation
94
Agree
(4)
Agree
(4)
3-5
33
All voices were heard during
the workshop
91
Agree
strongly
(5)
Agree
strongly
(5)
1-5
33
*Percent satisfied = Percent "agree" (4) plus percent "agree strongly" (5)
Several themes were evident among exit questionnaire responses to questions, "Overall, what did you
get out of today's workshop?" and "What was the best part of the day for you?" Table 8 quantifies the
number of responses relating to each theme. Overall, participants felt the workshop provided a valuable
learning opportunity. The majority of participants reported that they gained an increased understanding
of the nutrient problem in Three Bays and Cape Cod, including the "multifaceted nature of problem" and
the "complexity and scale of the problem."3 Participants also reported a broader sense of who is
working on these issues and the different work being done: "much better appreciation of work being
done on state, MA, and federal level" and "better understanding of priorities from stakeholders engaged
in Cape Cod water quality." An organizer noted that due to the large amount of information exchanged
during Day One, 'a lot of research questions can now start to be asked.' Thus, one of the biggest benefits
3 Double quotation marks indicate direct participant quotes from the evaluation questionnaires, as written.
23
-------
to both participants and organizers was an increased and broadened understanding of the different
aspects of nutrient loading issues in Cape Cod along with a better sense of the community of
stakeholders collectively working on these issues.
Table 8. Major themes from responses to "Overall, what did you get out of today's workshop?"
(N=36) and "What was the best part of the day for you?" (N = 36, participants mentioning the same
theme in both questions are counted once) (themes are not direct quotes).
Theme # of related responses
The workshop increased and/or broadened my overall
understanding of the problem, players, current efforts, and needs
The workshop was an opportunity for networking and
relationship-building
The workshop moved the ball forward for Three Bays nutrient
management
We are ready to put pilots in the ground in the Three Bays area
A second impact of the workshop was that it provided a valuable networking opportunity (Table 8). It
brought together most of the key stakeholders contributing to research or decision making in nutrient
management in the Three Bays area, as well as a number of experts from state and federal agencies and
other regions of the U.S. A participant reported, "Good cross section of stakeholders, great opportunity
to build relationships and identify needs." Another said, "I think today was helpful in moving the ball
forward in our discussions. Useful to hear from other experts w/ varying viewpoints." Organizers
observed a 'strong willingness to collaborate [among participants].' Feedback thus indicates that a key
expected short-term outcome of the program, to foster communications between a diverse group of
stakeholders, was met. This outcome is an important incremental step towards organizers' goal to build
relationships and develop partnerships and networks within and beyond the Three Bays watershed.
The workshop overall may have increased positive attitudes about the use of alternative technologies to
meet nutrient management requirements and environmental protection goals. Before and after the
workshop, all participants were asked the same question: "On a scale of 1 - 10, how likely do you think
Three Bays is to implement alternative technologies to meet their nutrient management needs (1 = 'Not
going to happen' to 10 = 'Definitely will happen')?" The rating before the workshop was lower (median =
7, range = 4-10) than after the workshop (median = 8, range = 5-10); however, values for responses
before and after the workshop were only significantly different within a confidence level of 90% (p =
0.056, Mann-Whitney U test). This result indicates with some uncertainty that the workshop may have
positively impacted participants' perceptions about the attainability of nutrient management goals
through use of alternative technologies. It suggests a possibility that participants felt more hopeful and
galvanized about research directions and progress toward meeting their nutrient management needs
than they did at the beginning of the day, which was one desired outcome of the workshop.
With nearly sixty participants, the breakout groups were important for fostering productive discussion.
In the morning, the four breakout groups ranged in size from eight to fourteen participants. Afternoon
breakout sessions ranged in size from seven to fourteen participants. Twenty-three participants
19
18
5
5
24
-------
reported the breakout groups were the best part of the day. A strong majority agreed or strongly agreed
that the morning breakout group was constructive (94 percent). A smaller majority agreed or strongly
agreed that the afternoon breakout group was constructive (66 percent) (Table 9). Overall ratings for
both breakout sessions were positive (morning breakout median/mode: 4/4; afternoon breakout
median/mode: 4/4). Some participants preferred the technology-focused discussion, and some
preferred the watershed-level discussion. This reinforced that both breakout discussions had utility for
different sets of participants. In general, breakout sessions enabled good conversation among a diverse
group of participants. A participant reported, "I really enjoyed working with groups from many different
agencies etc. and seeing how they see the issues and what they are doing on these issues."
Table 9. Responses about participants' experience in activities in the workshop
(Likert scale: 1-5; 1 = "Disagree strongly" and 5 = "Agree strongly").
Percent
Response
satisfied*
Median
Mode
Range
N
The opening statements were
a good use of time
84
Agree
(4)
Agree
(4)
3-5
31
The morning breakout group
was constructive overall
94
Agree
(4)
Agree
(4)
3-5
34
The afternoon breakout group
was constructive overall
66
Agree
(4)
Agree
(4)
2-5
32
The final, "Taking it Forward
Agree
(4)
Agree
(4)
Discussion" was a good use of
time
66
1-5
29
*Percent satisfied = Percent "agree" (4) plus percent "agree strongly" (5)
In most discussions, observers noted that participants sometimes disagreed about some of the knowns,
unknowns, and research needs for Three Bays. These interactions appeared constructive, rather than
confrontational, indicating that the atmosphere fostered in the workshop was low-conflict. Participants
appeared secure expressing their viewpoints. In the morning breakout session, for example, participants
were provided a handout to spark discussion, produced by ORD and BCWC, detailing what was known
and unknown about the non-traditional intervention of focus for each group. One participant in a
breakout group responded to the content in the handout, saying, 'We know a lot of the "need to know"
part, and don't know a lot of the "what we know" part,' suggesting some disagreement with information
in the handout. Another participant argued that the unknowns were accurate. Such conversations
indicated that participants were willing to share opinions and were comfortable disagreeing with both
one another and with the information provided by EPA and BCWC. This was a positive indication that
participants felt empowered to speak freely.
The final, plenary discussion was designed to meet a workshop goal to identify potential research
contributions and approaches for key workshop partners in resolving the identified knowledge gaps. The
objective of the discussion was not that all of the proposed actions must be achieved, or organizational
25
-------
contributions promised; rather, proposing next steps and actions helped to clarify, in concrete terms,
what participants thought solutions might look like. This was intended to be helpful to organizers for
planning next steps following the workshop. Some participants reported that they enjoyed the "focus on
solutions" in this activity, as well as the "identification of actions toward progress of this time [sic] and
issues blocking progress in this regard." Another commented that the "final discussion was too abstract.
Even though the request was concrete, it seemed too pie in the sky." An organizer expressed some
frustration that, "The groups tended to get too detailed about particular issues, without getting specific
enough about action steps/roles." Designed more as a brainstorming session than a planning session,
this final plenary will likely serve as just one of many similar exercises that will need to be conducted
going forward in smaller, focused planning groups around specific research or specific pilot intervention
projects. Smaller planning groups will be able to more productively and iteratively refine and flesh out
the details of specific projects, assess the attainability of those projects, and coordinate how partners
will contribute.
4.3.2.2 Benefits to the EPA
Day One of the Problem Formulation Workshop provided a useful opportunity for EPA scientists and
management to listen to and interact directly with stakeholders. With limited opportunities to interface
in-person with stakeholders, the workshop provided a space for EPA personnel to directly meet with the
network of practitioners, scientists, and experts working in nutrient management on Cape Cod, learn
about their perceptions of critical research needs, ask follow-up questions related to stakeholder
insights, and observe the dynamics of the group. One staff member noted, for example, 'I was
encouraged by the comradery [among participants]. Already, there is a great network foundation
established.' The focus-group format provided a low-conflict setting which fostered open conversation
and supported exploration of potential partnerships and relationship-building. An organizer observed,
'People were more open than expected. [We saw] collaboration opportunities, thinking outside the
box.'
The workshop also served to demonstrate ORD's commitment to solutions-driven research. Observers
heard several participants note that the large number of EPA personnel attending the workshop
reflected, to them, a serious intention within the EPA to listen and learn from stakeholders. Follow-up
efforts beyond the dissemination of meeting notes to participants will be needed to further validate
ORD's engagement-focused approach to research planning.
The Problem Formulation Workshop was a pilot effort to apply, test, and evaluate one element of a
solutions-driven research approach. One key goal of stakeholder engagement in problem formulation is
to increase understanding of a variety of perspectives and of the knowns and unknowns around a
particular issue. Part of solutions-driven research is finding the friction points or barriers that prevent
science from moving into practice. Those friction points can be directly related to research practices,
e.g., how to scale from a research pilot to operational best practices, or they can be external, e.g., lack
of funding, regulatory hurdles (permits). Many of these friction points can be identified and worked
through by collaborating with stakeholders via a solutions-driven research approach that involves them
throughout the planning process. The workshop provided an effective opportunity to collect information
26
-------
about needs, barriers, and opportunities before initiating further research planning. High-quality
information was documented and distilled into operable outputs useful to participants, other
stakeholders, partners, ORD management, and ORD scientists. In addition, progress was made in the
workshop to clarify the types of research outputs that would best inform development, implementation,
and evaluation of solutions.
Working closely with partners facilitates research planning and the translation of results in ways that
make them immediately applicable to stakeholder decision making and to nutrient management
activities. Overall, the positive reception by participants and the successful production of research
planning outputs underlines the success of this initial effort. The workshop also provided an opportunity
to ORD to extract lessons learned from solutions-driven research that can be shared with others using
similar approaches.
4.4 Lessons Learned
This section compiles lessons learned from the Problem Formulation Workshop, based on both
participant feedback and observations from organizers, facilitators, observers, and notetakers. Elements
were identified that contributed to the successful design of the workshop. Other elements were
ascertained that could be improved to increase the impact for participants and organizers in future
similar programs. This section also compiles lessons learned about the value of conducting a program
evaluation in conjunction with workshop planning, implementation, and documentation.
4.4.1 Lessons for Workshop Design
Participants appreciated the opportunity to interact with a variety of stakeholders from different
disciplines. The broad target audience of the workshop created valuable learning opportunities.
Breakout groups promoted constructive discussion, and effectually produced the desired information
outputs. The use of a mixed format facilitation structure that allowed for some flexibility (for example,
the option to use multi-voting in group prioritization exercises) enabled facilitators to adapt exercises to
their individual groups while maintaining comparability across groups.
Organizers found, and some participants commented, that the physical set-up of the room sometimes
made it difficult to hear one another in breakout groups. Observers noted that smaller groups had less
difficulty hearing one another and conversation was more natural. In the future, organizers may want to
plan smaller breakout groups of six to eight people or reserve a number of separate rooms for breakout
sessions. This is limited by the ability to secure enough facilitators and notetakers for each smaller
breakout group. A few participants also commented that they would have liked the ability to switch
groups and to switch groups more often. Breakout activities were longover an hour each. Afternoon
sessions might be shortened or interspersed with physical activity to be more compatible with
participants' dwindling attention spans.
Some participants commented that the focus on solutions and research needs did not fully incorporate
recognition of current efforts in the Three Bays area. A participant noted that organizers "could have
improved integration of already completed efforts (e.g., MEP models, Cape 208, etc.). It is important
27
-------
that engagement mechanisms such as this workshop adequately recognize current efforts. People are
more likely to trust efforts that recognize and are well-informed about participants' contributions
(Fischer 2000). The opening statements and round-robin introductions within small groups may be good
places in the future to more explicitly highlight and acknowledge participants' efforts and the broader
efforts in the area of focus. Organizers did endeavor to do this in the workshop; however, there
remained room for improvement. Alternatively, use of a pre-workshop webinar or an explicit activity
designed to highlight or compile different perspectives and experiences can help get participants
engaged and feeling like their experiences are validated. For example, in a previous problem formulation
workshop, EPA Wildfire Smoke and Health Risk Communication Workshop, the first day included a series
of short "perspectives" presentations that allowed representatives of different stakeholder groups to
highlight what was most important to them and why, building on their previous experiences (US EPA
2017).
4.4.2 Utility of Evaluation to Increase Program Impacts
Overall, one primary takeaway from this program evaluation must be the clear value of structure.
Deliberate efforts were made to provide structure for the planning and design process, for
implementation of the workshop, and for documentation. Structuring techniques improved
communication between planning team members, resulted in a workshop that was appropriately
designed to meet organizers' goals, increased the efficacy and comparability of facilitators across all
breakout groups, produced higher quality data, and improved measurement of workshop success and
identification of lessons learned. In addition, the value of social science techniques was evident.
Producing operable information from the large volume of raw data co-produced by participants and
organizers in the workshop was necessary if that information was to be shared with participants and
referenced by partners and EPA going forward.
Below, several key elements used in the program evaluation are highlighted as they were crucial for
increasing the success of the workshop and the impacts for participants and organizers:
1. Refine expected outputs and outcomes and design activities to produce those results. Setting
aside time before the workshop to define expected outputs and outcomes helped to formulate
the structure of workshop activities, increase the likelihood that the workshop would help
organizers meet longer-term goals, and strengthen the program's likelihood of success for all
participants (Chen 2014). Early in the planning process, two to three open-ended discussions
were planned in the workshop. As information outputs were refined, organizers iteratively
narrowed the focus of activities and modified the format to better produce higher quality
information that would be operable for organizers after the workshop. Defining concrete
outputs and outcomes also improved ease of communication between organizers, evaluators,
and contracted facilitators from CBI during the planning phase, by defining agreed-upon
objectives for Day One.
2. Design activities around compiling research needs, barriers, and opportunities, and plan to distill
and translate that information into research questions after the workshop. Organizers noted
some surprise about the difficulty of extracting specific research questions and research projects
28
-------
out of the discussions. One organizer noted, for example, 'I learned it was hard to cast things as
research questions.' The time constraint of the one-day workshop, combined with the range of
technical expertise among participants, made it infeasible to narrow down discussion to the
level of developing specific research questions. As another organizer noted after observing Day
One, "We need to know problems and hurdles. Then we come up with research questions."
3. Provide stepwise facilitator guidance and train facilitators in advance. A detailed and stepwise
facilitator agenda containing common discussion prompts and guiding questions (Appendix H)
helped facilitators run structured activities that were replicated, in a rough sense, across groups.
The agenda was paired with a series of flipcharts and posters that served to visually guide
discussion, for both facilitators and participants, so that information was co-produced as
planned and documented in an organized fashion (Appendices D-G). For future efforts, providing
more extended facilitator training would further improve comparability of activities across
groups and the production of high-quality data.
4. Develop structured notetaking templates that align with the activities to support higher quality
documentation and train dedicated observers in notetaking. Appointing dedicated notetakers as
observers for each breakout group was valuable. Notetakers were provided with training and
guidance (Appendix J) prior to the workshop to help them capture key information outputs in
their notes. Notetaking templates were designed to mirror the stepwise progression of activities
and flipcharts/posters used by facilitators (Appendix J). As much as possible, notetakers were
directed to record discussion in participants own words that could be used to contextualize
facilitators' summarized notes. This enabled triangulation between facilitator and notetaker
notes in order to reduce observer bias.
5. Set aside time immediately after the workshop for transcription and review of field notes. All
facilitator and observer notes were transcribed within a week of the workshop. These were
shared with facilitators, observers, and ORD participants to check for accuracy and to add
missing detail. Immediate review of the notes was critical for increasing the accuracy of the data
before memories of the workshop faded.
6. Use document and content analysis to produce transferable and operable information outputs
from the raw data (observer and facilitator notes). Analysis and synthesis of the large volume of
raw data from the workshop was necessary to produce operable and meaningful information for
research planning going forward. The facilitator and observer notes were not usable as is. It was
important that time was set aside to analyze the notes and create condensed and digestible
information outputs. Document and content analysis enabled systematic distillation of key
information from raw qualitative data (Patton 2002). The results (Appendix J) proved useful for
sharing with partners, higher-level managers, and participants to demonstrate what was
learned, collect additional feedback, and fuel retrospection, deliberation, and planning. Sharing
detailed meeting notes with participants also demonstrated that their voices were heard and
that the information they provided was valued.
29
-------
5. Day Two Meeting Synopsis
5.1 Day Two Meeting Overview
The organizing committee, several ORD scientists, and a number of members of ORD upper
management met on Day Two for a half day to debrief the Problem Formulation Workshop and initiate
discussion on how to move forward based on what was learned in Day One. As shown in the initial logic
model (Figure 3), the intention of the first day of the workshop was to identify research needs and
possible collaborators. This workshop was intended as an early step in a larger pilot to provide solutions-
driven research to stakeholders. This entails incorporating stakeholder-identified needs into ORD
research planning and undertaking appropriate collaboration and communication with relevant experts
and stakeholders. The goals following Day One (includes Day Two and beyond; Table 10) broadly relate
to ORD and BCWC beginning to develop an approach to refining actions and determining key next steps.
Table 10. Goals for ORD and BCWC following Day One (includes Day
Two and beyond)
Clarify key research needs, and themes
Identify additional questions ORD and BCWC still have
Develop research questions around key research needs
Begin to develop an action plan for implementing this research
Day Two participants initiated the debrief of the Problem Formulation Workshop by reflecting on what
they learned or were surprised by during the workshop. Participants noted that they were pleasantly
surprised by the level of overall comradery, dedication, and willingness to participate. They were also
happy to see that participants were satisfied with ORD's broad initiative to involve stakeholders in
research planning and many expressed early interest in participating in future research planning and
implementation.
It was noted that much still needs to be done to pilot non-traditional interventions, obtain adequate
funding for monitoring, and enable scaling-up from pilot phases to watershed-level implementation. It
was also noted that ORD's next step would need to be to flesh out research questions and identify roles
for ORD based on what was heard. Discussion identified three phases of a broad roadmap forward:
Phase One: Interim (roughly three months) - "Clarify the overall strategy."
Phase Two: Piloting - Implement demonstration projects with research and monitoring.
Phase Three: Scale Up - Clarify what was learned and develop a detailed approach to meet
TMDLs in Three Bays.
30
-------
Throughout these phases, ORD and BCWC hoped to integrate stakeholder engagement through smaller
planning and working groups to keep stakeholders informed; leverage and coordinate resources and
talents; solicit input; and refine research questions and clear actions.
5.2 Systems Mapping Exercise
Two types of systems mapping were piloted as methods to document the diverse set of concerns,
relationships, and partners articulated in the Problem Formulation Workshop. Reviewed in discussions
on Day Two, these maps helped to highlight some of the key discussion threads and important needs
and research gaps. Systems mapping can provide a way to encourage the generation of integrated
hypotheses and identify opportunities for inter- and transdisciplinary research, as well as individuals and
institutions that may contribute to that research (Okada et al. 2008). Visual maps can help team
members feel engaged and participatory in building a shared theory and feel a sense of shared
ownership for resulting hypotheses (Heemskerk et al. 2003). Two researchers used the online platform
Plectica (www.plectica.com), a tool for structuring information about a system that codifies specific
visual terms (called grammar). The mappers independently used Plectica to capture and organize the
relationships among discussion participants and components in visual maps.
The first mapping effort focused on linking organizations and individuals with specific research and
analytical needs. To structure this information, the visual grammar underlying Plectica, known as DSRP
(Distinctions, Systems, Relationships, Perspectives; see Cabrera and Cabrera 2015), was used to visually
link actors to their needs through action verbs. For example, "organization A (actor) has responsibility
(verb) for the nitrogen load which is informed by monitoring data (need)." These types of systems maps
help to explicate how stakeholders are connected to their highest priority needs. The simplified map
shown below was developed from the perspective of one ORD researcher who observed two breakout
sessions during the meeting (Figure 8). Although beyond the scope of this evaluation, this type of map
could be iterated with others to promote a common understanding of the 1) outstanding questions,
obstacles and opportunities, 2) actions to be taken, and 3) relationships among actors as each works to
advance project goals. Structuring these ideas in maps can quickly identify discrepancies in how group
members perceive the same situation. Resolving these discrepancies can help participants develop a
shared vision of the path forward.
31
-------
Three Bays Watershed
Institutions in attendance
Other institutions
hydrology
nitrogen load
~
monitoring <
watershed permit
1
integrates/allows credit for
*
Technology
p. * PRB
~ a IA septics
* Aquaculture
O Restoration
~ ...
Fertigation
Figure 8. One Participant's Map of Day One of the Problem Formulation Workshop. Arrows define
relationships among components. Carets indicate that subsystems within the map containing additional
detail have been collapsed (up-pointing caret) or expanded (down-pointing caret). Gray dots indicate
the number of nested parts. The eye symbol indicates that a perspective has been articulated, and
colored dots serve as place-holders for perspectives on nested components. The expanded map (shown
in detail in Appendix N) can be viewed in a number of ways by focusing on different participants'
perspectives.
The second systems mapping effort applied dialogue mapping, which graphically organizes dialogues
around complex problems (Conklin 2008). Using the Issue Based Information System (IBIS), dialogue
mapping translates the linear and temporal structure of a conversation to an issue- or topic-based
structure by linking a series of related ideas and questions (Kunz and Rittel 1970). In two breakout
sessions, a second ORD researcher tied participants' comments to the root questions from each activity:
(1) "What are broader systems level key research needs/questions?" and (2) "Given we are ready to
implement and monitor, how do we get moving?" Ideas and new questions that arose were linked to
the root questions or identified as new topic areas. Arguments were color-coded if they supported (pro)
or refuted (con) other comments. The result was an "issue map" (Conkiin 2008) that captured
participants' discussion using a topic-based structure (Figure 9).
32
-------
Modeling for Implementation of
scenarios
How to think about future concerns
now?
Balancing source level removal
strategies against downstream
removal
What is best way of addressing
multiple goals (holistic) nutrient,
groundwaters, CEC, hazard
resilience
How long will it take to see
impacts?
How to optimize technologies
Can we identify N hotspots?
What are broader systems level key
research needs/questions?
Natural systems in watershed are
altered by N addition/N sources +
water uses
What are the Impacts of climate
change on N loadings and algae?
How does climate change Impact the
efficacy of technologies' Role of
modeling
Monitoring of discharges and water
quality
Can affect efficacy of barriers
Accurate model for ground and
surface water coupled with short
and long term monitoring
How can we cost effectively
monitor Impacts?
Dont need modeling of watershed
level impacts.
Do need for evaluating Individual
technologies
What are the science and
regulatory questions that need to
be addressed to get the permits?
What are the dealbreakers?
How do you renaturalize watershed
functions at scale?
What are system level costs?
Water body report cards Sept 2019
Association to preserve cape cod
Given we are ready to implement
and monitor, how do we get
moving?
How to overcome the permitting
issue?
People make the final choices, not
the technologies
Competing priorities, e.g. ESA
present well vetted options, not
wasting their money
Better data visualization tools
can help communities understand impacts
and momentum in solving the problem
Focus on sites close to the bay or
river, identify the highest
contributing sites, implement l/A
systems on high contributing sites
Regulatory change to encourage adoption
Nitrogen cap and trade or tax
program?
Need landowner permission
Need more demonstrations of the
effectivness of new atterantive
technologies
address cost, unintended impacts
Need parameteriztlon in terms of
removal rates
Lack of political will, focus is on
increasing housing base
Look outside watershed to
understand Impacts of cranberry
bog restoration
Data validation
EPA can help set standards for monrtonng
Cape wide Oyster cooperative?
EDF investigating private funding
Economic development vs N
management?
List of model regulations
Look at FL model - DEP and Health
Dept.
springs now. coastal and estuaries later
Impact of seasonal residency on N
loadings
Is coastal N contamination a public
health issue?
Impact fees are there at a regional
level - offsets are available to
offset new N sources
Figure 9. Example dialogue map from the session "Concurrent Discussions Part 2: Putting it all Together,"
created with Plectica. The colors indicate the different grammar elements. Questions are purple, ideas are
light blue, "pro" arguments are green, and "con" arguments are red.
33
-------
6 Next Steps
As the Problem Formulation Workshop is part of a longer pilot effort, participants were asked to provide
feedback about next steps they wish to see, to inform the next phases of the pilot. Many stakeholders
indicated interest in continuing collaboration with ORD during ongoing research planning.
Two of the short-term outcomes that organizers wanted to achieve in the Problem Formulation
Workshop were that (1) participants feel that research needs are better understood and more clearly
defined as a result of the workshop, and (2) participants feel that research directions developed
following the workshop are useful to them and to other stakeholders. Achieving these outcomes may
depend on how organizers choose to communicate with participants about research directions following
the workshop. Participants expressed interest in receiving notes from the meeting and providing
feedback (Table 11). Clearly communicating the research needs distilled from the workshop and making
them available to participants for feedback are critical first steps to demonstrate that participants were
listened to and that the priority research needs extracted from the workshop (a) accurately reflect what
participants said, (b) are clear and transparent, and (c) are useful to them. Organizers chose to share the
condensed notes with participants and requested feedback on those notes (Appendix H). A number of
participants also expressed interest in participating in ongoing small planning groups and in continuing
to network and engage with partners; therefore, participants were asked to respond if they wish to
continue to collaborate or contribute.
Table 11. Participant responses to the question, "How would you like to receive
information about research efforts going forward?" indicate interest in future
collaborations.
Near Term
Email updates (6 participants, all other comments represent 1
participant)
"Please share notes from the meeting with group"
"Opportunity to contribute and comment"
Future Term
"More opportunities to network with some of the folks at the
workshop would be good"
"1 would participate in the working group, calls"
"Continue to engage local/regional partners as plan is formulated"
"Might include ongoing groups in planning"
"Share draft plan for comment/partnership"
"Funding and partnering mechanisms"
Given the broad range of information co-produced in the workshop, ORD's next step will be to focus on
identifying the research approaches and areas to which ORD scientists can contribute most productively
in order to help achieve nutrient management and environmental protection goals on Cape Cod. As
34
-------
observed in the final plenary discussion, stakeholders did not identify a clear role for ORD on many of
the projects that were suggested. This is an opportunity to demonstrate the value that ORD can provide
and also to coordinate research done by ORD scientists, partners, and other stakeholders. Further
dialogue and planning within smaller planning groups will be important to refine and orchestrate the
details of specific projects and approaches and determine who contributes and how. There will also be a
longer-term opportunity to educate partners and other stakeholders about the kinds of research and
support ORD can offer, with the goal of building greater awareness about the ways in which ORD can be
a resource. Ideally, one of the benefits of initiating this solutions-driven research process will ultimately
be to open new avenues for partnership and two-way communication about the co-production of
useful, relevant, and applied research efforts.
In addition to establishing smaller planning groups in the near term, organizers will need to meet
individually with key stakeholdersi.e., those who have both high power and high interest in nutrient
management decisions (Susskind et al. 1999)so that those stakeholders are adequately heard, and
their interests represented. Organizers met with state representatives in advance of the workshop;
however, they were unable to meet with town representatives. One participant noted, "EPA needs to
talk to the town on what plans are in place, today ignored that there is a plan in process by the Town."
This suggests some dissatisfaction that the towns' perspectives were not fully heard and recognized. In
the future, pre-workshop, one-on-one meetings with key stakeholders are recommended so they know
what to expect from the workshop and they have an opportunity to communicate how the workshop
might benefit them. Key town stakeholder meetings are still possible after the workshop has happened,
and this is an important next step given their key role in implementing nutrient management.
Participants recommended a range of additional roles that they believed should be included in a broader
communications and outreach strategy, ranging from the governor of Massachusetts to various specific
public groups (Table 12).
Table 12. Participant recommendations of roles not represented at the workshop
that they believe should be engaged going forward.
Stakeholders
Governor of Massachusetts/legislators/congressional staff
Elected officials/local officials/Town Council for Barnstable
Conservation Agents and Commissioners
"Permitting agencies who are blocking progression"
"People who write RFPs [requests for proposals] for proposals and
grants"
Citizen leaders
Suffolk County, NY
Buzzards Bay Coalition
Publics
Landscape professionals
"Educators, e.g., teachers"
"Let the public know how hard people are working on the problem"
35
-------
Conducting a structured stakeholder mapping exercise may be a useful activity going forward, to plan
for effective and strategic engagement among all types of stakeholders (Figures 10 and 11). Stakeholder
characterization can help to illuminate both how different groups might be engaged as well as how to
more effectively leverage value from them (Shirey 2012). Given that ORD's research planning will
involve coordinating and aligning with research efforts and engagement among a variety of key
stakeholders and partners, this exercise can help to structure consideration of ORD's role and others'
roles in producing research and in engaging with different sectors. In particular, consideration of which
partners are best situated to engage different groups can help to maximize efficiency and increase trust
in engagement and outreach efforts. For example, towns may be better suited to engage their public
sectors than ORD is. Additional maps may be formulated to elucidate engagement and communications
strategy in unique phases of the process as well as to explore the potential roles of different partners in
producing research and engaging different groups of stakeholders around that research.
Developing a structured approach to planning and coordinating the above-mentioned next steps is
recommended for organizers as they move to develop a multidisciplinary research and action plan for
ORD and other researchers that addresses critical research gaps and produces transferable results to
decision makers in Three Bays and other watersheds. The initial model connecting short-, medium-, and
long-term outcomes (Figure 3) should be refined and expanded now that the workshop has concluded.
Next steps should involve identification of incremental actions and activities (outputs) linked with
expected outcomes. Mapping these incremental steps can help organizers be more deliberate in their
planning to build from the information produced in the Problem Formulation Workshop. This will also be
important for the longer-term evaluation of the uses of workshop information in planning, research, and
engagement efforts by EPA, partners, and stakeholders following the workshop, as well as the outcomes
that result from the overall solutions-driven research pilot
Ultimately, developing a comprehensive approach to nutrient management in Three Bays and other
watersheds and meeting TMDLs and water quality goals will require a coordinated and structured effort
among scientists and local, state, and federal level decision makers and likely, implementation of
watershed-scale approaches. Developing a better understanding of the kinds of interventions that will
most effectively remove nutrients from the system, which are most cost-effective, and which are most
socially acceptable will not only benefit the Three Bays system and the communities that live there, but
also may benefit other watersheds on Cape Cod and elsewhere in implementing successful nutrient
mitigation efforts.
36
-------
s_
MA Governor
Other * MassDEP Towns
O)
>
permitting Cape Cod Commission
>
o
CL
Grant writers
agencies
i k
County Health
MANAGE, BUILD INTEREST (PARTNERS ENGAGE?)
PARTNER WITH, ENGAGE/CONSULT CLOSELY
Citizen leaders
Consultingfirm NGO
Landscape professionals
Academics
Homeowners
ENGAGE/INFORM THROUGH PARTNERS
CONSULT, COORDINATE EFFORTS, DELEGATE TO
Interest
Figure 10. Sample stakeholder map of workshop participants and other stakeholders (adapted from
Twichell et al. 2018a). In blue text, suggested engagement strategies are based on the stakeholders'
level of power and interest in each quadrant. Key stakeholders (blue-shaded quadrant) have both high
power and interest in nutrient management, and thus require the highest-intensity engagement.
MANAGE, BUILD INTEREST (PARTNERS ENGAGE?)
White papers
Guidance materials/support
Website
Web tools
Fact sheets
PARTNER WITH, ENGAGE/CONSULT CLOSELY
Periodic 1-on-l meetings, calls
Focus groups
Advisory committee
Multi-stakeholder dialogue
PARTNERS ENGAGE/INFORM
Focus groups
Website/clearinghouse
Newsletters/fact sheets
Media coverage
Webtools/how-tos
CONSULT, COORDINATE EFFORTS, DELEGATE TO
Focus groups
Science advisory committee
Multi-stakeholder dialogue
Online research hub/clearinghouse
Share research plans, request feedback
Build partnerships
Listservs
~ Interest
Figure 11. Sample stakeholder map of specific potential engagement techniques appropriate for the
stakeholders falling in different quadrants of the paired Figure 10 map (adapted from Twichell et al.
2018a). The premise of this map is to match targeted engagement techniques to different quadrants of
stakeholders. Where indicated, partners might be better suited than ORD to undertake engagement and
outreach for some groups.
37
-------
7 Conclusion
The Problem Formulation Workshop provided an important opportunity for EPA to interface with
scientists, technical experts, and decision makers working on nutrient reduction to discuss knowledge
gaps and partnership needs related to possible solutions for nonpoint nutrient management in the
Three Bays watershed. Among the outputs were strong needs identified for pilot projects on Cape Cod,
monitoring, permitting and funding support, technology characterization and performance assessment,
behavioral change and social acceptance research, and a centralized online hub for information sharing.
A number of research areas were identified that can inform research planning and move the science
toward meeting long-term goals to significantly reduce nutrient loading to surface waters and coastal
embayments and achieve water quality goals in Three Bays and Cape Cod.
As an early step in a broader pilot solutions-driven research effort, the Problem Formulation Workshop
incorporated a number of key elements rooted in translational science (Figure 12).
Develop Problem
Formulation Workshop
Design workshop around
extracting key information
outputs
Identified Need:
Research plans better
align with stakeholders'
needs
i
i
Identify
next
steps/phases
-Better align
with identified
research needs
-Better engage
partners and
stakeholders
\opNew Approach
YEnvirorurwnt and
y\ Public Health J
Training and
Application
Stakeholder
Involvement
p
Tool
Development
Field Research
V & Synthesis .
ediui sjcnieAB
Evaluate quality of
information outputs
and program
impacts
Launch
workshop
60 stakeholders
Carefully
document co-
produced key
research needs
Develop synopses
Collect
feedback
Participants
network,
momentum
builds, trust
increases
Disseminate back
to participants,
partners
Figure 12. Problem Formulation Workshop solutions-driven research cycle, adapted from
the National Institutes of Health Translational Science Spectrum (National Institutes of
Health 2018).
38
-------
First, a need was identified for research plans to better align with nutrient management stakeholders'
needs. ORD partnered with BCWCto develop and implement a Problem Formulation Workshop
designed around co-producing key information about research needs in nutrient management from a
broad group of stakeholders in the area. This evaluation demonstrates that participants were
constructively engaged, and that trust, momentum, and networks were fostered among participants and
with EPA. The information co-produced in the workshop was well-documented and synthesized in
synopses for the purpose of developing research plans. The value of program evaluation and social
science was clear in improving the structure, documentation, and transferability of results from the
workshop. Synopses were also disseminated back to participants and partners for their feedback.
Meanwhile, analyses of information outputs and participant observation enabled evaluation of the
quality of information produced as well as the program impacts.
The workshop documentation, evaluation, and resulting ongoing stakeholder collaborations collectively
enabled better identification of next steps that can be used to deliberately align priority research needs
and facilitate future engagement with stakeholders and partners. This extensive level of engagement
may not be appropriate for all research efforts, but in this application, EPA personnel reported that they
felt prepared to develop the next phases that better involve stakeholders and their needs. The effective
elements from this early phase provide a good roadmap that can help ORD to continue developing next
steps that incorporate mechanisms for two-way information sharing and that continue to produce
benefits for stakeholders.
39
-------
References
Bowen GA. 2009. Document analysis as a qualitative research method. Qualitative Research Journal
9(2) :27-40.
Cabrera D, Cabrera L. 2015. Systems thinking made simple: new hope for solving wicked problems.
Odyssean Press, Ithaca.
Cape Cod Commission. 2015. Cape Cod Area Wide Water Quality Management Plan Update. Cape Cod
Commission, Barnstable, MA.
https://sp.barnstablecountv.org/ccc/public/Documents/208%20Final/Cape Cod Area Wide Water Qu
ality Management Plan Update June 15 2015-Printable.pdf. Accessed 21 February 2019.
Cape Cod Commission. 2016. Preliminary Guidance for Piloting, Monitoring, and Evaluating Non-
Traditional Water Quality Improvement Technologies on Cape Cod. Cape Cod Commission, Barnstable,
MA. http://www.capecodcommission.org/resources/208/appendices/Appendix%20D%20-
%20Technologv%20Monitoring%20Guidance%20and%20Protocols.pdf. Accessed 21 February 2019.
Cape Cod Commission. 2017. GIS Hub. http://gis-cccommission.opendata.arcgis.com/. Accessed 21
February 2019.
Cape Cod Commission. 2018. Technologies Matrix.
http://www.capecodcommission.org/index.php?id=656. Accessed 21 February 2019.
Cash DW, Clark WC, Alcock F, Dickson NM, Eckley N, Guston DH, Jager J, Mitchell RB. 2003. Knowledge
systems for sustainable development. Proceedings of the National Academy of Sciences 100(14):8086-
8091.
Chen HT. 2014. Practical program evaluation. Sage Publications, Thousand Oaks.
Conklin J. 2008. Growing a global issue base: An issue-based approach to policy deliberation. In:
Directions and implications of advanced computing; Conference on online deliberation (DIAC-
2008/OD2008).
Heemskerk M, Wilson K, Pavao-Zuckerman M. 2003. Conceptual models as tools for communication
across disciplines. Conservation Ecology 7(3):8. [online] http://www.consecol.org/vol7/iss3/art8/
Howes BL, Samimy Rl, Dudley B. 2003. Site-specific nitrogen thresholds for southeastern Massachusetts
Embayments: critical indicators interim report. Massachusetts Estuaries Project.
https://www.mass.gov/files/documents/2016/08/mp/nitroest.pdf. Accessed 21 February 2019.
Fischer F. 2000. Citizens, experts, and the environment: The politics of local knowledge. Duke University
Press, London.
Krippendorff K. 1989. Content analysis. In: Barnouw E, Gerbner G, Schramm W, Worth TL, Gross L
(eds) International encyclopedia of communication, vol. 1. Oxford University Press, New York, pp 403-
407.
40
-------
Kunz W, Rittel H. 1970. Issues as elements of information systems. Working paper 131. Institute of
Urban and Regional Development, University of California, Berkeley.
MA DEP (Massachusetts Department of Environmental Protection). 2018. Watershed Permit for the
Pleasant Bay Watershed. http://pleasantbav.org/wp-content/uploads/Pleasant-Bay-Watershed-Permit"
copy.pdf. Accessed 21 February 2019.
Martin DM, Piscopo AN, Chintala MM, Gleason TR, Berry W. 2019. Developing qualitative ecosystem
service relationships with the Driver-Pressure-State-Impact-Response framework: a case study on Cape
Cod, Massachusetts. Ecological Indicators 84:404-415.
McLaughlin, J.A. and G.B. Jordan. 1999. Logic models: a tool for telling your program's performance
story. Evaluation and Program Planning. 22:65-72.
Merrill NH, Atkinson SF, Mulvaney KK, Mazzotta MJ, Bousquin J. in review. Using data derived from
cellular phone locations to estimate visitation to natural areas: an application to water recreation in
New England, USA.
Mulvaney KK, Atkinson SF, Merrill NH, Twichell JH, Mazzotta MJ. In review. Quantifying recreational use
of an estuary: a case study of Three Bays, Cape Cod, USA.
National Institutes of Health. 2018. Translational science spectrum. National Center for Advancing
Translational Sciences, https://ncats.nih.gov/translation/spectrum. Accessed 21 February 2019.
Okada A, Buckingham-Shum S, Sherborne T (eds). 2008. Knowledge Cartography: Software Tools and
Mapping Techniques. Springer, London.
Patton MQ. 2002. Qualitative Research and Evaluation Methods, 3rd edition. Sage Publications,
Thousand Oaks.
Shirey MR. 2012. Stakeholder analysis and mapping as targeted communication strategy. Journal of
Nursing Administration 42(9):399-403.
Spradley JP. 2016. Participant Observation. Waveland Press, Long Grove.
Susskind LE, McKearnen S, Thomas-Lamar J. 1999. The consensus building handbook: A comprehensive
guide to reaching agreement. Sage Publications, Thousand Oaks.
Tague N. 2005. The tools. In: O'Mara P (ed) The quality toolbox, 2nd edition. ASQ Quality Press,
Milwaukee, pp 93-521.
Twichell JH, Mulvaney KK, Hulet C, Secunda J, Lei by A, Buchholtz ten Brink M. 2018a. "Getting
Community Buy-In for Stormwater Funding" A Four-Session Participatory Workshop: Participant
Workbook. U.S. Environmental Protection Agency, Office of Research and Development, National Health
and Environmental Effects Research Laboratory, Atlantic Ecology Division, Narragansett, Rl. EPA/600/R-
18/213.
Twichell JH, Mulvaney KK, Hulet C, Secunda J, Lei by A, Buchholtz ten Brink M. 2018b. "Getting
Community Buy-In for Stormwater Funding" A Four-Session Participatory Workshop: Facilitator Manual.
41
-------
U.S. Environmental Protection Agency, Office of Research and Development, National Health and
Environmental Effects Research Laboratory, Atlantic Ecology Division, Narragansett, Rl. EPA/600/R-
18/214.
University of Wisconsin. 2019. University of Wisconsin-Extension Program Development and Evaluation
Logic Models Flowchart.
US EPA (U.S. Environmental Protection Agency). 2017. Wildfire Smoke and Health Risk Communication
Workshop: Workshop Report. Research Triangle Park, NC. EPA/600/R-17/076.
42
-------
Appendix A
Sample invitation letter to participants
SE
% ^
^ prO^
I Three Bays Nonpoint Source Nutrient Management
Problem Formulation Workshop
Clean Water
Coalition
A
Barnstable
September 21, 2018
Dear [invitee],
The Barnstable Clean Water Coalition (BCWC) and the U.S. Environmental Protection Agency's Office of
Research and Development (ORD) are pleased to invite you to take part in the Three Bays Nonpoint
Source Nutrient Management Problem Formulation Workshop to be held at the Marine Biological
Laboratory (Loeb Laboratory G70) in Woods Hole on October 30, 2018. The goals for this workshop are
to: 1) develop a comprehensive approach to reduce nitrogen loading to meet the Total Maximum Daily
Load (TMDL) and achieve water quality goals in Three Bays; 2) identify data, knowledge, resource, and
regulatory constraints and develop plans to address them; 3) build partnerships that can foster
innovative approaches to reducing nitrogen loadings; and 4) develop models, tools, and strategies that
can be transferred to other watersheds. While the workshop will focus specifically on the Three Bays
Watershed, we anticipate that the results of this work will be transferable to other communities facing
similar problems. Following introductory and stage-setting remarks, workshop participants will break
into smaller groups to identify how ORD and partners can best contribute to meeting the technological,
modeling, monitoring, and social science needs of a watershed-based solution. The agenda for the
workshop is attached [refer to Appendix G],
Nonpoint source nitrogen and phosphorous loading is a significant problem across Cape Cod and
throughout the nation. There are about 34 TMDLs for nitrogen impairment in coastal embayments on
Cape Cod, and most do not have comprehensive and active programs to reduce nonpoint source
nitrogen loadings. Resolving the nutrient issue through installation of public sewer systems, while
technologically feasible, can be prohibitively expensive due to the distribution and seasonality of the
population on Cape Cod and the Three Bays Watershed. Additionally, new public sewer systems would
not address legacy nutrient pollution of surface and groundwaters.
BCWC and ORD are partnering with key stakeholders to promote development of a watershed-based
approach that utilize nature-based solutions to significantly reduce nutrient loading to surface waters
and coastal embayments of the Three Bays Watershed, the third most compromised watershed on the
Cape. Given the prohibitive costs of centralized sewering, research is needed to develop performance
43
-------
characteristics for alternative nutrient reduction technologies and their placement in the landscape. The
results will move us closer to adoption of nature-based methods here on the Cape. To accomplish this,
we must agree on the critical research gaps. A structured problem formulation exercise will identify that
priority research and establish expectations among key stakeholders on approaches and tools to solve
this problem. This workshop is focused on nutrient issues in the Town of Barnstable, although these
approaches will be useful to other watersheds on the Cape and nationwide.
The agenda for the workshop is attached. Please respond to Stacie Smith, Consensus Building Institute
(617-844-1124, stacie@cbi.org) concerning your availability to contribute to the workshop. We look
forward to working with you to develop ways to solve the nutrient problem in the Three Bays
Watershed and beyond!
On behalf of the Workshop Steering Committee,
Zenas Crocker
Barnstable Clean Water Coalition
zcrocker@bcleanwater.org
617-947-9898
Anne W. Rea
U.S. Environmental Protection Agency
rea.anne@epa.gov
919-541-0053
Attachments:
Draft Problem Formulation Workshop Agenda
Problem Statement (below)
Directions to Loeb Laboratory at the Marine Biological Laboratory
Problem Statement:
Nonpoint source nitrogen and phosphorous loading is a significant problem across Cape Cod and
elsewhere. There are about 34 watersheds on Cape Cod with Total Maximum Daily Loads (TMDLs) for
nitrogen impairment in coastal embayments and most do not have comprehensive and active programs
to reduce nonpoint source nitrogen loadings. Resolving the nutrient issue through installation of public
sewer systems, while technologically feasible, can be prohibitively expensive due to the distribution and
seasonality of the population on Cape Cod. Additionally, new public sewer systems would not address
legacy nutrient pollution of surface and groundwaters. A watershed-based solution is desired by
44
-------
stakeholders in the Cape's Three Bays Watershed to significantly reduce nutrient loading to surface
waters and coastal embayments and to achieve water quality goals.
Research is needed to inform decisions about possible solution sets for nonpoint nutrient management
in the Three Bays Watershed. New information is needed about the performance characteristics of
alternative nutrient reduction technologies and their placement in the landscape. Monitoring is required
to evaluate the effectiveness of technologies to remove nutrients, to document baseline water quality
conditions, to track system responses to placement of interventions, and to promote regulatory
acceptance. Modeling is required to best target intervention placement and to predict estuarine
responses. Restoration research is required to inform decisions about nature-based interventions for
nonpoint nutrient management. Social science contributions are required to understand public
acceptance of alternative technologies and their placement in the landscape, to quantify the economic
benefits of clean water to tourism, and to provide support to decision making. Also needed is
development of an approach to transfer solutions and lessons learned to other areas facing similar
nonpoint source problems.
45
-------
Appendix B
Participant agenda
PROBLEM FORMULATION WORKSHOP
Identify key knowledge gaps and research needs for sustainable management of nutrients in
the Three Bays watershed
Hosted by US EPA Office of Research and Development
Agenda
Initiative Goals:
1) Develop a comprehensive approach to reduce nitrogen loading to meet the TMDL and achieve
water quality goals in Three Bays.
2) Identify data, knowledge, resource, and regulatory constraints and develop plans to address
them.
3) Build partnerships that can foster innovative approaches to reducing nitrogen loadings.
4) Develop models, tools, and strategies that can be transferred to other watersheds.
Problem Formulation Workshop Goals:
1) Clarify the key knowledge gaps and relationship/partnership needs related to potential nitrogen
removal technologies and the watershed as a whole.
2) Identify the potential research contributions and approaches for key workshop partners to
resolving the identified knowledge gaps.
3) Build relationships and identify additional needs and approaches for fostering partnerships and
networks within and beyond the watershed.
4) Explore how the work in Three Bays watershed can be transferred to other watersheds facing
similar challenges.
8:00 Registration
8:30 Welcome, Overview of the Initiative and Day
Welcome: Suzanne van Drunick, EPA/Office of Research and Development, National
Program Director, Safe & Sustainable Water Resources Research Program
Solutions-Driven Research & Translational Science: Bryan Hubbell, EPA/Office of
Research and Development Senior Advisor on Social Science
Agenda and Overview of the Day: Stacie Smith, CBI
8:50 The Vision for Three Bays - Zenas Crocker, Barnstable Clean Water Coalition (BCWC)
Overview of the nutrient management challenge in Three Bays
History of recent management efforts
Key areas of focus for research
Questions, comments, and clarifications
46
-------
9:40 Break
10:00 Introduction to Concurrent Discussions - Stacie Smith, CBI
10:15 Concurrent Group Discussion 1: What do we need to know about alternative nutrient
removal technology interventions for Three Bays (and beyond)?
Small group discussions each focusing on a subset of approaches to identify the key,
priority technical/scientific questions we need to answer about these technologies
in order to move forward with piloting and implementation in Three Bays
11:45 Report out and full group discussion on Concurrent Discussions
Groups share their top 3-5 research needs, and invites questions and input from the
full group discusses
12:15 Lunch
1:15 Introduction to Concurrent Discussion Part 2 - Stacie Smith, CBI
1:25 Concurrent Discussions Part 2: Putting it all Together
Small group discussions in reorganized groups discuss key watershed-wide
questions, and identify highest priorities from those and the needs defined in the
morning sessions.
2:45 Report out and full group discussion on Concurrent Discussions
Groups share their top priorities from their discussions
Full group helps to identify priorities, concurrence and disagreement, and further
discussion needed
3:15 Break
3:35 Full Group Discussion: Taking it Forward
Clarifying the pathway forward to address priority research needs
Identification and discussion of unresolved issues or missing pieces from
any/each of the earlier sessions
Discussion about how to translate the research (and/or initiative) to other
towns, regions, states - opportunities and challenges
4:50 Closing and Thank You - Zenas Crocker, BCWC
5:00 Adjourn
47
-------
Appendix C
Handouts for the morning breakout session
Shellfish & Aquduilture
What do we know already?
There is a range of considerations on Cape Cod for using shellfish for nitrogen (N) removal.
Using shellfish is applicable across Cape Cod.
Many different shellfish species are being considered for use.
There are permitting challenges.
There are issues with closed vs. open waters and permission to sell shellfish.
Industry vs. town considerations are different.
We roughly know how much N removed when animals are removed.
Siting can be challenging.
We have identified potential areas for aquaculture in Three Bays system.
In watershed report, shellfish was one of the biggest alternatives cited. In the "bookend"
scenarios, 41 acres of aquaculture/oyster beds accounting for over 50% of the non-traditional N
removal.
We know the co-benefits, but it is not always easy to measure them.
Monitoring of growth, condition and harvest is important to understand benefit.
It requires maintenance.
What do we need to know?
How would you get credit toward the Total Maximum Daily Load (TMDL) for N removal?
How to get regulatory credit if you don't know how many are harvested?
How will the market react to big increases in shellfish supply?
How to validly represent recreational harvest?
What is the best way to monitor changes in N due to shellfish?
What happens if shellfish experience disease and cannot be harvested/used?
Will depuration be permitted for addressing human health concerns/ is it appropriate?
How many acres could you leaseecologically, socially, economically? (What is the carrying
capacity for Three Bays?)
Where within Three Bays should shellfish be located to have an impact on N for TMDL?
How to reconcile trade-offs in water use (i.e., recreational use)?
What are the potential funding sources for shellfish/aquaculture efforts?
What other species besides oysters should be considered for use in Three Bays? Should we only
focus on edible species?
48
-------
Permeable Reactive Barriers (PRBs)
What do we know already?
PRBs are used extensively in the groundwater remediation industry.
PRB experts claim that PRBs can attenuate 90% of nitrogen (N) that flows through them.
Town of Falmouth conducted a feasibility study for PRBs and estimated that costs would be
high.
Waquoit Bay PRB (shallow trench and wood chip method) has been installed for 14 years with
high removal rates and minimal wood decomposition.
Orleans PRB (injection wells) has demonstrated mixed results to date.
Cape Cod has good water table/groundwater flow maps that can be used for preliminary PRB
location planning.
Cape Cod's 208 Plan Update identified dozens of potential PRB locations throughout Cape Cod
using GIS screening tool (Watershed MVP).
What do we need to know?
What are costs (design, installation, rejuvenation, monitoring)?
At what depths do we expect to find high N concentrations in groundwater?
What are best locations for PRBs?
How to optimize location of PRB to maximize N capture (zone of capture)?
What is the best way to monitor changes in N from PRBs (see Cape Cod Commission Monitoring
Report; Cape Cod Commission 2016)?
What is the best way to provide credits for PRBs?
49
-------
Fertigation Wells
What do we know already?
Fertigation wells can provide local irrigation sources and re-capture and re-use nitrogen (N)-
enriched groundwater.
"Pump and Treat" wells have been used extensively in the groundwater remediation industry.
Turfgrass is assumed to attenuate 80% of the applied N (Cape Cod Commission, Massachusetts
Estuary Program (MEP)).
There are extensive turfgrass areas (golf courses, athletic fields, cemeteries, lawns).
The Pinehills in Plymouth has successfully deployed fertigation wells to recapture N in
groundwater downgradient from their Waste Water Treatment Plant and apply to golf courses.
Use of localized/on-site fertigation wells would reduce demands on potable public supply wells.
Reducing demands on public supply wells will reduce impacts on nearby streams/wetlands.
Massachusetts Department of Environmental Protection (MassDEP) has provided credits for
fertigation wells in the Pleasant Bay Watershed Permit.
What do we need to know?
Where are the existing irrigations wells for golf courses, athletic fields, cemeteries, lawns?
What is a reasonable recapture rate for N?
What is the cost to re-locate an existing irrigation well to a more favorable N capture location?
How to optimize location of fertigation well to maximize N capture (zone of capture)?
What is the best way to monitor changes in N from fertigation wells?
Is the MassDEP credit system provided in the Pleasant Bay Watershed Permit the best way to
provide credits?
50
-------
Innovative & Alternative (I/A) Septic systems
What do we know already?
Annual nitrogen (N) loading to Three Bays = 46,221 Kg-N/yr; reduction target is 20,578 Kg-N/yr
(Watershed MVP).
Septic systems contribute approximately 85% of the N load to the Three Bays estuary
(Massachusetts Estuary Program (MEP)).
Three Bays has more than 7,000 residential parcels.
Future development potential (buildout) could add 10,000 kg-N/yr (MEP).
Septic system performance:
o Assumed MEP Title 5 systems effluent = 26.25 mg-N/L
o l/A septic systems effluent = 19 mg-N/L (Cape Cod Commission technologies matrix;
Cape Cod Commission 2018)
¦ Most l/A systems do not remove enough N to achieve watershed goals.
o Enhanced I/As = 13 mg-N/L (CCC technologies matrix; Cape Cod Commission 2018)
¦ Enhanced l/A systems could be part of the solution. Some technologies might
achieve considerably lower N concentrations.
Barnstable County Department of Health and Environment maintains database of l/A systems in
use on Cape Cod.
What do we need to know?
Can we narrow the range for N reduction target for septic systems, since it will influence the
types of systems (l/A vs enhanced l/A) considered as part of the solution?
What is the actual effectiveness and cost effectiveness of currently permitted l/A systems and
enhanced l/A systems?
Can improved operations and maintenance (O&M) management system improve overall
performance?
How many enhanced l/A designs currently permitted in Massachusetts? How well do they
perform? Cost?
How would you get credit toward the Total Maximum Daily Load (TMDL) for N removal?
Testing: For new enhanced l/A systems (e.g., not yet permitted in Massachusetts)
o How many are currently being, or soon to be, field-tested?
o How far along are they in the permitting process?
o Could testing in Three Bays accelerate the permitting process?
¦ How many systems?
¦ How would this be funded?
¦ Coordinate with testing in other watersheds?
o There seem to be questions about how to most effectively evaluate performance -
monitor influent and effluent?
o How are nitrogen reduction credits determined - monitoring protocols?
Implementation: Given estimates of effectiveness and cost for enhanced l/A systems:
o What are the most cost-effective enhanced IA systems ($/Kg-N removed)?
51
-------
o Are any enhanced I/As comparable to the cost effectiveness of a wastewater treatment
plant?
o Can N removal be improved, while costs are reduced?
o How many systems are needed to achieve the watershed N reduction goals (>50%?)?
o Where should they be placed?
o How to ensure operations and maintenance happens?
o Who pays, home owner, watershed nutrient utility?
52
-------
Bogs
What do we already know?
Headwater location intercepting nitrogen (N) enriched groundwater from upper watershed.
Weekly monitoring by SMAST started as of mid-July.
Farmers are willing to help identify groundwater discharge areas/springs and their knowledge
about hydrology.
Alex Hackman (DER) has done some preliminary map analysis.
Bogs are being farmed with various degrees of intensity.
Marston Mills River originates and runs directly through the bogs.
Several small ponds exist within the system.
Two distinct wetland conditions exist within the system where cranberry farming has ceased.
What do we need to know?
We need to understand the hydrological conditions in and around the bogs (how much
groundwater discharge occurs there/is there underflow?).
Do bogs contribute to or remove nutrients?
What are the contributions from groundwater (springs?)?
What are the existing wetlands doing and can their performance for mitigating nutrients be
measured?
Could there be treatment opportunities within the bogs (woodchips)?
Could the bogs be restored as wetlands and also achieve nutrient mitigation goals?
What types of wetland design might maximize/optimize N removal (gravel-based wetlands vs
organic soils)?
How do we measure N removal and agree on N credits?
53
-------
Marston Mills River
What do we know already?
Monitored for > 1 year for total nitrogen (N) and flow.
Total nitrogen concentrations of 1 mg/liter are common.
Some "hot spots" identified (horse farm, bogs, storm water).
Flow data indicates large impact from seasonal Centerville-Osterville-Marstons Mills district
(COMM) municipal well pumping (flow in the river is reduced by approximately 50% during
summer months).
ORD has measured isotopes at 6 locations.
Woods Hole Research monitors one site monthly and identifies Marstons Mills River having
among the highest N concentrations of all rivers they monitor on Cape Cod.
Massachusetts Division of Ecological Restoration (MassDER) has installed water level recorders
at 2 locations.
First contaminants of emerging concern (CEC) measurements have been tested by ORD.
What do we need to know?
What do the regulators need as "baseline" data in the river?
What role does groundwater play, well data?
How would management of COMM pumping rates improve conditions in the estuary (reduced N
loading, more flow of cleaner water)?
Which interventions may be appropriate?
Horse farm treatment swale/permeable reactive barrier (PRB)?
How might the River inform modeling to understand how several different interventions are
impacting nutrient attenuation?
54
-------
Ponds (Mill Pond Restoration)
What do we know already?
Mill Pond is the gateway to the upper watershed.
It was created 300 years ago and has filled with sediment.
School for Marine Science and Technology (SMAST) Massachusetts Estuary Project (MEP)
showed a nitrogen (N) attenuation rate of approximately 20% in early 2000s.
This is significantly lower attenuation rate than the default value of 50%.
Recent water quality monitoring on inlet vs. outlets by Barnstable Clean Water Coalition (BCWC)
suggest current attenuation rate might be even lower now.
A draft dredging plan was filed with MEPA 2012 (Massachusetts Environmental Policy Act).
State agency review comments included listed species Bridle Shiner and Water Willow Borer
(Natural Heritage & Endangered Species Program)
The Town is supportive of dredging.
Mill Pond is viewed as a community asset.
Dredging volume could be up to 40,000-50,000 cubic yards.
It is an important herring run and fish passage area.
What do we need to know?
Will dredging Mill Pond attenuate nitrogen?
How does volume and residency time play into the equation?
Can smaller pilot projects in the upstream bog ponds allow us to study this relationship in
advance?
Permit hurdles? Massachusetts Department of Environmental Protection, Natural Heritage?
Dredge materials disposal/beneficial reuse?
Costs?
Intersection with herring/herring run restoration?
Total Maximum Daily Load (TMDL) credit math?
55
-------
Appendix D
Blank grid posters for the morning breakout sessions
Shellfish & Aquaculture Breakout Group
Monitoring
Modeling
Social Science
Other
PRBs & Fertigation Wells Breakout Group
Monitoring
Modeling
Social Science
Other
56
-------
IA Systems Breakout Group
Monitoring
Modeling
Social Science
Other
Restoration (Cranberry bogs, streams, ponds) Breakout Group
Monitoring
Modeling
Social Science
Other
57
-------
Appendix E
Blank prioritization posters for the morning breakout sessions
Shellfish & Aquaculture Breakout Group
Priority Research Needs
Need 1:
Need 2:
Need 3:
Need 4:
IA Systems Breakout Group
Priority Research Needs
Need 1:
Need 2:
Need 3:
Need 4:
PRB & Fertigation Wells Breakout Group
Priority Research Needs
Need 1:
Need 2:
Need 3:
Need 4:
Restoration (Cranberry bogs, streams, ponds)
Priority Research Needs
Need 1:
Need 2:
Need 3:
Need 4:
58
-------
Appendix F
Blank grid posters for the afternoon breakout sessions
Afternoon Breakout Group A
Monitoring
Modeling
Social Science
Other
Afternoon Breakout Group B
Monitoring
Modeling
Social Science
Other
59
-------
Afternoon Breakout Group C
Monitoring
Modeling
Social Science
Other
Afternoon Breakout Group D
Monitoring
Modeling
Social Science
Other
60
-------
Appendix G
Blank prioritization posters for the afternoon breakout sessions
Afternoon Breakout Group A
Priority Research Needs
Need 1:
Need 2:
Need 3:
Need 4:
Afternoon Breakout Group C
Priority Research Needs
Need 1:
Need 2:
Need 3:
Need 4:
Afternoon Breakout Group B
Priority Research Needs
Need 1:
Need 2:
Need 3:
Need 4:
Afternoon Breakout Group D
Priority Research Needs
Need 1:
Need 2:
Need 3:
Need 4:
61
-------
Appendix H
Facilitator agenda
PROBLEM FORMULATION WORKSHOP
Identify key knowledge gaps and research needs for sustainable management of nutrients in
the Three Bays watershed
Hosted by US EPA Office of Research and Development
Facilitator's Agenda
Broad Initiative Goals:
1) Develop a comprehensive approach to reduce nitrogen loading to meet the Total Maximum Daily
Load (TMDL) and achieve water quality goals in Three Bays.
2) Identify data, knowledge, resource, and regulatory constraints and develop plans to address them.
3) Build partnerships that can foster innovative approaches to reducing nitrogen loadings.
4) Develop models, tools, and strategies that can be transferred to other watersheds.
Problem Formulation Workshop Goals:
1) Clarify the key knowledge gaps and relationship/partnership needs related to monitoring,
modeling, intervention selections, and social science for the potential nitrogen removal
technologies and the watershed as a whole, and other key issues that need to be resolved in order
to achieve the initiative goals.
2) Identify the potential research contributions and approaches for key workshop partners - ORD,
USGS, and others-to resolving the identified knowledge gaps.
3) Build relationships and identify additional needs and approaches for fostering partnerships and
networks within and beyond the watershed to help achieve goals.
4) Explore how the work in Three Bays watershed can be transferred to other watersheds facing
similar challenges.
8:00 Registration opens
8:30 Welcome, background/context, workshop purpose
Welcome: Suzanne van Drunick, EPA/Office of Research and Development, National
Program Director, Safe & Sustainable Water Resources Research Program
Solutions-Driven Research & Translational Science: Bryan Hubbell, EPA/Office of
Research and Development Senior Advisor on Social Science
Goals, agenda, ground rules, logistics, introductions of participants - CBI
8:50 The Vision for Three Bays - Presentation by Zee to give background of the big picture
Overview of the nutrient management challenge in three bays
History of recent management efforts
Key areas of focus for research
Questions, comments, and clarifications
62
-------
9:10 Questions/comments/conversation/clarifications to the day
Stacie invites questions/comments about any of the background and framing, other
areas for focus that we missed, other questions about the set-up of the day.
9:40 Break (return sitting in assigned small groups)
10:00 Introduction to Concurrent Discussions - Stacie
Stacie provides and overview of session goals, format, and goals, including overview
of what we are trying to identify - information that is salient, legitimate, credible,
and attainable
10:15 Breakout Group Discussion 1: What do we need to know about alternative nutrient removal
technology interventions for Three Bays (and beyond)?
Participants break into four pre-set small groups of 8-12 [Shellfish & aquaculture:
Facilitator-Marty Chintala, notetaker-Hale Thurston; PRBs & Fertigation: Facilitator-
Walter Berry, notetaker- Wayne Munns; lAs: facilitator-Tim Gleason, notetaker-
Casey Dannhauser; Restoration: facilitator-Sofia Soto Reyes, notetaker- Heather
Rockwell]
Step One: Brief review of small group packet, and especially known/unknowns.
Remind group - goal of session is to identify the key, priority technical/scientific
questions we need to answer about this technology in order to move forward with
piloting and implementation and prepare for potential usage at scale. (10 min)
o For groups with multiple interventions, provide a brief intro to both
technologies, allow some time for discussion of each individually, but focus
as much as possible on questions they have in common
Step Two: Refining the Knowns and Unknowns - Group Discussion: (20 min)
o Did we get the knowns/unknowns right (flipchart)? If not, what did we get
wrong?
o Have we identified the key questions about this technology related to
modeling, monitoring, regulatory issues, economic/financing issues, and
social acceptability? Are there other barriers we need to overcome, or
opportunities that we can draw on?
Step Three: Research Needs Identification exercise: Based on the identified
unknowns, what are the key research needs/questions? (30 min)
o Use the 4-grid breakout grid worksheet to help guide the discussion and
encourage identification of questions about monitoring, modeling, social
science, and other areas.
Shellfish & Aquaculture Breakout Group
Monitoring
Modelling
Social Science
Other
63
-------
Step Four: Group Prioritization exercise: What are the highest priority research
needs and questions we should seek to answer in the next 2-5 years? (30 min)
o If helpful: Write up list of key needs/questions on a flip chart, give each
participant 3 sticky dots to help prioritize
o Record top 3-5 on the "Priority Research Needs" poster for report out
Shellfish & Aquaculture Breakout Group
Priority Research Needs
Need 2:
Need 3:
11:45 Report out and full group discussion on Concurrent Discussions
Groups share top 3-5 questions and findings about them from their discussions
Questions, answers, and comments to help frame up the next round of discussions
During lunch, planning team completes afternoon posters (in case people need
them)
Priority Research Needs from Morning
Breakout Groups
PRBs & Fertigation Wells
Restoration {Cranberry bogs, streams, ponds)
IA Systems
12:15 Lunch
64
-------
1:15 Introduction to Concurrent Discussion Part 2 - Stacie
Stacie provides and overview of session goals, format, framing and expected
deliverables
1:25 Breakout Discussions Part 2: Putting it all Together
Participants break into four different small groups (2-3 from each of the morning
groups)
[Group A: Facilitator-Marty Chintala, notetaker-Hale Thurston; Group B: Facilitator-
Walter Berry, notetaker- Wayne Munns; Group C: facilitator-Tim Gleason,
notetaker-Casey Dannhauser; Group D: facilitator-Sofia Soto Reyes, notetaker-
Heather Rockwell]
¦ Step One: Identifying the watershed-level research needs - Individual sticky notes
(10 minutes)
o Ask the following question: Above and beyond intervention-specific
questions, what are the broader, systems-level key research needs/
questions for the watershed as a whole related to modeling, monitoring,
regulatory issues, economic/financing issues, and social acceptability?
o Invite participants to write 3-5 key needs/questions and post them on the
four grid watershed posters in the correct categories.
Afternoon Breakout Group A
Monitoring
Modeling
Social Science
Other
o Some sample questions:
How do these interventions and this research fit into the watershed
permit and process?
What are the impacts of these interventions as a whole, bay-wide?
What does watershed and estuary-wide recovery look like?
Where might there be siting conflicts and timing issues?
How can findings from Three Bays be transferred to other areas and
from other areas to Three Bays?
65
-------
¦ Step Two: Identifying the watershed-level research needs - Group Discussion: (30
min)
o Review the post-its as a full group. Discuss:
o Are these the right categories?
o What themes / similarities/ subtopics do we see?
o Are the questions here that can be answered with existing
data/research?
o Do we have follow-up questions about any of these?
o Is there anything missing that this exercise made you think of?
Step Three: Group Prioritization exercise: (35 min)
o Drawing from the systems-level questions/needs above, what are the
highest priorities for research in the next 2-5 years? Faciitators- ask
participants to refine and synthesize statements from above
o Where is there concurrence, where does disagreement or uncertainty
remain? What needs more discussion?
o If helpful: Write up key questions on a flip chart, give each participant 3
sticky dots
Afternoon Breakout Group A
Priority Research Needs
Need 1:
Need 2:
Need 3:
Need 4:
Need 5:
Need 6:
Need 7:
Need 8:
2:45 Report out and full group discussion on Concurrent Discussions
Groups share key findings from their discussions
Questions, answers, and comments to help frame up the next round of discussions
3:15 Break
66
-------
3:50
Full Group Discussion: Taking it Forward - Stacie facilitates
Group discussion of top priority research needs: (Going Forward Discussion):
o What needs to happen to move this agenda forward?
o Who are the parties and networks (in this room and beyond) to help carry
this forward?
o Is there existing information we might draw on to help answer the
unknowns?
o What approaches/methods and partners are available to learn this
information?
o What other details would be helpful to identify? Locations, costs,
timeframes, resources, partners, etc?
o Are there other barriers we need to overcome, or opportunities that we can
draw on?
Identification and discussion of unresolved issues or missing pieces from any/each
of the earlier sessions
Discussion about how to translate the research (and/or initiative) to other towns,
regions, states - opportunities and challenges
4:40 Closing and final survey, thank you - Zee
5:00 Adjourn
Facilitators should plan to meet for 15 minutes with notetakers to pull together a
quick recap on the events of the day/ identified needs/ lessons learned, to prepare
to provide reflections/report out for Day Two.
67
-------
Appendix I
Notetaker guidance and templates
Notetaker Roles and Responsibilities
(please read ahead)
Thank you so much for serving as a Notetaker in the Problem Formulation Workshop! We're trying to (1)
capture whether we accomplished what we set out to do in the two breakout sessions and (2) record
information that is structured in a way that is useful for research planning in future.
Below are some general tips for understanding how to fill out the notetaking templates. Overall, your
goal should be to capture the needed "Information out" as well as the flow of the conversations from
the various perspectives represented in your breakout group. The left-hand column is structured to
help you fill in "information-out" for each sub-activity. The facilitator will structure the discussion
around capturing and distilling this information, and will be flip-charting in a similar structure. The right-
hand column is designed as a space to take running notes about the flow of discussion: elaborations or
threads that don't fit in the righthand column, group dynamics, as well as your reading on how the
discussion is going. After each breakout, the notetaker will fill out an assessment about how it went.
Any statements and information you record will be anonymized, but it helps us to interpret the data later if you
include participants' names. If for some reason a participant is uncomfortable, there is no need to use their name.
Left-hand column (required "information-out"):
¦ The facilitators will be flip-charting the same information, but as much as possible, please try to
capture the statements in the speaker's own words (because it may not match the facilitator's
summarized version). This will help broaden our understanding of what was said.
Additional elaboration on discussion around identifying and refining "information-out" can be
squeezed around the edges (If a statement was refined, why? If a few people disagreed, what
was their point of view? Link statements to names, where possible.)
Please remember to record sub-discussion start times
Right-hand column (other running notes):
¦ Quotation marks for participants' statements are helpful.
Record who said what where possible
¦ Record value statements (about activities, the workshop as a whole, peer interactions)
Record statements about strong positions, concerns, needs, etc.
¦ "Parking Lot" items: Tangential comments/questions that are relevant for future exploration
¦ Group dynamics: Mood, energy-level, emotions, communication quality, receptivity,
disagreements, understanding, finding common ground, problem-solving, "aha" moments,
sticking points, or rabbit holes, etc. Please record the context in which these observations occur.
¦ Facilitator guidance: Notes about what questions were asked to help the group stay on track
¦ "Anything else that seems relevant": Use your best judgment, record as much as you can
Post-breakout 1-Page Evaluation
¦ After the breakout group concludes, we'd appreciate your assessment of how the discussion
went. We'll have as many ORD folks as possible fill this out too, not just notetakers. If filling
these out directly after the discussion isn't possible, below are some other times to do it.
¦ Breakout 1: lunch, Breakout 2: afternoon break
68
-------
Notetaker's Template - Three Bays Problem Formulation Workshop - Breakout 1
What do we need to know about alternative nutrient removal technology interventions for Three Bays (and beyond}?
Your group's intervention category is:
Please list who is in your breakout group:
Step One: Brief review of Read-Ahead packet (10 min). List
questions/comments that come up during this activity (about the materials or
about anything else that you think is relevant)
This column provides space for recording the flow of discussion:
1. Sub-discussion start times
2. Notes on discussions that are off topic from the deliverables, questions, or
concerns that seem relevant for future exploration
3. Observations about group dynamics (record participant concerns or
frustrations, equality of air time, "aha" moments, group confusion)
4. Discussion questions or clarifications posed by facilitators
5. Anything else that you think is relevant to record (more is always fine, also,
if you're not sure where something fits, put it here and then mark it later)
69
-------
Step Two: Refining the Knowns and Unknowns (20 min).
List any new or refined knowns/unknowns identified by the group
Refining the Knowns/Unknowns start time: am
Discussion notes, group dynamics observations, facilitator questions, other
Please also mark a copy of the read-ahead handout:
S Did we get the knowns/unknowns right?
X Which pieces did we get wrong?
70
-------
Continue to list any new or refined knowns/unknowns identified by the group
Discussion notes, group dynamics observations, facilitator questions, other
71
-------
Step Three: Research Needs Identification exercise. Based on the identified
unknowns, what are the key research needs/questions? (30 min)
Record the research needs in the 4 categories on your mini-poster ^
MODELING (overflow notes)
MONITORING (overflow notes)
Research Needs Identification start time: am
Discussion notes, group dynamics observations, facilitator questions, other
72
-------
SOCIAL SCIENCE (overflow notes)
OTHER (overflow notes)
Discussion notes, group dynamics observations, facilitator questions, other
73
-------
Step Four: Group Prioritization exercise. What are the highest priority
research needs and questions we should seek to answer in the next 2-5 years?
(30 min)
List the highest priority research needs or research questions identified by
the group
*if the group uses multi-voting in the next Group Prioritization
exercise, please include the tally of votes for ALL questions
1.
2.
3.
4.
5.
6.
7.
8.
9.
Group Prioritization start time: am
Discussion notes, group dynamics observations, facilitator questions, other
74
-------
Discussion notes, group dynamics observations, facilitator questions, other
75
-------
Please fill this section out after Breakout 1. Your intervention group:
For the following statements, please circle a rating (l=disagree strongly 5=agree strongly)
The group produced the expected deliverables (left-hand column information)
The discussion was well-organized and well-directed
The technology overview (Step One) was a good use of time
The group was prepared to discuss knowns and unknowns (Step Two)
The group agreed on knowns and unknowns
The group was able to clearly identify and define research needs (Step Three)
The group agreed on priority research needs (Step Four)
The pace and scope of this discussion was appropriate
The group communicated constructively and all voices were heard
The group appeared hopeful and galvanized by the discussion
Members of the group learned from one another
2 3 4 5
2 3 4 5
2 3 4 5
2 3 4 5
2 3 4 5
2 3 4 5
2 3 4 5
2 3 4 5
2 3 4 5
2 3 4 5
2 3 4 5
Overall, how do you feel this discussion went?
Please use this space to provide context for some/all of the statements above about which you disagreed (i.e., scored: 1-2)
Please describe any "aha" moments or sticking points in the group discussion
Is there anything else you feel is important to share relating to evaluation of how well-organized or constructive this activity
was, participants' experience, or anything else?
76
-------
Notetakers Template Three Bays Problem Formulation Workshop Breakout 2
Putting it all Together
Please list who is in GfOlip
This column provides space for recording:
6. Sub-discussion start-times
7. Notes on discussions that are off-topic from the deliverables, questions, or
concerns that seem relevant for future exploration
8. Observations about group dynamics (record participant concerns or
frustrations, equality of air time, "aha" moments, group confusion)
9. Discussion questions or clarifications posed by facilitators
10. Anything else that you think is relevant to record (more is always fine, also, if
you're not sure where something fits, put it here and then mark it later)
Extra space any notes for any introductory conversation (questions,
clarifications, Breakout 1 follow-ups, etc.)
77
-------
Step One: Identifying the watershed-level research needs - Individual post-
its (10 minutes)
No need to record the needs/questions is on all post-it notes - we will keep
the posters. Use this page for noting discussion during this exercise.
Individual post-its: Identifying watershed-level research start time: pm
Discussion notes, group dynamics observations, facilitator questions, other
78
-------
Step Two: Identifying the watershed-level research needs - Group
Discussion: (40 min)
Please record the discussion around grouping, refining, and building on the
post-it exercise. Record themes and topic areas on your mini-poster, and
discussion around thoseprovide some examples of needs/questions to
contextualize your notes. ~
MODELING (overflow notes)
MONITORING (overflow notes)
Group Discussion: Identifying watershed-level research start time: pm
Discussion notes, group dynamics observations, facilitator questions, other
79
-------
SOCIAL SCIENCE (overflow notes)
OTHER (overflow notes)
Discussion notes, group dynamics observations, facilitator questions, other
80
-------
Step Three: Group Prioritization exercise (35 min)
List the key research needs identified by the group
*if the group uses multi-voting in the next Group Prioritization exercise,
please include the tally of votes for ALL questions
*Note if the group Agrees, Disagrees, or remains Uncertain about the
importance. Please add comments to the right.
1.
2.
3.
4.
5.
6.
Group Prioritization start time: pm
Discussion notes, group dynamics observations, facilitator questions, other
81
-------
Discussion notes, group dynamics observations, facilitator questions, other
82
-------
Please fill this section out after Breakout 2 for Group ,
For the following statements, please circle a rating (l=disagree strongly 5=agree strongly)
The group produced the expected deliverables (left-hand column information) 1 2 3 4 5
The discussion was well-organized and well-directed 1 2 3 4 5
The group was able to clearly identify and define watershed-level research needs (Step One and Two) 1
2 3 4 5
The group agreed on the key research priorities identified for the next 2-5 years (Step Three) 12 3
4 5
The pace and scope of this discussion was appropriate 1 2 3 4 5
The group appeared hopeful and galvanized by the discussion 1 2 3 4 5
The group communicated constructively and all voices were heard 1 2 3 4 5
Members of the group learned from one another 1 2 3 4 5
Overall, how do you feel this discussion went?
Please use this space to provide context for some/all of the statements above about which you
disagreed (i.e., scored: 1-2)
Please describe any "aha" moments or sticking points in the group discussion
Is there anything else you feel is important to share relating to evaluation of how well-organized or
constructive this activity was, participants' experience, or anything else?
83
-------
Appendix J
Full meeting summary shared with participants
Three Bays Nonpoint Source Nutrient Management Problem Formulation Workshop:
Draft workshop report
A one-day problem formulation workshop sponsored by the U.S. Environmental Protection Agency (EPA)
and Barnstable Clean Water Coalition (BCWC) was held on October 30, 2018, at the Marine Biological
Laboratory in Woods Hole, Massachusetts. In the workshop, 60 participants (see Supplement A)
convened to identify data, knowledge, resource, and regulatory constraints in addressing nutrient
pollution in the Three Bays watershed (Barnstable, MA) and develop plans to address them.
Workshop Goals
5) clarify the key knowledge gaps and relationship/partnership needs related to monitoring,
modeling, intervention selections, and social science for each of the potential nitrogen removal
technologies.
6) identify the potential research contributions of workshop partners to resolve the identified
knowledge gaps
7) build relationships and identify additional needs and approaches for fostering partnerships and
networks within and beyond the watershed to help achieve goals
8) explore how the work in the Three Bays watershed can be transferred to other watersheds facing
similar challenges
The workshop consisted of a number of breakout groups as well as facilitated, full-room discussion (the
Participant Agenda (Appendix B) was included as a supplement). EPA, BCWC, and the Consensus Building
Institute provided notetakers for the group and sub-group discussions and also captured identified
research needs through the guided facilitation process. The following pages of this document are the
combined outcomes of those groups' notes, condensed for consistent themes. The groups are identified
by the technology in the morning and grouped all together in the afternoon. Going forward, EPA and
BCWC would like to convene smaller planning teams resolve the identified knowledge gaps.
This packet contains 5 sets of bulleted identified research needs for each of the technology-
focused breakout group discussions (4) and all afternoon watershed-level discussions (1 for
groups A-D combined). The end goal of these are take-away bullets that can be useful for EPA
and partner organizations.
Notes about the content:
The research needs distilled from each group are all identified needs, barriers, or questions that emerged from
content analysis of the notes documented by facilitators and notetakers during the small group activities. These
needs vary in terms of specificity, importance to various participants, and attainability. The bullets are listed in no
particular order. Some needs may conflict with others, indicating conflicting views among different participants.
The identified research needs were grouped into rough thematic categories. In many cases, they may apply to
multiple categories.
Items marked with a star were priority items identified by participants in their small groups. They may not
reflect the views of the whole group. These are likely not the only priorities, as time in the workshop was limited.
84
-------
Shellfish & Aquaculture
Regulatory
Structure/
Permitting
¦
¦
¦
~
What local ordinances/permits are needed? Do they allow for scaling up?
How to credit in TMDL? What is the best metric?
How to quantify N removal by shellfish harvest and aquaculture? Is quantifying harvest accurate?
¦ What is success from a regulatory point of view (N reduction or habitat restoration per TMDL)?
Can 3 Bays (or another case study) show that aquaculture can help meet the target?
Siting
¦
¦
~
Leasing?
How to balance with human use (anchoring, mooring, etc.)?
Ecological disturbance to bottom
Implementation
¦
¦
¦
¦
What density of shellfish are needed?
How to balance with other restoration (eelgrass, scallops)?
What infrastructure is needed?
~ infrastructure is an unintended consequence of increased shelIfishing and aquaculture
What is the best method of oyster farming and harvest?
Monitoring
¦
¦
What's the baseline? (historic levels versus what shellfish were in place when they calculated the
load numbers for the TMDL)
Need to know how to monitor and measure harvest
Economics/
Market
~
~
Economic impact study for increase in shellfish harvest/aquaculture
¦ Supply/demand - changing price impacts, impact to existing industry
What secondary markets can be developed for oysters and other shellfish?
¦ Community market/secondary market
¦ Shucked oyster market/ non-edibles
Pilot/Case
Study
~
¦
"The Cape needs an effective case study"
Need a pilot with transferable results
Social
Acceptance &
Behavior Change
¦
¦
¦
Need for realistic framing about what shellfish can/cannot do to meet TMDLs
What is the best participatory format? ["how is everyone participating? What is the mix needed on
these sorts of projects, the responsibilities [from a management standpoint]- what's the process?
what are the responsibilities?"
A case study will be a social science research opportunity
Scalability
~
¦
¦
What happens when you scale up? (What can the system handle ecologically, economically and
socially? What is the carrying capacity of Three Bays in these three contexts?)
How big does a successful program need to be? [note: related to target set in regulatory goal]
Need a case study that achieves a piece of the N reduction goal (focusing on a smaller area and make
a bigger impact, sub-watershed reduction goals)
Secondary
Effects
¦
¦
Are there pathogen/closure impacts?
What are the impacts to other uses of the bay (both human and ecological)?
~ priority need identified by workshop participants, may not reflect the views of the whole group
85
-------
Innovative Alternative Septic Systems
Regulatory
Structure/
Permitting
~
¦
¦
How can the permitting process better allow for piloting and implementation of IA systems?
¦ Need new/creative permitting umbrellas for IA piloting and implementation
Need statistically significant datasets [of IA effectiveness data for general/provisional use or for
permitting] for the approval process
Need to explore the effectiveness of the watershed permit
Modeling
~
Need sensitivity analysis of MEP (concerns that there is a higher N load into septic systems than
originally assumedwhat is the impact of adjusting N load in the original MEP model?)
¦ Pleasant Bay is revisiting MEP model
Compare
Alternatives
¦
~
¦
Need to know total costs of different technologies (electricity, maintenance, etc.)
Update costs/lb N removed
Ultimately, need better understanding of system performance and cost effectiveness
Implementation
¦
¦
~
¦
Need attenuation analysis
¦ Need to know how small a leach field can be after an IA intervention installed
Need quality control
Develop sensors for diagnostic and onsite operation and maintenance enhancements to lower costs
and improve performance. Low-tech sensors providing real-time feedback to operators and
homeowners could improve system performance and lower monitoring costs.
Community infrastructure models.
Monitoring
¦
~
¦
¦
Need better removal estimates for standard systems (they are low)
Need better influent and effluent measures
Usefulness of remote sensing for oversight (role for sensors in diagnostics)
Financing
¦
~
How do we pay for implementation (water districts, ways to defray costs)?
Need to update cost/benefit analysis
Pilot/Case
Study
¦
¦
¦
Need to show that IA wastewater management systems are competitive with conventional
treatment (both in terms of N reduction and cost/kilogram of N removed)
Need to demonstrate success before making the jump to general use
Total nitrogen effluent data from different areas but with similar climates are acceptable to MADEP
(opportunity to increase the data available for evaluating and permitting new enhanced IA systems)
Social
Acceptance
& Behavior
Change
~
What are different models for promoting implementation of 1 As/ effective incentives?
Secondary
Effects
¦
Need to know how these systems process CECs
~ priority need identified by workshop participants, may not reflect the views of the whole group
86
-------
Restoration
tu)
c
~ Demonstrate N attenuation benefits of already restored bogs on Cape Cod
¦ What is the range of N attenuation from restoration? (cranberry bog data is lacking; study 3
restored bogs on Cape Cod; data would assist State in restoration planning and prioritization)
~ Produce scientifically sound evidence of water quality impacts from bog restoration (from
pilots, see below)
¦ Collect pre- and post-data from restoration/bog pilots
¦ Are there climate benefits and other co-benefits?
±f ~ Measure groundwater outflow/inflow (groundwater levels, N levels) and groundwater travel times in
o the mosaic of habitats (emergent wetlands, ponds, streams) associated with restoration; this
information is needed for estimating the benefits of the already restored bogs as well as for planning
new restorations
~ Monitor N and P, soil moisture, sediments, plant communities: demonstrate restoration of wetland
structure and function, in addition to water quality improvements
¦ Understand baselines
¦ Challenge: all bogs are unique systemsthey vary (e.g., some are sinks and some are sources)
CD
CO
a.)
>
fO 4-J
CL C13
Evaluate technology and engineering innovations in wetland design, creation, and restoration
p ci ¦ Need data to compare types of restoration from "engineered" to "natural"
., ¦ Plan bog restorations in conjunction with IA technologies
iis ¦ Employ active farms in public educationuse opportunity for living labs, adaptive management
U1 ¦ Restore Mill Pond
'2 l-
9 Q o
u) ~ Identify methods to maintain public interest where there is a lag time for positive/visible outcomes
aesthetic and cultural benefits) and how to educate the public about those benefits?
u m
e . Are there social/cultural costs to bog farmers or to residents?
CD
CD
W
c
ID
""O
O
Parameterize models (i.e., hydrological models, N attenuation models)
>¦
¦ Dredge spoils - how much produced? where to dispose? how to manage high costs of disposal?
c S
O t|r
« LiJ
IS)
~ priority need identified by workshop participants, may not reflect the views of the whole group
87
-------
Permeable Reactive Barriers and Fertigation Wells
>-\ no
.E
ro 3 ±f " Need to know how to get N removal credit
¦ Need pilots to inform permitting
(D +3 O)
CC t/1 Q-
TfT ¦ Need characterization of where N is in groundwater, including characteristics to depth for
-S to
g c treatment zones
2E ¦ Need modeling tool for placement at scale for engineers [also related to siting]
¦ PRBs seen as effective when sited appropriately [less so for injection wells], but siting is difficult.
Fertigation wells less tested.
¦ Injection wells - need performance information
¦ Mass balance (N) of fertigation wells - what is the ultimate fate of N?
~ For PRBs, how do different carbon sources and approaches compare?
¦ Can PRBs be useful for river restoration?
¦ How effective is fertigation? Seasonal effects? Saturation? Other limiting factors?
~ How long do PRBs last? What is the longevity of the carbon source?
Siting
¦
¦
~
Focus on site characterization
~ Placements of PRBs (Advantages? Points of confluence?)
¦ What are the challenges to placement in 3 Bays? (including local groundwater dynamics)
Cost of determining appropriate PRB placement is not trivial
Social siting challenges
ii
C
Q)
E
¦
ro £
T3 |j
CI Cl
O S£:
v ILLJ
i/)
-------
Watershed Research Needs (Groups A-D)
Regulatory 1
Structure/ 1
M5
C
4~J
4-J
E
L_
QJ
CL
+ * * ¦ ¦
What is needed to permit the different interventions? To meet requirements for a watershed permit?
Identify critical permitting needs (early in the process - any deal-breakers?)
Regulatory flexibility and streamlining needed; modify regulatory barriers; deal with variability
Incorporate nitrogen into open space protection plan
Are some of the permitting/ regulatory structures from other places appropriate for use in 3 Bays?
E.g., Require offsets or other systems from new development, so that there is no net increase (there is
already a no net N rule in place for big projects, so we have the mechanisms)
Modeling
¦ * *
We have good TMDL loading models, time to implement solutions to reduce loading
Need to feed numbers back into the models to identify mistakes in the models
Need flexibility in coefficients - must be adaptable to new information
Need groundwater (water budget and flow) and stormwater model (with long-term monitoring)
How well is groundwater flow defined? (Single groundwater/surface water system?; Water budget-
Flow patterns; Effects of pumping and wastewater)
What are the criteria to use in modeling for siting projects?
Include climate change stressors
Optimization (Across technologies? By time? By cost effectiveness ($/kg)?)
What are the time goals?
Siting
¦
¦
¦
¦
How to decide what to put where (with limited funding)?
Does the town have lands protection areas to prevent future N loading?
Need to better understand concentrations and travel times close to rivers to inform placement of IA
systems [also related to monitoring/modeling]
Could we offset the loss of Bridled Shiner habitat in Mill Pond with habitat creation in restored bogs?
Monitoring
~
*
¦
¦
¦
~
~
Monitoring 3 ways: how interventions perform, verifying nutrient loads, ecosystem recovery
Need hydrology/groundwater data (nitrogen distribution, loads, hotspots, travel times, siting, surface
water interaction, key rates of attenuation along the pathway from source to discharge)
How frequently do different alternatives need monitoring to effectively track progress over time?
Determining where to monitor?
Monitoring must be long-term monitoring
Costs of monitoring? What will the regulators require for monitoring?
Identification and protection of existing N sinks in the watershed
Pilot/Case
Study
~
~
¦
¦
~
¦
Now is the time to put interventions in the system and monitor
Hit a small watershed heavy and hard - measure and monitor
Take adaptive approach
Demonstrate success - focus on where you can show improvement
Use interventions to increase public engagement, leading to buy in
Need to demonstrate effectiveness and lack of unintended consequences through pilots
Compare Alternatives
~
~
¦
¦
¦
¦
¦
¦
Need comparisons using cost/lb N removal
Need to prioritize N reduction alternatives for implementation
What is the appropriate balance between source control and in-stream remediation?
Determine if some technologies are more appropriate downstream vs. upstream, and how this affects
time to meeting the TMDL goals
Identify best treatment schemes to address other known/unknown contaminants (including CECs)
Comparisons across restoration types for N remediation: existing hydrology vs. restored hydrology
What about alternatives within the context of expanded sewering?
What's the consensus on the different interventions?
89
-------
~ Identify funding sources
¦ Go to town, state, feds for funding pilots - the cost of pilots shouldn't always fall to the town
¦ Explore options for private investment, impact investing, risk-sharing
'O ¦ Some say money is not the issue (once the best options are clear), but rather permitting
J2 ¦ Increase tax base through enhanced development (which can be returned to N control)
¦ Need funding to prioritize measures to most immediately achieve N reduction across the watershed
~ Who pays and who benefits from the different interventions?
~ What are the most effective funding mechanisms? Most fair/equitable?
00
u
E
>
TO
TO
00
Cost-benefit analysis of economic impacts from non-trad, approaches (on property values, recreation)
o ¦ What's the cost of doing nothing to property values/recreation?
g ¦ Develop trading schemes for N
LU
~ What are the social barriers to reducing nitrogen? To use using the different technologies?
^ ¦ Fundamentally this is a social science problem - how to balance all different perspectives?
¦ What do residents value the most? Climate change mitigation/ adaptation/ open space/ etc.?
¦ Need to develop community support and buy-in for costs of N removal
co ¦ Cultivate political will (currently doesn't exist)
^ ¦ Need to involve citizen's groups
" ~ Invite and engage watershed population in on-the-ground visits or stakeholder meetings
Need to educate public about N reduction needs, diff. alternatives, costs, lag time in positive results
TO
4_i
cd ¦ Calculate large potential costs vs. smaller project costs to make costs easier to swallow
^ ¦ Provide certifications and online trainings to landscapers for lawn care
¦ Get buy-in from homeowners on lawn care, change expectations/norms around lawn care and
g fertilizer use, reward behavior that reduces N inputs
¦ Enforcement of alternatives (upkeep, implementation (e.g., fertilizer ban))
~ What are the best incentives (individual, collective) for watershed population to reduce N inputs?
u .
ro Is estuary N a public health issue? Red tide? Shellfish?
CL X
>-
bD-tf
c * Need to share information on different pilots with towns across the Cape
2 ~ Need a data repository / test center / think tank
QJ
00 ¦ Need better data visualization for translation to the public
(T5 £
to ¦ How can data/information from other places apply to 3 Bays and vice versa?
c
2 ¦ Need approaches based on data that is acceptable to everyone and a validation process
ro -S ¦ Need standardization
O
¦ Need Cape-wide aquaculture cooperative and extension service to vouch for data quality
>-
_Q
jro
TO
Q
U)
~ Need modeling for scaling up from pilot level to watershed level
¦ Is the infrastructure in place to be able to ramp up technologies?
Id ¦ What is the best holistic water management that includes not only nutrient management, but also
r~ ^
£ > Id groundwater recharge, CECs, hazards resilience, and more
.2 «~ ^ ¦ How do we integrate social, economic, and environmental goals?
cu ¦
<; ¦ What are the real impacts of development? How do we align solutions with increased development?
~ priority need identified by workshop participants, may not reflect the views of the whole group
90
-------
Proposed Actions/Roles - Final Plenary Discussion Synthesis
Related Afternoon
Topic Discussion Themes Proposed Actions Proposed Key Players
(Prompts)
bD
c
X
CD
~o
c
-
T3
ZJ
00
CD
00
TO
u
0
01
Pilot: identify sites,
put projects in the
ground, monitor
them
Prioritization:
understand cost/kg
for changes of
different options
Need funds to extract and share lessons from
existing pilots around the region (e.g., PRBs) (need
ongoing funding/grantee structure)
New pilots (Criteria: close to source of issue, high
density interventions, in a priority area, short
travel time, in pliable community, socially
acceptable interventions)
¦ Public participation early for siting
¦ Set monitoring plan teams (N & P monitoring,
experiments, monitor costs)
¦ Prioritize the technologies
¦ Make sure town can get credit from pilots,
through monitoring
¦ Focus on transferable/scalable results
Marston Mills (watershed-based permit, PRB at
Prince Cove (MBL), Cranberry Bog restoration
(BCWC/DER), stormwater technologies that have
been designed (all 42) (APCC), Dredging Mill Pond
(Town), l/A systems on 2 islands in Prince's Cove
Feasibility study for bog N attenuation in current
active and inactive cells and hydrology aimed at
alteration
Horse Farm - potential for small PRB
Local practitioners take
the lead
Locals report/monitor
Contractors implement
State/Fed funders
DPW and other
municipal depts.
BCWC
MBL
DER
APCC
>
TO
JZ
OJ
m
o3
OJ
u
c
TO
CL
OJ
u
u
<
~ro
u
o
if)
OJ
no
c
TO
JZ
u
Social Science:
identify the right
incentives/
engagement for
citizen action and
education
Compile case studies outside of Three Bays (e.g.,
Chesapeake Bay and other places to learn from;
first step before a pilotwhat interventions were
socially acceptable?)
Citizen education/engagement/participation,
reach diverse audiences
¦ Understanding the socioeconomic
perspectives on ecological solutions
ID local, state, federal sources to supplement the
cost or tax
Investigate ways to get people to change their
behavior, develop economic incentives
CCC (clearinghouse)
ORD social scientists
91
-------
no
c
c
o
Monitor: coastal,
freshwater, (and
interventions)
Assessment of where we are monitoring to ID
where to monitor/take actions
Expert/Lit review of state of science for monitoring
technologies for this type of effort
Continuation/formalization of monitoring surface
waters (long-term, large-scale, start own
database), including baseline on lakes, wetlands,
estuaries
Make monitoring plan w/ state (consultants)
Regional and local
agencies
_Q
ro >;
+-»
IS) =
CD
+-» 03
03
?/)
C
-------
Supplement A- Participant List
Ahern, Jack - UMass Amherst
Barbaro, Jeff - USGS
Barth, Ed - U.S. EPAORD
Baumgaertel, Brian -
Massachusetts Alternative Septic
System Test Center (MASSTC)
Belaval, Marcel - U.S. EPA Region 1
Berry, Walter - U.S. EPAORD
Bowden, Alison - The Nature
Conservancy
Burns, Sara - The Nature
Conservancy
Cambareri, Tom - Cape Cod
Commission
Caseau, Sheri - Martha's Vineyard
Commission
Charette, Matt - WHOI
Chintala, Marty - U.S. EPA ORD
Crocker, Zenas - Barnstable Clean
Water Coalition
Dannhauser, Casey - Barnstable
Clean Water Coalition
Dixon, Sean - U.S. EPA Region 1
Dore, Bryan - U.S. EPA Region 1
Dudley, Brian - MassDEP
duPont, Carolyn - Quantified
Ventures
Ely, Matt - USGS
Erban, Laura - U.S. EPA ORD
59 Total
Participants
Feuerbach, MaryJo - U.S. EPA
Region 1
Foreman, Ken - Marine Biological
Laboratory
Forest, Mark - Town of Yarmouth
Giblin, Ann - Marine Biological
Laboratory
Gleason, Timothy - U.S. EPA ORD
Goff, Kendra - FL Bureau of
Environmental Health, Division of
Disease Control and Health
Protection
Gottlieb, Andrew - Association to
Preserve Cape Cod (APCC)
Hackman, Alex - MassDER
Heinemann, Kristina - U.S. EPA
Region 2
Heufelder, George - Massachusetts
Alternative Septic System Test
Center (MASSTC)
Horsley, Scott - Water Resources
Consultant to BCWC
Howes, Brian - UMass School of
Marine Science & Technology
(SMAST)
Hubbell, Bryan-U.S. EPAORD
LeBlanc, Denis - USGS, New
England Water Science Center
Lee, Tom - Conservation
Commission/Health Dep
Lloret, Javier - Marine Biological
Laboratory
I Academic
I Consulting
I County
i EPA non-ORD
l EPA ORD
I Federal (non-EPA)
INGO
I Regional Planning Orsanization
Martin, Dave - U.S. EPA ORD
Materne, Meg - Barnstable Clean
Water Coalition
McKean, Tom - Town of Barnstable
Merrill, Nate - U.S. EPAORD
Mulvaney, Kate - U.S. EPA ORD
Munns, Wayne - U.S. EPA ORD
Neill, Chris - Woods Hole Research
Center
Osei, Andrew - MassDEP
Perks, Chance - SMAST
Perry, Erin - Cape Cod Commission
Rea, Anne - U.S. EPA ORD
Reitsma, Joshua - Cape Cod
Cooperative Extension
Rockwell, Heather - Barnstable
Clean Water Coalition
Roeder, Eb - FL Bureau of
Environmental Health, Division of
Disease Control and Health
Protection
Santos, Dan - Town of Barnstable
Steen, Robert - Town of Barnstable
Thurston, Hale - U.S. EPA ORD
Turkington, Eric - Falmouth
Turner, Adam - Martha's Vineyard
Commission
Twichell, Julia -SSC, U.S.EPA, ORD
Valiela, Ivan - Marine Biological
Laboratory
van Drunick, Suzanne - U.S. EPA
ORD
West, Michelle - Horsley Witten
Wigand, Cathy - U.S. EPA ORD
15
93
-------
Appendix K
Consensus Building Institute Day One DRAFT meeting summary
Day One: Problem Formulation Workshop
Hosted by US EPA Office of Research and Development
Tuesday, October 30, 2018
Loeb Laboratory
18 MBL Street, Woods Hole, MA 02543
DRAFT Meeting Summary from Consensus Building Institute
Overview
A problem formulation workshop, hosted by the US EPA Office of Research and Development
ORD), was held on October 30, 2018. The purpose of the workshop was to identify key
knowledge gaps and research needs for the sustainable management of nutrients in the Three
Bays watershed. The overarching goals of the initiative are to:
9) Develop a comprehensive approach to reduce nitrogen loading to meet the TMDL and
achieve water quality goals in Three Bays
10) Identify data, knowledge, resource, and regulatory constraints and develop plans to
address them
11) Build partnerships that can foster innovative approaches to reducing nitrogen loadings
12) Develop models, tools, and strategies that can be transferred to other watersheds
At the workshop, stakeholders from relevant towns, research institutions, and regulatory
agencies gathered for the day specifically to identify the potential research contributions and
approaches for key workshop partners to resolve the identified knowledge gaps, build
relationships and identify additional needs and approaches for fostering partnerships and
networks within and beyond the watershed, and to explore how work in the Three Bays
watershed could be transferred to other watersheds that face similar challenges. The goals of the
workshop were to:
5) Clarify the key knowledge gaps and relationship/partnership needs related to potential
nitrogen removal technologies and the watershed as a whole
6) Identify the potential research contributions and approaches for key workshop partners to
resolving the identified knowledge gaps
7) Build relationships and identify additional needs and approaches for fostering
partnerships and networks within and beyond the watershed
8) Explore how the work in three bays watershed can be transferred to other watersheds
facing similar challenges
94
-------
Meeting Opening
The morning began with welcome and introductory remarks from Suzanne van Drunick, ORD
National Program Director for Safe & Sustainable Water Resources Research Program. She
introduced ORD's pilot project focused on innovative, solutions-based research and its role in the
development of ORD's annual research plans, which would include nutrients as a key topic. She
also mentioned EPA's role during the workshop, emphasizing that it was to listen and hear the
issues.
Bryan Hubbell, ORD's Senior Advisor on Social Science presented in more detail on solutions-
driven research and translational science. He defined the translational science model as one that
works to "help us, help you". He stated that ORD is committed to producing research that
specifically addresses "real world problems in a real-world context" allowing for moving towards
solutions that are implementable with partners. He elaborated that this model is centered on
stakeholder involvement, with a desire to engage with stakeholders from the beginning to the
end of a process (e.g. beginning with developing research plans through the delivery of results).
He concluded by highlighting his hopes to learn about the ways to effectively engage with
stakeholders to better understand needs, collaborate, and to efficiently integrate research
activities (e.g. problem formulation, science approach, evaluation of effectiveness, and
documentation).
Stacie Smith, Senior Mediator from the Consensus Building Institute and meeting facilitator, then
provided an overview of the day, some suggestions for "How to Be," and reviewed logistical
items. She then introduced Zenas Crocker, Executive Director of the Barnstable Clean Water
Coalition (BCWC).
Mr. Crocker shared a presentation on the Vision for Three Bays. In his overview of the nutrient
management challenge in the watershed, he noted Three Bays wins the "Bronze award for
nutrient overload" with rivers as the main contributors. He noted that the requirement was to
reduce half of the nutrient load in the watershed, and that any potential for municipal treatment
was far into the future, making the watershed ripe for more immediate alternative technologies.
He reminded the audience that at any increases in housing would ultimately lead to a dramatic
increase in the nutrient load and therefore increased removal requirements. Mr. Crocker then
gave a brief history of recent measurement and management efforts, noting that BCWC had
been measuring the Marston Mills river weekly for just over a year. He highlighted key areas of
focus for research; in particular, price performance for the different intervention options
available, developing alternatives to integrate the various solutions, and Barnstable's new
financing study with The Nature Conservancy to understand options for bringing these different
alternatives to scale. He concluded by stating Three Bays needed to reduce it's nutrient load by
20K kilograms per year (kg/year), which could rise to 30K kg/year if greater development was
allowed.
In response to Mr. Crocker's presentation, meeting participants offered the following questions
and comments, grouped here by key themes:
95
-------
Conceptual Clarifying Questions:
What is an l/A System; what is PRB? Innovative/Alternative Septic System; PRB is a Permeable
Reactive Barrier, a technology that has been used to clean superfund sites and is now being
tested for nutrients.
What is fertigation and how is that used? Fertigation wells have the potential to be
successful; they intercept nutrient plumes and pump up the nutrient loaded water to use for
fertilizing. This can use nitrogen in the water to treat grass and plants.
What about phosphates? While regulations driving the 208 Plan focus on nitrogen, we are
certainly also trying to understand what can be done with phosphorous. We believe there are
ways to address this as well as other elements such as contaminants of emerging concern
(CECs). A recent test to understand CECs actually found fewer problems than we expected.
Where did the removal numbers come from? The MEP studies provided pretty good numbers;
these are the source ofTMDL numbers; The TMDLs are an ecological target set into EPA and
MADEP's regulatory structure. DEP sees the removal numbers as placeholders for projection
of anticipated reduction of a nitrogen load that will be substantiated by future monitoring
and evaluation of the implementation approaches over next 5-10 years as installed in the
town;
Emphasis on the importance of pilot programs was made by Mr. Crocker who stated"...pilot
programs are important because we have to figure out smaller scale testing to get some
numbers that will tell us to move on to the next step for scaling."
Permitting
Given the importance of treating the river flow, the issue becomes one of permitting. If you
divert the rivers, how do you deal with this? This is the whole point of the watershed permit-
to bring in everything that is being proposed under one umbrella, including things that
already have a regulatory/permitting pathway and things that do not. For example,
aquaculture and fertigation don't have distinct permitting pathways - there needs to be a
method of folding this in, permitting it and assigning a nitrogen credit for each element of an
overall watershed management plan.
Does this mean that the watershed permit designation is applicable to testing the technology
as opposed to only implementing a full-scale solution? Yes, the watershed permit is set up as
a 20-year permit w/a 40 year basic design horizon. The first 20-year permit is broken up into
5 year phases which includes testing as part of an adaptive management approach that
allows us to evaluate effectiveness of alternative approaches and collect data to assign
credits to use this to move to the next phase. Then, we look at the data and determine actual
effectiveness. Where results don't work as proposedwe scale back and/or give no credit. In
first 20 years, there is the possibility of looking at several approaches before looking at
contingency plan to ensure approaches work.
The science is easy, the permitting is the death of many of these processes. Have to be
conservative and have had this problem, but we have gotten relief from DEP. For example,
we didn't used to be able to touch cranberry bogs, and now we can. However, Mill Pond is a
perfect target for restoration and hasn't happened because haven't been able to get the
permits. If we can't come to resolution on permits, this will not take 20 years, but 70.
Price performance basis can be very successful, conditions are deteriorating; everything
needs to be data-based
Persons trying to implement this have the political will/funding, but what we run into is
"green-on-green crime": jurisdictional permitting that different agencies feel they have.
96
-------
Concurrent Discussion Part 1: Known & Unknowns about Alternative Nutrient Removal
Technology Interventions for Three Bays
Following the presentations, the group broke into four pre-assigned small groups for concurrent
discussions, to identify knowledge gaps that existed regarding alternative nutrient removal
technology interventions for Three Bays (and beyond). These breakout discussions focused on a
subset of approaches (Shellfish/Aquaculture, PRBs/Fertigation Wells, Restoration, and
Innovative/Alternative Septic Systems) to identify key priority technical/scientific questions that
need to be answered about these technologies in order to move forward with piloting and
implementation in Three Bays. The groups were charged with answering the following questions:
5) Did we get the knowns/unknowns right? If not, what did we get wrong?
6) Have we identified they key questions about this technology related to modeling,
monitoring, regulatory issues, economic/financing issues, and social acceptability? Are
there other barriers we need to overcome or opportunities that we can draw on?
7) What are key research needs/questions?
8) What are the highest priority research needs and questions we should seek to answer in
the next 2-5 years?
They were asked to report back to the full group on the top 3-5 key take-aways from their
session. The following summarizes each group's report and the comments or questions raised by
the full group:
Shellfish/Aquaculture
Need 1: Demonstrate a project can reach its target; what is success? We need an effective
case study
Need 2: How does an increase in shellfish harvest affect economics?
Need 3: Comparative analysis of the destruction of the bottom based on aquaculture vs.
dredging, moorings, etc.; What is the balance?
Need 4: Are their secondary markets for shellfish? If we are going to expand outputs, we will
need to look further.
Need 5: How do you scale research to waterbodies from smaller scale experiments?
Questions/Commen ts:
There is opposition to shellfish being used to remove nitrogen because it has a detrimental
effect on local shellfishing. We would need to conduct an economic impact study and be
sure to protect against these impacts.
Did this group look at other types of aquaculture?
PRBs/Fertigation
Need 1: What about mass balance?
Need 2: What about pathogens?
Need 3: What about contaminants and other CECs (iron, phosphorous, etc.)?
Need 4: Opportunities to scale? What is needed to sell this idea to Barnstable?
Need 5: Tool to design PRBs (after learning how they work side-by-side); constructability
No questions were asked following this report out.
97
-------
Restoration
Need 1: Better understanding of hydrology for semi-managed bogs for the restoration
process
Need 2: Outcome on water quality from restored bogs. We need to identify what we're
looking at early on
Need 3: Types of restoration that we're looking at; what is a definition of restoration;
innovation in wetlands design. Is there opportunity for engineered restoration:
"engineered" vs. "natural"
Need 4: Understanding trajectories of restoration paths; need to be adaptive
Need 5: Need to articulate community engagement/social benefits/education co-benefits
of restoration approaches.
No questions were asked following this report out.
I/A Systems
Need 1: Better understand nitrogen loads coming into the system (influent)
Need 2: Updated cost analyses/cost-benefit analysis
Need 3: How sensitive is MEP to adjusting loads to/from sites? Higher concentrations of
septic tank effluent, how does this translate to what is coming out of the soil; what kind
of sensitivity analysis can be performed to see if those new data can be accounted for?
Need 4: Innovative ways to get enhanced l/A systems more widely tested. There are not
many available right now (watershed permits?).
Need 5: Develop sensors for diagnostic and onsite performance enhancements IOT;
lower cost/improve performance
Need 6: Different models for promoting implementations of I/As/Enhanced I/As
o Community infrastructure
o Utility model
o O&M
o Lowering costs
Questions/Comments
Clarification: Understanding the load and/or the concentration? Group disagreed on whether
or not this is well-known.
Social acceptance: any discussion about treating it right at the source, whereas most of the
other groups were not No, we did not discuss this.
Did you discuss any alternative models? Not any specific, but another model: manufactured
incentives (get preferred status if donate x number of systems for trial)
Would there be interest in considering the role of green covers? Biomass of forest in Cape Cod
has doubled. Currently ignoring one potential source of nitrogen; conservation easements as
another tech option.
98
-------
Concurrent Discussion Part 2: Putting it all Together
After lunch, Ms. Smith prepared the group for the second set of breakout sessions, reminding
participants to focus on key watershed-wide questions and to identify highest priorities from
those and the needs defined in the morning sessions. She informed the group that the same
facilitators from the morning would guide each of the four sessions and report out on the top 3-5
highlights from each groups' conversations.
She asked the working groups to focus on the following question:
Above and beyond intervention-specific questions, what are the broader systems-level key
research needs/questions for the watershed as a whole related to modeling, monitoring,
regulatory issues, economic/financing issues, and social acceptability?
After the break-outs, Ms. Smith thanked the whole group for its productive conversations and
opened the floor to the small-group facilitators, to share summarized report-outs from their
breakout groups. The facilitators provided the following ideas, themes, and questions that had
been raised by group members in the small group discussions.
Group A:
Need 1: Monitoring three ways - need freshwater delivery, salt water estuary response, and
end load reduction monitoring
Need 2: Modeling to scale up to watershed level: have tools for some interventions and not
others, uncertainties on some model outputs
Need 3: Locating Interventions: many unknowns in Nitrogen removal, SL impacts and need
this information from monitoring three ways (above)
Need 4: Groundwater and surface water modeling: will need site-specific information for
interventions
Need 5: Permitting: Watershed permits and pilot scale permits: where does it make sense to
optimize?
Need 6: Social barriers & acceptability to implementation of technologies
Group B:
Need 1: Stop modeling and put things in the ground - fine points of modeling left, but
we have enough information to start putting things in the ground
Need 2: Permitting - one huge impediment to doing this; how do we deal with this?
Need 3: Fear of solving all of these problems over the next couple of decades only for
nitrogen - need to think about other issues (e.g. CECs)
Need 4: Financing - do so in a way that that shares risk
o Possibility of using cap/trade or some solution to the funding dilemma
Need 4: Climate change: Need to do what we're doing/thinking in terms of climate
change. Also, how do we align these solutions with increased development?
Group C:
Need 1: "Just do it" - pick one or more approaches and one or more sub-watershed, and
"hit 'em hard, hit 'em heavy"; identify funding sources execute and support the effort.
o Pick small watersheds so people can see the investment impact made
Need 2: Permitting - expedite the permit process
99
-------
Need 3: Testing - need a way to pull this information together and make it available
Group D
Need 1: Incentives: which incentives and at what level of incentives
Need 2: Cost/kg of nitrogen removed - site specific and per each technology
Need 3: Social aspect: how do we educate the watershed population
Need 4: Prioritization of technology interventions in their implementation - not just
prioritizing the interventions, but also their implementation
Need 5: Interventions: how to increase engagement and understanding leading to buy-in
Need 6: Climate change: Including stressors in modeling
Drawing from these small group report-outs, the planning team identified nine key themes that
emerged as priorities.
Pilot: identify sites, put projects in the ground, monitor them
Funding: what the sources, innovative, opportunities, etc.?
Monitor: coastal, freshwater, (and interventions)
Social Science: identify the right incentives/engagement for citizen action and education
Permitting: identify opportunities for changes and coordination
Prioritization: understand cost/kg for changes of different options
Prepare for change: impact on environmental loading and on effectiveness of
interventions due to climate change, CECs, second order effects, increased
development/load
Scaling up: what else do we need to know to move scale?
Centralizing information and data sharing
A plenary discussion followed the previous session during which major themes that emerged
were further refined. To spark discussion, Ms. Smith had the group break into groups of two to
three to identify actions and roles for each of the nine themes. The aim of this activity was to
identify what the barriers could be as well as what could be done collectively to move forward.
The chart below illustrates the roles and actions identified by the small groups for each theme.
For each theme, the ideas presented by each group were combined for continuity and clarity.
Pilot: identify sites, put projects in the ground, monitor them
Actions
Roles
If we had the permits, the town is ready
to do dredging on the pond;
aquaculture; (resolve permitting)
feasibility of bogs (baseline work);
restoration of bogs; learn more about
hydrology of the system (resolve
permitting)
Town examining potential of a small PRB
at horse farm where the will is there
(resolve permitting)
Role of a pilot is to extract lessons:
promising practices have been installed
throughout the region as pilot projects.
Funds are what is missing;
Need to extract lessons and share
lessons w/ other practitioners (beyond
Three Bays) to figure out what other
people have done or help them learn to
apply what's been learned here
Local practitioners
Funding partners
100
-------
Apply political pressure w/ respect to
permitting
Pilot-specific monitoring by various
members of the scientific community
Determine how small a pilot can be in
order to get information to scale
Funding: what are the sources, innovative, opportunities, etc.?
Actions
Roles
Develop a list of potential funding
agencies; would describe the funding
partners and create a list, since people
may not be aware of funding partners
Identify different opportunities for
funding different activities; transparent
discussion about funding and partners
would be helpful since we haven't really
done anything on that yet.
Pilot study w/TNC launching with
Barnstable with results that could be
shared
EPA could play role in providing list
Private funding: Impact investments for
private funding to come into this process
and help share some of the cost of why
it's spread construction and play a role in
the fact that tax payers are giving their
money to something that is making
progress
Traditional funding sources
Monitor: coastal, freshwater, and interventions
Actions
Roles
Need to work with partners who are
putting projects in the ground and
monitoring them so we can understand
nitrogen removal dynamics.
Expert lit review; set the monitoring
plan; do the monitoring; calculate the
credit
Don't reinvent the wheel; need to tie
this monitoring data in: not the pilots,
but the baseline monitoring for the
whole system (mainly aquatic
monitoring); looking at aquatic systems
we are altering and also place we aren't
supposed to monitor.
o Not just focus on estuaries
o Urgency about adding additional
things in
o Data gap analysis as a result of
this workshop?
Towns (a lot), potentially interactions
with the towns to carry out the
monitoring and potentially with the DER;
o What would the role of EPA be
(most of this is at the local level);
o What are the barriers to carrying
this out as funding?
o Ex: roles like George at STC >
great model/standards for
monitoring things that are
protocols, value in setting up
structures like this in terms of
roles to look at some of these
other
State
Other constituents of interest; other
groups who are already doing it (should
continue and expand up to the level we
want to get up to an adequate baseline);
other groups in the region
101
-------
Social Science: identify the right incentives/engagement for citizen action
and education
Actions
Roles
Merge pilot w/ social science: looking at
the 500 homes of relatively affluent
community right near the shorelines to
put in tests technologies
o Even if affordability is a part of it;
need a lot of coordination/
organization/incentives to make
things happen
o Disruption there but great
potential for lessons learned;
would have a concentrated area
where all these systems went in
the ground
o Launch cost needs to be
considered
Bridge perspective at the local level to
the problem of nitrogen loading
This has been done in other places and
need to look at the acceptability studies;
make sure citizen science and education
is involved in terms of acceptability and
alternatives; no sense in piloting a
technologies somewhere that no one
would use them.
Would have to be a private/public
partnership (orgsTBD); is it just
overcoming inertia
Way for social scientist to investigate the
ways to be involved; everyone would
need to work on this issue
CCC or somewhere else
Permitting: identify opportunities for changes and coordination
Actions
Roles
See above on pilots!
Expanding on the watershed permitting
process; pilot project of what we need to
do to recruit; need to develop watershed
permits for them and codify approach
with some regulatory initiative; use
experiences to further advance the idea
of watershed permitting
Three bays community as well as
different regulatory levels
102
-------
Prioritization: understand cost/kg for changes of different options
Actions
Roles
Tie prioritization in with the social
science; identify more specific economic
instruments (cap and trade, reverse
auction) tied in with a quarter
mechanism and education portions
Collect information on the different
technologies that are available, how
effective they are cost-wise, and can
someone do it for cheaper cost/kg of
removal?
Nail down some of the models of how
much it costs/kg if you're using some of
these different interventions
(shellfish/aquaculture, l/A systems,
PRBs, Restoration, etc.)
[Participants of the workshop] "people in
this room"
Prepare for change: impact on environmental loading and on
effectiveness of interventions due to climate change, CECs, second order
effects, increased development/load
(Suggested) Actions
"[Climate change] slips down the list because we are already running as fast as we can to keep
up the monitoring that has already been happening for decades. Committed to sustaining and
adding rigorous monitoring to what's already being done (without an effort change)"
When working on other areas, can we make sure we're building in some preparedness for
these kinds of changes?
o Include CC; CECs, etc. in how choose to respond
Clarification of second order effects: making sure we understand that whatever is being
created is not going to make something worse
If put something new in now and then 4 years down the line a [previous technology]
works, as communities, we have to be cognizant that there are other things out there and
look at [those technologies] in terms of what is happening now.
Scaling up: what else do we need to know to move scale?
Actions
Roles
Don't want so much idiosyncrasy that
this impedes scaling up; need to pave
the way for ease of scaling up
In order to scale up need to provide
evidence of these promising practices;
work to compile findings and lessons
from around the region and share them
Standardization of how we design
processes: there needs to be enough
Cape Cod Commission (CCC) is a good
clearinghouse
Top-down centralized organization
EPA should set the goals and "let science
do science"
103
-------
comparability to get at the core interests
of what people are saying
Centralizing: information and data sharing
Actions
Roles
Build database and website structure for
these projects
Clearinghouse for Cape Cod projects
Listing of efforts: who is doing what and
where the money is coming from
What do we do about the impacts of
growth? If they do occur, how do we
share the cost?
o Currently have pricing at
$1550/kg (from 2005); redoing
will be $8200/kg; Should not be
able to allow yourself to buy that
mitigation offset
CCC and BCWC
o Plan by CCC out now for review
and public comment: proposal of
actionable things that could help
with institutional accumulation of
funds to finance pilots and other
types of incentives and stop
future loads from occurring at the
same time
104
-------
Appendix L
Pre-workshop questionnaires
PROBLEM FORMULA T/ON WORKSHOP - HOSTED BY EPA AND BCWC
Pre-Session Participant Evaluation Questionnaire
1. How have you been engaged with Three Bays in the past?
2. What are the top three challenges for nutrient management in Three Bays?
1)
2)
3)
3. What information or research would help decision making about nutrient management?
4. How likely do you think Three Bays is to implement alternative technologies to meet their nutrient
management needs? (Please circle a score)
Not going to happen 0123456789 10 Definitely will happen
Why that score?
5. What is your role in nutrient management?
a. Scientist
b. Government official
c. Manager
d. Other, please specify:
Circle one: Local / State / Federal
Circle one: Local / State / Federal
If you want to include your name, please feel free to add it here:
105
-------
Appendix M
Post-workshop (exit) questionnaires
PROBLEM FORMULA T/ON WORKSHOP- HOSTED BY EPA AND BCWC
Post-Session Participant Evaluation Questionnaire
Problem Formulation Workshop Goals:
13) Clarify the key knowledge gaps and relationship/partnership needs related to monitoring,
modeling, intervention selections, and social science for the potential nitrogen removal
technologies and the watershed as a whole, and other key issues that need to be resolved in
order to achieve the initiative goals.
14) Identify the potential research contributions and approaches for key workshop partners - ORD,
USGS, and others - to resolving the identified knowledge gaps.
15) Build relationships and identify additional needs and approaches for fostering partnerships and
networks within and beyond the watershed to help achieve goals.
16) Explore how the work in Three Bays watershed can be transferred to other watersheds facing
similar challenges.
For the following statements, please circle a rating (1 = disagree strongly 5 = agree strongly)
The workshop achieved its goals (above)
The workshop reflected broad, multidisciplinary participation
All voices were heard during the workshop
Overall, the workshop was constructive
The opening statements were a good use of time
The morning breakout group was constructive overall
Which intervention group?
The afternoon breakout group was constructive overall
The final, "Taking it Forward Discussion" was a good use of time
2 3 4 5
2 3 4 5
2 3 4 5
2 3 4 5
2 3 4 5
2 3 4 5
2 3 4 5
2 3 4 5
1. Overall, what did you get out of today's workshop?
2. What was the best part of the day for you? What would you have changed?
3. Is there anything else we did not address today that you want addressed in future?
106
-------
4. After today, what are the top three challenges for nutrient management in Three Bays?
1)
2)
3)
5. What information or research would most help decision making about nutrient management?
6. Who else do you recommend should be contacted about information discussed at today's meeting?
7. How likely do you think Three Bays is to implement alternative technologies to meet their nutrient
management needs? (Please circle a score)
Not going to happen 0123456789 10 Definitely will happen
Why that score?
8. EPA will be formulating a research plan using input from today's workshop. What other next steps
would you like to see? How would you like to receive information about research efforts going
9. Anything else you want the facilitator or organizers to know about your experience today?
10. What is your role in nutrient management?
1)
2)
3)
forward?
a. Scientist
b. Government official
c. Manager
d. Other, please specify:
Circle one: Local / State / Federal
Circle one: Local / State / Federal
If you want to include your name, please feel free to add it here:
107
-------
Appendix N
Detailed systems map of Day One, Problem Formulation Workshop, created with Plectica using Distinctions, Systems, Relationships, Perspectives
(DSRP) visual grammar (Cabrera and Cabrera 2015). This map is an expanded version of Figure 8,
Three Bays Watershed
private property (
horse farms
Marsfons Mills (1/2 of N load)
N load
collects data on
to estimate mass of
Hydrology * /
groundwater flow
institutions In attendance
wr
O BCWC
bari:-3lab« ciean water coalition
riB
" WHOI
wontfe. hols onBancqraphir
MASSTC
Q Heufeklfcf /
© MA DEP
depterwipttecllon
ccc
cape cod commission
TNC
the nature conservancy
© Quantified Ventures
© Horsley Wrtten
* EPA
MBL
martne biological laboratory
v <> Towns
homeowners
0 UMASS
© CC & MV Commissions
capo and vine-yard
integrates/allows credit ft
watershed permit
assigns numbers
/
loT
Internet of twngs
/
Z
could inform
Technology
PRB
** IA septics
COSlB «
------- |