SEPA
EPA 600/R-20/411 | November 2021 | www.epa.gov/research
United States
Environmental Protection
Agency
Communication Planning
In Solutions-Driven Research
Office of Research and Development
Atlantic Coastal Environmental Sciences Division
-------
EPA/600/R-20/411
November 2021
Communication Planning
In Solutions-Driven Research
by
Alexie Rudman
Katherine Canfield
Kate Mulvaney
Atlantic Coastal Environmental Sciences Division
Center for Environmental Measurement and Modeling
Narragansett, Rhode Island, 02882
Caroline Ridley
Health and Environmental Effects Assessment Division
Center for Public Health and Environmental Assessment
Research Triangle Park, North Carolina, 27709
Office of Research and Development
Atlantic Coastal Environmental Sciences Division
Center for Environmental Measurement and Modeling
Narragansett, Rhode Island, 02882
-------
Notice
The U.S. Environmental Protection Agency, through its Office of Research and Development, developed
this communication planning report as related to the Nutrients Solutions-Driven Research Pilot that
focused on Cape Cod, Massachusetts. This report has been subjected to the Agency's peer and
administrative review and has been approved for publication as an EPA document. Mention of trade
names or commercial products does not constitute endorsement or recommendation for use. This is a
contribution to the EPA Office of Research and Development's Safe and Sustainable Water Resources
Research Program.
-------
Abstract
This report documents the approach used to create a communication plan for an engagement-
intensive research pilot at the U.S. Environmental Protection Agency's Office of Research and
Development. The research pilot aims to support agency partners and stakeholders in meeting their
nutrient reduction goals in impaired waterbodies on Cape Cod, Massachusetts. The communication
planning process and methods used for this pilot, the Nutrients Solutions Driven Research Pilot
(Nutrients SDR Pilot), can be applied when designing communication efforts for future research
projects. The methods and strategies presented in the report are derived from a collaborative process
to develop a project-specific communication plan for the Nutrients SDR Pilot focused on partnering
with stakeholder organizations to investigate solutions to reducing excess nutrients on Cape Cod,
Massachusetts. The report details the process of creating this plan, including setting goals, identifying
stakeholders, analyzing relevant stakeholder involvement, implementing a targeted communication
strategy, and evaluating the effectiveness of communication efforts. The approach outlines methods
and considerations for preparing a communication plan in the context of the Agency and its
community of stakeholders. The materials presented in this report are for agencies and organizations
who conduct research or work with communities to develop and implement a communication plan
focused on effectively engaging stakeholders throughout the entire research process. Users of this
report will need to adapt the tools presented here to the context of their organizations and research
questions, as appropriate, to add value to specific research activities. The underlying tenets of the
Nutrients SDR Pilot's implementation and communication efforts were trust building, consistent two-
way communication, targeted updates, flexible timelines, and openness to iteratively updating the
stakeholder network. This is a pilot research effort: the scientific research as well as the
communication tools are experimental. As the different communication tools, including the methods
for planning that are outlined in this report, are implemented, important lessons for adaptation may
become clear. These methods have been developed and applied to further best practices in
communication and engagement for stakeholder-centric research, and many of these methods are
valuable tools for application in future research efforts focused on meeting community needs that
require extensive community engagement and communication.
-------
Foreword
Under the EPA's congressional mandate to protect land, air, and water resources, the EPA's Office of
Research and Development (ORD) is responsible for conducting scientific research to inform Agency
decisions and support the emerging needs of stakeholders. These stakeholders include other federal
agencies, state and local agencies, tribal nations, localities including towns, regions, non-profit
organizations, and other community members. ORD's research contributes to the scientific knowledge
base required to manage ecological resources, understand how pollutants affect human health, and
prevent or reduce future environmental risks.
ORD provides the scientific and technical foundation the Agency relies on to fulfill its statutory
obligations and address the most pressing environmental and public health challenges. By working
closely with external stakeholders throughout research planning and implementation, ORD ensures a
collaborative, transparent, and highly coordinated research program. ORD oversees six research
programs, including the Safe and Sustainable Water Resources Research Program. The Safe and
Sustainable Water Resources Research Program delivers the data and information that Agency
program and regional offices need, while also providing resources that help state and local agencies,
tribes, and communities inform actions to protect their environment, safeguard public health, and
support a robust economy.
ORD has emphasized solutions-driven research in the current set of Strategic Research Action Plans
(StRAPs). Solutions-driven research focuses on stakeholder needs and involvement and spans from
problem formulation, research execution, to the evaluation of relevancy, usefulness, and timeliness of
deliverables. By working closely with partners throughout the research cycle, ORD also ensures that
the results - and their translation - address real-world problems and help our stakeholders make
timely decisions based on science.
The Center for Environmental Measurement and Modeling (CEMM) within ORD builds necessary
methods and models to implement environmental statutes and improves the Agency's ability to
measure and model environmental contaminants. Research in CEMM encompasses tracking
contaminants in the natural environment, including contaminants of emerging concern, and
developing regulatory methods, models, and environmental indicators. Researchers also work to
develop tools that inform and evaluate environmental management practices and policies. This report
serves as one such tool to inform communication planning that prioritizes stakeholder involvement in
environmental research. Here, we present this transferable tool within the context of solutions-driven
research on nutrient management on Cape Cod, Massachusetts.
Wayne R. Munns, Jr., Director
Atlantic Coastal Environmental Sciences Division
Center for Environmental Measurement and Modeling
Office of Research and Development
iv
-------
Table of Contents
Notice ii
Abstract iii
Foreword iv
Table of Contents v
Table of Figures vi
List of Tables vi
Acknowledgments vii
Glossary viii
Acronyms and Abbreviations x
Executive Summary xi
Introduction 1
Process for Solutions-Driven Research Communication (A) and Evaluation (B) 2
A. Communication Planning Process 5
A.1. Problem and Project Formulation 5
A.2. Establish Communication Goals 7
Stakeholders & Partners 12
A.3. Map Stakeholders 16
Limitations of Stakeholder Involvement 20
A.4. Implementing Strategic Communication 22
Timelines 28
B. Building an Evaluation Approach 30
Preparing for Evaluation 30
Logic Models in Evaluation 31
B.1. Identify Potential Metrics 33
B.2. Develop Evaluation Methods & Tracking Tools 35
Identifying & Implementing Lessons Learned 37
Conclusion 39
References 40
Appendix A: Metrics to Monitor and Evaluate Communication Plan Goals for SDR 44
Appendix B: Social Network Analysis: An Additional Communication Tool for SDR 46
Appendix C: Additional Logic Model Information 48
Logic Model Tables 49
Appendix D: Interview Questionnaire 55
Scripted Project Description 55
Questionnaire for ACESD Researchers 55
Questionnaire for Stakeholders 57
Appendix E: Nutrients SDR Pilot Tracking Tools for Metrics 59
Appendix F: Summary of Communications Plan 60
V
-------
Table of Figures
Figure 1. Illustration of a Solutions-Driven Research Cycle 1
Figure 2. Steps of Communication and Evaluation Planning 3
Figure 3. SMART Goal Characteristics in Communication Planning 8
Figure 4. Overall Pilot Goals and Communication Goals for the Nutrients SDR Pilot 9
Figure 5. Diagram of the Relationship Between Stakeholders and Partners 13
Figure 6. Identifying Stakeholder Groups in a Project and Communication Plan 14
Figure 7. Questions to Help Identify Stakeholders 15
Figure 8. Stakeholder Map Identifying How to Prioritize Time and Communication 16
Figure 9. Stakeholder Map Identifying Stakeholder Groupings 18
Figure 10. Example of a Stakeholder Map 19
Figure 11. Steps to Formulate Communication Targeted to Each Stakeholder's Role 24
Figure 12. Simple Format and Description of Logic Models and Their Application 32
Figure 13. Compendious Logic Model for the Nutrients SDR Pilot 33
Figure 14. Metrics Associated with Two Communication Goals 34
Figure 15. Example of a Social Network Analysis Map 47
Figure 16. Worksheet for Nutrients SDR Logic Model Development 48
List of Tables
Table 1. Stakeholder Categories and Responsible Parties for Engagement 20
Table 2. Stakeholder Categories and Group-specific Communication Approaches 25
Table 3. Methods for Communicating with Diverse Stakeholders 26
Table 4. Group Timeline for SDR Project Research Team 29
Table 5. Monthly Report Semi-structured Data Entry Form of Stakeholder Engagement 36
Table 6. Correspondence Between "What We Do" boxes on the Logic Model in
Figure 13 and Nutrients SDR Pilot Goals 49
Table 7. Correspondence Between "Short-term Impact" Boxes on the Logic Model in
Figure 13 and Nutrients SDR Pilot Goals 52
Table 8. Metrics for the Nutrients SDR Pilot Organized by Tracking Tool Type 59
vi
-------
Acknowledgments
Timothy Gleason, Marnita Chintala, Walter Berry, Nathaniel Merrill, Laura Erban, and Josh Sawyer
(ORISE Fellow) of the Atlantic Coastal Environmental Sciences Division (ACESD) provided extensive
feedback on the communication plan about which this report is written. We would also like to thank
the reviewers Kathleen Williams of the Great Lakes Toxicology and Ecology Division, Beth Hassett-
Sipple of the Center for Environmental Measurement and Modeling, Brittany Kiessling of the Center for
Environmental Solutions and Emergency Response, Anne Rea of the Office of Research and
Development, Emily Smith of the Office of Research and Development, Michelle Latham of the Office
of Research and Development, Julia Twichell of the Narragansett Bay Estuary Program, and Rens van
der Vegt of McGill University for their thoughtful comments on the report. This is EPA ORD STICS
tracking number EPA/600/R-20/411. The views expressed in this paper are those of the authors and do
not necessarily reflect the views or policies of the U.S. Environmental Protection Agency. Any mention
of trade names, products, or services does not imply an endorsement by the U.S. government or the
U.S. Environmental Protection Agency. The EPA does not endorse any commercial products, services,
or enterprises.
vii
-------
Glossary
Key terms and technical terminology as used throughout the report are defined here. These terms are
in bold lettering throughout the report upon their first use after the executive summary.
Aims of Communication: purpose in communicating with a certain audience.
Communication: traditional efforts to inform community members focused on research updates along
with numerous stakeholder engagement strategies to cooperatively design and implement research
projects and incorporate findings into management efforts. In this report, solutions-driven research
(SDR) encourages a holistic approach to communication that centers around stakeholder involvement
throughout the research process.
Communication Plan: a project-specific document that outlines a coordinated strategy of engagement
with communities outside the research institution leading a project. Components may include a
statement of purpose for the plan, communication goals, a stakeholder list that denotes which aspects
of the project are relevant to each stakeholder, an evaluation plan, and a timeline of planned tactics
and activities.
Evaluation: the process of assessing whether goals are achieved, including both in-progress (formative)
and summative (impact) assessment.
Formative Evaluation: a method forjudging progress towards goals while the project is going on. This
method can be applied multiple times throughout a project to check performance and incorporate new
insights.
Frequency: how often a type of communication is used.
Goal: describes an aspect of the broad-reaching vision and desired outcome of the project.
Impact Evaluation: a method used at the conclusion of a project for judging how well project
researchers achieved goals.
Key Partners: stakeholders that are most interested and influential in the project. These are
organizations and individuals that will be conducting research collaboratively with research leads and
may also be conducting research that is intended to complement the project.
Key Stakeholders: organizations and individuals that are the main end users for research results and
who have a strong influence on research prioritization.
Logic Models: a project-planning technique to ensure activities align with the short- and long-term
goals of a project. Allows participants to envision outcomes at different timescales (i.e., short-,
medium-, and long-term).
Messaging: the information that is being conveyed to a stakeholder through communication.
Metrics: either qualitative or quantitative measures used to track progress towards a goal.
Modes: mediums used to communicate.
Other Stakeholders: organizations and individuals that are the end users for research results, who may
be affected by the implementation of research findings and have some influence on research
prioritization.
viii
-------
Partner: a stakeholder who is actively depended upon to conduct, assist with, or consult on a project.
While a partner is a stakeholder, not all stakeholders are partners.
Revisiting: checking in periodically with researchers and stakeholders about their collaborations and
their perspectives on a project's communication efforts.
Social Network Analysis: a statistical method to visualize and quantify relationships between
individuals or entities. Individuals are "nodes" and the known relationships between nodes are
"edges." This method can be used to quantify which node is most critical to a network, based on
having the most edges shared with other nodes, and/or by being a key connector between
communities of nodes.
Solutions-Driven Research: an ORD term for translational science. It is an approach to scientific study
that prioritizes community involvement of those who are likely to be impacted by a project. This
approach aims to center research design, implementation, and evaluation around stakeholder needs,
working closely with stakeholders throughout the research process, and iteratively adapting projects to
meet evolving stakeholder needs.
Stakeholder: any individual or entity that may have influence over, be impacted by, and/or have a
vested interest in the project.
Translational Science: a stakeholder-focused approach to science that entails meaningfully translating
scientific research and observation into results and interventions that improve peoples' lives, coming
out of the medical and public health field as a way to improve communication between doctors and
patients. In our application of translational work at EPA, translational science ensures that stakeholder
needs and perspectives are included in each stage of the research process. Sustained stakeholder
engagement and the incorporation of a diversity of public needs and ideas throughout the research
process; collaboration among multidisciplinary scientists, communicators, and researchers; a focus on
innovative approaches; and the identification of best practices are all key characteristics of
translational science. This definition of translational science has been adapted from the University of
Arkansas for Medical Science's Translational Research Institute (UAMS Translational Research Institute,
2021).
ix
-------
Acronyms and Abbreviations
ACESD
Atlantic Coastal Environmental Sciences Division
EPA
U.S. Environmental Protection Agency
ORD
U.S. Environmental Protection Agency Office of Research and Development
SDR
Solutions-Driven Research
SMART
Specific, Measurable, Actionable, Relevant, Time-based (goals)
X
-------
Executive Summary
This report details the communication planning process for a stakeholder-centric research pilot on
Cape Cod, Massachusetts, in which the U.S. Environmental Protection Agency's (EPA) Office of
Research and Development (ORD) is working closely with stakeholders to design and implement
research to inform watershed-scale solutions to nutrient pollution. This pilot, the Nutrients Solutions
Driven Research Pilot (hereafter referred to as the Nutrients SDR), uses a solutions-driven research
approach to center research design and processes around stakeholder needs to ensure that the
research and translation of results best meet their needs. Excess nutrients on Cape Cod,
Massachusetts, have caused impairments in more than 30 small estuaries and have resulted in
reductions in water clarity, increases in macroalgae, fish kills, and impacts to coastal home values and
recreation. A primary goal of the Nutrients SDR Pilot is to support stakeholders in reducing excess
nutrient loading to meet their policy goals by listening throughout the research process, adapting
research to suit these needs, and using effective communication platforms to clearly translate research
results. Another goal of this pilot is to serve as a test case for designing and enacting extensive and
thoughtful communication efforts throughout ORD's research process. By detailing the communication
planning methods and processes used in the Nutrients SDR Pilot, along with researchers' experiences
with their implementation, this report is designed to serve as a transferable tool to inform the future
design of communication plans for other research projects actively engaging with internal and/or
external stakeholders.
The communication planning methods and strategies described in this report are rooted in the
principles of solutions-driven research (SDR; also called translational science) that emphasize
stakeholder-centric approaches to research. These methods and strategies were derived from a
collaborative process in which the ORD researchers developed a project-specific communication plan
for the Nutrients SDR Pilot. The Nutrients SDR Pilot Communication Plan was developed by ORD
researchers to guide the implementation and communication of research with stakeholders. The
extensive communication planning undertaken for the Nutrients SDR Pilot may not be appropriate for
all ORD research efforts. The methods described here can be used individually or to varying degrees as
needed. The plan developed for the project is a living document designed to adapt to the changing
needs of partners and stakeholders throughout the project's lifespan. Examples are pulled from the
plan to illustrate the implementation of processes and strategies described throughout this report. The
methods described are not an exhaustive list of methods for SDR and other communication plans, but
rather are the subset of strategies used in communication planning for the Nutrients SDR Pilot.
"Communication," in SDR and this report, encompasses traditional efforts focused on research updates
along with numerous stakeholder engagement strategies to cooperatively design and implement
research projects and incorporate findings into management efforts. In SDR, and in this report, there is
a push for a participatory approach to communication centered around stakeholder involvement. The
momentum underlying the pilot project's implementation and communication efforts was built
through trust building, two-way communication, targeted updates, flexible timelines, and openness to
updating the stakeholder network.
xi
-------
This report covers the development of a communication plan based in SDR principles, We begin by
introducing our framework of the communication planning process, which has two major components:
the communication planning process described in section A of this report, and the evaluation approach
in section B (Figure i).
The communication planning process, section A, begins with an overview on engagement in research
problem formulation and provides guidance on establishing both overall project goals and targeted
communication goals. The establishment of clear, measurable, and specific project goals early provides
a strong backbone for projects and development of the engagement strategies. These goals also define
the major outcomes desired for both project research and engagement. Section A also guides
researchers on identifying stakeholders. Due to the breadth of research areas within the Nutrients SDR
Pilot, numerous stakeholders and partners were identified using this process. An understanding of
each stakeholder's level of interest and influence in a project enabled researchers to define
stakeholders' roles in the project. The degree to which project researchers engage with various
stakeholders is achieved by stakeholder mapping. Stakeholder mapping is a method for organizing
stakeholders based on their interest and influence, which guides the development of targeted
communication strategies for different stakeholders. Communication approaches should be revisited
throughout an SDR project's lifetime to ensure efforts reflect stakeholders' most current interests and
needs.
Process for Solutions-Driven Research
(SDR) Communication & Evaluation
A. Communication planning
B. Evaluation approach
Identify stakeholders
& implement strategic
communication
Establish communication goals
Develop & maintain
stakeholder
Identify evaluation
methods &
collect data
relationships throughout
Identify lessons learned
*
Implement lessons learned into future SDR projects
Figure i. Communication and Evaluation Planning Processes for the Nutrients Solutions-Driven
Research Pilot.
xii
-------
Following stakeholder mapping, we describe an array of communication modes to accommodate the
interests of a wide range of stakeholders. The diversity of modes is intended to ensure there are
relevant and usable products for communicating research updates and outcomes. Modes range from
traditional academic articles to a project bulletin and a series of Twitter feeds. We identify appropriate
uses and audiences of these modes of communication. This section also includes examples of project-
wide and team-specific timelines developed to improve coordination and communication among
project researchers.
Section B of this report provides guidance on developing an evaluation for gauging the effectiveness of
the communication efforts. Evaluation is used to understand the relevance, usefulness, and timeliness
of communication efforts, and allows for the incorporation of 'lessons learned' from communication
efforts to improve future communication strategies. This section provides an overview on the use of
logic models as one way to frame an evaluation, provides guidance on establishing metrics to track
progress towards achieving goals, and outlines tools that can be used to track metrics.
The 'Identifying and Implementing Lessons Learned' section highlights considerations that were most
important in the development of the Nutrients SDR Pilot communications plan. Intentional and
appropriate science communication can facilitate broader engagement by including different
stakeholders who can improve the research direction and incorporation of findings into decision-
making processes.
xiii
-------
Introduction
The U.S. Environmental Protection Agency (EPA) Office of Research and Development (ORD) is
piloting solutions-driven research (SDR) approaches to ensure that the research and translation
of results pertaining to solving environmental challenges best meet the needs of the public.
Solutions-driven research, ORD's term for translational science, aims to meaningfully
incorporate stakeholder needs and involvement from the problem formulation stage through
research execution and results communication (Figure 1; all terms that are defined in the
glossary are bolded upon first defined use in this report), This cyclical approach to research
continually revisits stakeholder priorities and ensures research is designed and adapted to meet
their needs. Meaningful engagement works to ensure a collaborative, transparent, and
coordinated research effort. The Nutrients SDR Pilot involves collaborations with a community
partner on Cape Cod, numerous other stakeholders, and over a dozen scientists making up
seven research teams.
This report is a guide to support the creation of future communication plans, project-specific
documents that outline a coordinated strategy of engagement, for SDR projects within ORD or
for other researchers seeking to implement translational science principles in their work.
Communication, in SDR and this report, encompasses traditional efforts focused on research
updates along with numerous stakeholder engagement strategies to cooperatively design and
implement research projects and incorporate findings into management efforts. Highlighted in
this report are the communication planning processes and evaluation approaches applied in the
Nutrients SDR Pilot for FY2019-2022. The communications planning builds from the Nutrients
SDR Pilot problem formulation workshop (Twichell et a I., 2019).
Figure 1. Illustration of a Solutions-Driven Research Cycle. Adapted from the National Institutes
of Health Translational Science Spectrum (National Institutes of Health, 2020).
1
-------
This report provides methods, templates, and recommendations for building a communication
plan that ensures stakeholders are identified and included throughout the research process. In
addition, we propose ways to evaluate the success of the communication approach at the end
of the project. Evaluation is crucial to determine whether an approach is meeting its goals. It
allows project members to identify the aspects of an approach that are most and least
effective, insights that can be incorporated to improve an ongoing project or future projects.
This report describes the tools and approaches employed by the Nutrients SDR communication
team in the development of their communication plan and does not serve as an exhaustive list
of methods for creating a communication plan. This report details the communication efforts of
this targeted approach to research for ORD and provides examples from the Nutrients SDR Pilot
project-specific communication plan to serve as an initial model in future SDR communication
planning. Here, we outline the process of developing a communication plan, the various kinds
of potential stakeholders involved in a solutions-driven research project, and the distinct
approaches to communication for the different stakeholders.
Although the plan for the Nutrients SDR Pilot was prepared after research was underway, we
recommend that a communication plan be established early in a project and revisited often. A
communication plan is intended to guide and inform strategies for communicating and
engaging with stakeholders and is complementary to a separate overall project plan.
Establishing a communication plan from the phase of question development and preliminary
background research is important so that stakeholder communications are not ad hoc and
contribute to achieving desired research outcomes (Druschke & Hychka, 2015). Communication
efforts need to be intentionally integrated as an essential aspect of the project and organized to
reduce stakeholder burden and facilitate synergy across efforts. These plans ensure that the
needs and interests of diverse stakeholders are accounted for and meaningfully included
throughout the process. While ORD scientists have long integrated many of these principles
into their research, the SDR approach is innovative for ORD in that it requires a concerted shift
to integrating stakeholder involvement at every stage of the research process and careful
planning for engagement and communication.
Process for Solutions-Driven Research
Communication (A) and Evaluation (B)
A communication plan allows researchers to streamline their outreach, engagement, and
communication processes; increase their likelihood of meeting stakeholder needs; meaningfully
communicate with different stakeholders; and better evaluate the contributions of SDR
communication-related practices. The process graphic depicts the organization of steps in
communication and evaluation planning in solutions-driven research (Figure 2). It is based on
the framework of a structured decision-making process, informed by the steps taken and
lessons learned in communication for the Nutrients SDR Pilot. The steps will often, but not
always, proceed largely chronologically, with some steps existing as steps in both the
communication and evaluation strategies. Several steps are actively implemented throughout
the entire communication process, such as developing relationships and data collection for
evaluation. To ensure lessons learned are integrated into planning for future projects, the
2
-------
dotted arrows serve to demonstrate structurally where we anticipate these lessons could be
most helpfully incorporated. This report moves through these steps, with section numbers
reflecting the identified topics in the boxes in Figure 2.
Process for Solutions-Driven Research
(SDR) Communication & Evaluation
A. Communication planning B. Evaluation approach
Figure 2. Steps of Communication and Evaluation Planning. The steps bridging the two
processes are shown in the longer boxes in the dark blue. The steps exclusive to the
communication planning and implementation process are shown in light blue boxes on the left
side of the graphic, and those exclusive to the evaluation process are shown in green on the
right side of the graphic. The larger pink box surrounding most of the graphic, "Develop &
maintain relationships throughoutis a continuous effort that is a part of the entire planning
and implementation of SDR communication and evaluation. The dotted arrows show where the
lessons from one SDR project can both be integrated into an ongoing project and inform future
project development. The major headings throughout the remainder of the report follow the
boxes labeled in Figure 2.
3
-------
As shown in Figure 2, communication strategy for SDR (section A) relies on problem formulation
that prompts the development of the project. During problem formulation, stakeholders and
researchers identify what aspects of the problem need to be solved and identify the desired
project outcomes that will demonstrate the project's success. With the project defined, an
important early step is developing communication goals through an iterative and collaborative
process to define the observable and measurable accomplishments that must happen to
achieve the identified project outcome. Goal development is followed by mapping the network
of stakeholders as it currently stands and the varying influences and interests they have in the
project to identify roles in the project. Modifications and additions to this stakeholder map will
occur throughout the project as stakeholders' roles change and as networks expand or
contract. Once roles are established, it is time to implement strategic communication targeted
towards the needs and context of each stakeholder. On larger projects, this step may also
include developing team-level timelines for products produced as part of the communication
process, based on an overarching project-level timeline. After product completion, identifying
and applying lessons learned throughout the SDR communication process will enhance future
SDR projects.
An evaluation strategy (section B) is developed in parallel to the communications strategy
(Figure 2). Evaluation is important to the Nutrients SDR Pilot because of the deliberate focus on
trying new communication methods and identifying successful strategies for use in future
projects. An important bridge between evaluation and communication is defining
communication goals. To ensure that goals are achieved and to measure progress in achieving
them, the first step of the evaluation process is to identify potential evaluation metrics and
assign them to each goal. Based on the identified metrics, the evaluation method is selected,
and tracking tools are developed. At the conclusion of each formative or impact evaluation, the
evaluation identifies lessons learned. If the evaluation is during the project, those insights can
be incorporated into continuing evaluation and communication efforts. If the evaluation is at
the end of the project, lessons learned can be consolidated to inform future projects.
In progressing through these steps and the research process itself, the Nutrients SDR
communication team continually revisits the plan, revises it as is appropriate, and anticipates
future adaptations to the plan. Revisiting the plan means checking in periodically with
researchers and stakeholders about their collaborations and their perspectives on the
communication efforts. This occurs annually, as well as after major conferences, workshops, or
events in which stakeholders participate. While many researchers have engaged in some
aspects of SDR, the scope of communication associated with an SDR approach may be
unfamiliar to others. For example, researchers may be familiar with sharing results with
interested stakeholders at the conclusion of the overall project, or doing educational outreach
events, but not as familiar with the work of building stakeholder relationships and co-creating
research plans. Researchers should be committed to focusing on stakeholder communication
throughout research planning and implementation and identify valuable updates and
communication products for diverse stakeholders. It is also important to collect data
throughout the process for use in evaluation to improve the project's "memory" of
communication efforts.
4
-------
A. Communication Planning Process
A.1. Problem and Project Formulation
How do we define the problem?
The first step in an SDR project is the identification of a catalyst that invites or necessitates
research identifying a problem that needs to be solved, in the case of the Nutrients SDR Pilot,
the catalyst was a lawsuit, and the following refiled suit, of Conservation Law foundation vs.
EPA in 2013, which called into issue the lack of a functioning Section 208 Water Quality
Management Plan for monitoring nonpoint nutrient sources on Cape Cod (CLF vs EPA, 2013).
The agreed upon settlement of the suit was to update the Cape Cod Section 208 Plan, among
other EPA actions. While the plan itself has been updated, the Nutrients SDR Pilot is part of an
ongoing effort to investigate creative, stakeholder-informed solutions to address point and
nonpoint nutrient sources that impact water quality on Cape Cod,
Estuaries on Cape Cod are impacted by excess nutrients in the form of nitrogen primarily due to
the use of backyard septic systems. The cost, the layout of housing, and the seasonal changes in
population make it challenging to widely use wastewater treatment plants on Cape Cod. In the
Three Bays Watershed where this research is primarily taking place, there are over 5,000 septic
systems. Traditional septic systems are not designed to remove nitrogen. This untreated
nitrogen moves easily from the septic systems to the Cape's groundwater, where it is
transported to ponds, streams, and estuaries. Additional sources of excess nitrogen in the
region include lawn fertilization, atmospheric deposition, and stormwater (EPA, 2020).
5
-------
This Nutrients SDR Pilot has grown out of the existing relationships EPA researchers had from
past research projects to address water quality on the Cape. In building on those relationships
to identify the problems of interest to stakeholders in this pilot, an early step was to hold a
problem formulation workshop. We recommend holding such a workshop to identify initial
relevant stakeholders, ensure the research will address the needs of those stakeholders as well
as the catalyst identified in the previous step, and set the tone for stakeholder involvement
with research throughout the project.
For the Nutrients SDR Pilot, the problem formulation workshop was held in October 2018 with
more than 50 Cape Cod stakeholders from local, state, and federal agencies; the research
community; and environmental NGOs to define the problem and identify research needs
related to nutrients and water quality in the Three Bays watershed of Cape Cod. This workshop
was implemented in partnership with the Barnstable Clean Water Coalition (BCWC), which
demonstrated collaboration between the government and local community. The full workshop
report details the workshop attendees and procedures (Twichell et al., 2019).
What is the desired final state or condition we are working towards in research and
communication?
Identifying project outcomes provides an overarching vision for the project that encompasses
both the research and communications aspects of the project. The process of defining these
outcomes occurs in the research planning stage and is informed by the overarching aims noted
in grant or research proposal documents.
For the Nutrients SDR Pilot, the research planning documents informed our identification of
two desired project outcomes. The first is to work with and support our partners in developing
a watershed-level plan to reduce nutrient loading in the Three Bays Watershed on Cape Cod to
meet the total maximum daily loads (TMDLs) for nitrogen as set to improve pollutant levels in
impaired waterbodies. The second desired project outcome is for the research and
communication products serve as examples to support other communities as they address
similar issues of nonpoint source nutrient management.
What We Learned:
We found the project formulation workshop to be an insightful way to gather
ideas from a range of stakeholders and hear from many different perspectives.
Quickly sharing the aggregated results from the problem formulation workshop
helped continue the momentum catalyzed by the workshop for collaborative
research.
6
-------
A.2. Establish Communication Goals ]
What observable and measurable accomplishments need to happen to achieve the
identified project outcomes?
Establishing clear and precise project goals and objectives is a critical early step of project and
communication planning. Goals broadly define what the project is trying to achieve.
Establishing goals enables researchers to determine communication priorities and clearly
identify specific actions needed to accomplish these goals. The process of goal setting is also
valuable to ensure all researchers and partners agree on the vision for the project and expected
outcomes. It is valuable to include partners and stakeholders in the process of establishing
goals for an SDR project to ensure their needs are identified and incorporated into the research
plan (refer to the Stakeholders and Partners section). Given their importance in setting the
stage for researchers' actions, establishing goals should be a priority early on in project
planning.
If the overall project lacks clearly defined overarching goals, the step before identifying the
purpose and goals of the communication plan is to identify the goals for the project. Based on
those goals, the communication goals will become clearer. When it came to overall goals for
the Nutrients SDR Pilot, we first spent several weeks negotiating and establishing overall pilot
goals. Four of the overall project goals were related to communication and stakeholders, which
then shaped the specific communication goals. These goals define the framing of what
communication efforts need to achieve.
Strong goals are commonly described as specific, measurable, actionable, relevant, and time-
based, or "SMART" (refer to Figure 3; Doran, 1981; Sydney Public Relations Agency, 2019; Aus,
2017). While a variety of approaches to goal setting exist, such as the Objective and Key Results
(OKR) approach (Sun, 2017; Washington State Bar Association, n.d.), Locke and Latham's 5
principles of goal setting, and aspirational goals (Washington State Bar Association, n.d.), we
chose the commonly utilized SMART approach for developing communication and project goals.
A SMART approach to goal setting is particularly useful in communication planning to establish
a clear motive for engagement efforts rather than having the team creating ad hoc materials as
results come in or events occur. Designing goals based on this model will also keep people
oriented towards an overall vision and clarify team expectations from the outset. These
expectations should be challenging but realistic to achieve. For example, if using stakeholder
participation in a poll on Twitter to elicit feedback about a topic, does the communication team
have the time, resources, and expertise to assemble and analyze these social media posts? How
does the team plan to measure the change in engagement they are hoping to achieve?
Additionally, having goals in place that provide guidance on strategic communication will
ensure the team maximizes their impact with targeted communication for stakeholders.
7
-------
B* Specific
Goals should specify relevant actors, frequency of measurement
0* Measurable
Establish goals with associated clear quantitative and qualitative
metrics, and mechanisms for recording them to track progress
B* Actionable
Make goals challenging, yet within capabilities, resources, and budget.
D* Relevant
Ensure all goals meet necessary organizational requirements and
regulations and are applicable to the project and stakeholders
D* Time-based
Establish specific date targets for completion, while ensuring the goals
can allow for changing project timelines.
Figure 3. SMART Goal Characteristics in Communication Planning. SMART goals have been
applied in a number of settings and can help establish clear expectations (Doran, 1981; Sydney
Public Relations Agency2019; Aus, 2017).
In the Nutrients SDR Pilot, the communication team implemented a collaborative goal-setting
process. Project researchers and managers spent several months working together to develop
and tailor overall project and communication goals. Given the project's reach, and the variety of
project teams involved with different expertise, the involvement of people with diverse project
perspectives and experiences was invaluable to creating SMART project and communication
goals. With SDR, planning documents, particularly communication plans, are living documents
because of the substantial stakeholder collaboration throughout the entirety of the research
process and the problem-responsive goals. As desired outcomes, plans, availability of
resources, or higher-level visions change, so might the goals. It is therefore necessary to
maintain and update the list of project and communication goals as needed. Outlined in Figure
4 are the overall project goals and the communication goals identified for the Nutrients SDR
Pilot. Identifying these goals was essential in defining with whom and how to communicate
throughout the project. To measure progress towards these goals, we established a number of
quantitative and qualitative metrics for the evaluation. This process is discussed in part B of the
report under B.l. Identify Potential Metrics.
8
-------
Nutrient SDR Pilot Project Goals
1. Support partners to develop a watershed-wide approach to reducing nutrient loading
to meet their TMDL goals.
2. Build and maintain relationships and partnerships to identify and evaluate the most
promising nutrient remediation strategies.
a. Communicate throughout the research, planning, implementation, and evaluation
processes with diverse external stakeholders.
3. Develop an understanding of the economic, social, ecological benefits of and
barriers to each approach for removing nitrogen.
4. Evaluate the effectiveness and cost effectiveness of the most promising nutrient
remediation approaches through pilots and demonstration projects by:
a. Conducting baseline benthic condition surveys in the Three Bays estuary.
b. Conducting a neighborhood scale demonstration study of innovative septic
systems, evaluating system effectiveness, social acceptance of l/A systems, and
environmental impacts in ground water and receiving waters.
c. Contributing to pilot studies on Cranberry Bog remediation approaches.
d. Evaluating potential beneficial uses of sediments dredged from ponds and the
estuary.
e. Evaluating the co-benefits of improved water quality by shellfish aquaculture.
5. Ensure the utility and transferability of research to partners and other interested
stakeholders.
6. Evaluate the contributions of SDR methods to developing promising practices for
engagement with stakeholders.
a. Help researchers understand the value of communication and engagement.
Nutrients SDR Pilot Communication Goals
1. Encourage and sustain partner and stakeholder engagement and trust in a targeted
manner based on each stakeholder's role in the Nutrients SDR Pilot.
2. Produce and deliver accessible products tailored to specific stakeholder needs to
make EPA science more usable to Cape Cod communities and transferable to other
areas.
3. Facilitate two-way communication between pilot researchers and stakeholders
through-out the research process.
4. Evaluate communication and engagement methods to identify promising practices for
future research.
5. Facilitate and encourage internal and external communication with researchers and
managers about the Nutrients SDR Pilot to foster a better understanding of the value
of stakeholder communication.
6. Develop a project-specific SDR Communication Plan to act as a communication model
for future SDR projects in ORD.
Figure 4. Overall Pilot Goals and Communication Goals for the Nutrients SDR Pilot
While the SMART approach guided the creation of the Nutrients SDR and communication goals,
these goals do not adhere to every aspect of SMART goal setting.
9
-------
The goal setting process required much negotiation and deliberation as the Nutrients SDR pilot
encompasses such a wide variety of research areas which made rendering some of these goals
more specific or time-based was difficult. While none of the pilot or communication goals are
explicitly time based, each of these are to be accomplished by the completion of the pilot in
FY2022.
The following breakout box provides an example of how the SMART approach was applied to
one of the pilot goals:
Example Application of SMART Principles
Nutrients SDR Pilot Goal 2:
2. Build and maintain relationships and partnerships to identify and evaluate the
most promising nutrient remediation strategies.
a. Communicate throughout the research, planning, implementation, and
evaluation processes with a diverse array of external stakeholders.
Specific. This goal identifies when and how building and maintaining relationships
will be accomplished.
Measurable. Communication throughout the research process can be measured
by tracking the frequency of formal and informal meetings, site visits, conference
calls, production of research products, and which stakeholders were involved.
Tools can be developed to track these metrics, see Table 5 for an example of our
monthly tracking tool.
Actionable. The infrastructure and resources necessary to build and maintain
relationships through communication already exist and are accessible to EPA
researchers. The modes of communication that EPA researchers can use to engage
with stakeholders, their appropriate uses, and a point of contact for each mode,
are provided on EPA websites and are amalgamated in the communication plan for
the Nutrients SDR Pilot, see Table 3 for a modified version. The social scientists on
the Nutrients SDR Pilot have scientific communication expertise and committed to
providing engagement support.
Relevant. This goal is relevant to the Nutrients SDR Pilot and all stakeholders as
solutions-driven research is centered on incorporating stakeholder needs and
involvement through each phase of the research process, and communication is
the vehicle through which stakeholder needs and voices are incorporated.
Time-based. The timeframe for this goal is "throughout the research, planning,
implementation, and evaluation processes," and while not specified in each goal,
all Nutrients SDR Pilot project and communication goals are to be complete by the
;nd of FY 2022.
10
-------
Who is responsible for keeping on track with communication goals? What are the roles
of researchers in communication? What active role do the various stakeholders assume
in the project currently, and how might this evolve? Who has the most influence in
determining project outcomes?
Clear delegation of communication responsibilities for the project team is important for
meeting communication goals. The team who developed the plan includes a researcher who
listens in on weekly calls between the Science Lead and our key partner, three social scientists
who have led development of the overall and sub-team communication plans and creation of
communication products, and two EPA researchers who serve as planning partners and support
the preparation of the plan and goal identification. Depending on a project's size, it may be
appropriate to adjust the number of communication team members. As SDR requires a
significant time commitment, having a team member with expertise in stakeholder engagement
and time allocated to communication planning and implementation is important.
What We Learned:
While the collaborative goal setting process was demanding, this process was
necessary to reach consensus on pilot and communication goals and establish a
shared vision for a project with many moving parts and stakeholders.
The process of establishing goals and the identification of metrics needed to occur
simultaneously to ensure these goals were measurable, attainable, and that we
had the means and ability to measure progress towards achieving these goals.
11
-------
Stakeholders and Partners
While relationship development may begin at the start of a project or may flow from existing
relationships, moving through each step of the planning process will help further define the
intended audience of project communications (Druschke & Hychka, 2015). Early engagement is
critical to identifying and understanding the needs and values of a broad array of stakeholders,
identifying and agreeing on metrics, acquiring data, and acquiring feedback on project goals,
processes, and results. Stakeholders include "any individual, group, or organizations that can
affect or be affected (positively or negatively) by the system under study and have direct or
indirect influence on its requirements" (Ballejos & Montagna, 2008, p.282). Every research
project will have a unique set of stakeholders depending on the subject matter, geographical
scope, and community impacts. For example, a project in one municipality may not have the
same type of stakeholders for the same topic in a different municipality. In addition to having
more general stakeholders, an SDR project will often have partners as well. The distinction
between stakeholder and partner is that partner implies a dependence on the organization or
individual actor to conduct, assist with, or consult on a project. A stakeholder is directly
affected by a project or could affect its design or implementation in some ways but might not
have as active of a role in research planning or implementation. In other words, a partner is
more directly involved in conducting the research. While a partner is a type of stakeholder, not
all stakeholders are partners (Figure 5).
12
-------
Figure 5. Diagram of the Relationship Between Stakeholders and Partners. The diagram
indicates that while not all stakeholders are partners, all partners are stakeholders.
A significant commitment to building trust with stakeholders is needed for these relationships
to be truly productive (Vangen & Huxham, 2003). This is not a quick process, but it is essential if
researchers hope to truly understand stakeholder needs and work effectively with individuals
outside of their research team. In the case of the Nutrients SDR Pilot, trust building began
before the pilot itself. Key relationships and partners in this project are the direct result of past
and continued efforts among EPA researchers to engage and build trust with local stakeholders.
Trust has been built through frequent, ongoing communication, and demonstrated listening.
Stakeholder participation is important for numerous reasons, including the improvement of
scientific research, as stakeholders bring valuable information and access to data, a deeper
understanding of complex problems, and new ideas or creative solutions to problems (Beierle &
Cayford, 2002). Stakeholder participation can also increase the "buy-in" of local communities
for supporting specific research efforts, increasing the overall likelihood of acceptance of the
work. It also ensures that those who depend on natural resources will be informed about
changes in natural resource governance due to the research project. Incorporating a diverse set
of interests in the research components of the environmental decision-making process can also
help increase perceived fairness (Constanza and Ruth, 1998).
It is important to be as inclusive as possible when engaging stakeholders. Stakeholder processes
can be derailed or thwarted through a failure to include diverse perspectives. Some of these
perspectives include those who can influence the direction of a project, those who are likely to
be affected by the project, and anyone who might be resistant to the project if not intentionally
included (Figure 6). Those who are actively resistant to a project require intentional and
continued engagement to find a path forward that satisfies all actors' needs Invite dialogue
with those who may be resistant early in the process so that their concerns can be adequately
13
-------
addressed in research planning and implementation (Fair-Wright and Juli, 2016). Identifying and
providing meaningful involvement opportunities for marginalized stakeholders who may be
historically and/or continually left out of the research process is considered a best practice
(Bevan et al., 2018). Relative to most of the country, the county in which the Nutrients SDR Pilot
is being implemented is predominantly middle-to-high income and white, though it is stil!
important to consider who might be left out if engagement is not carefully pursued. Reaching
out to community-based organizations and community leaders can be a good way to identify
groups within a community with historically marginalized backgrounds, and to help identify
how to thoughtfully and effectively engage with these groups. Additionally, referring to tools
and resources is useful in identifying people from marginalized backgrounds that need to be
included. Such tools include the Government Alliance on Racial Equity's Racial Equity Toolkit
(Nelson & Brooks, 2015), the U.S. EPA's EJ Screen (EPA, 2021), and state mapping tools of
environmental justice populations such as Massachusetts' Environmental Justice Maps Viewer
(Commonwealth of Massachusetts, 2021) that identifies marginalized groups by ethnic minority
status, language isolation, and income. The practice of identifying stakeholders is highly
dependent on spatial, logistical, social, economic, and regulatory variables.
Individuals or groups
who might work
against the project if
excluded
Individuals with skills
and expertise
specific to the
project
Groups representing
a diversity of
perspectives and
interests in the
project
People from
marginalized
backgrounds
historically excluded
from decision-
making processes
Individuals or
organizations who
can make decisions
that affect the
project or who might
be affected by
project decisions
Who should be involved?
Figure 6. Identifying Stakeholder Groups in a Project and Communication Plan.
In the following sections, examples of the broad set of stakeholders that might need to be
involved are provided through a snapshot of relevant stakeholders working on or affected by
the issue of excess nutrients on Cape Cod. Stakeholders involved in the Nutrients SDR Pilot
ranged from citizen leaders to non-profit organizations and federal and state agencies.
Different stakeholders have varying levels of influence and interest in a project.
14
-------
As a result, communication activities were specifically tailored to ensure the different types of
stakeholders were engaged appropriately through the applicable communication methods with
consideration of their level of influence and interest in the project (or aspects of the project).
The methods and processes that were used in the Nutrients SDR Pilot to identify and
differentiate stakeholders are described in Figure 6.
Answering questions about who will be impacted by, benefit from, contribute to, and have
interest in the project aids in developing a list of potential stakeholders (Maryland DolT, 2021;
Measure Evaluation, 2020; Kenny, 2014). The questions the Nutrients SDR communication team
used to develop their list of stakeholders are listed in Figure 7. Another technique to identify
additional stakeholders is to conduct interviews with known stakeholders and local participants
in related projects that (refer to Twichell et al., 2018b).
Who might impact the project?
Does the person or their organization hold a position from which they can influence the
project?
Can this project exist without the person or organization? Can they be easily replaced?
At what point does the person have the greatest impact on the project?
Are there relationships among stakeholders that might impact the project?
Who stands to benefit from the results?
Who will use the data or results of this project?
Who is impacted most by nutrient pollution issues?
Will the person or their organization be directly or indirectly affected by this project?
Is the person or organization receptive to the project?
Does the person potentially benefit from the project or are they in a position to resist this
change?
Who might contribute to the project?
What does each actor contribute to the process?
Does the person or their organization have any special skills or capabilities the project will
require?
Who might have an interest in the project?
What interest do actors have in this work? Is it positive or negative?
Who is aware of this project? Who might not be aware of this project?
Who will support this project? Who might oppose it? Why, and how does the research team
plan to address this?
Figure 7. Questions to Help Identify Stakeholders. Adapted from Maryland DolT, 2021; Measure
Evaluation, 2020; and Kenny, 2014.
15
-------
A.3. Map Stakeholders
k *
Once a list of stakeholders is assembled, stakeholder maps can be used to identify whom to
engage and how. Stakeholder mapping determines who should participate to ensure lasting
consensus, and how to engage each stakeholder group so their concerns are heard and
represented (for more information on stakeholder mapping, refer to Twichell et al. 2018a, b).
Through a collaborative process, stakeholder groups were plotted on a map according to their
influence and interest/stake in the project and outcomes (Twichell et al., 2018a). Those with
the highest influence and interest are key stakeholders and should receive the most thorough
and frequent communication. Other stakeholders are less frequently engaged and often
targeted with less technical communication efforts (Figure 8). In the Nutrients SDR Pilot, most
homeowners and Cape residents would fall in the lower interest x lower influence category as
they do not have an active role in the research planning and implementation process, and
therefore it is important to keep them informed with quality information on the project without
as sustained engagement as might be sought in other quadrants. It is important to recognize
that the level of engagement required for the public that typically falls in the bottom right
quadrant depends on the context of each project. For projects engaging with those in
underrepresented or historically marginalized communities, those stakeholders may require
resources to amplify their influence and build their interest.
Meet their
needs
Key
stakeholders
Meet needs and engage
with often
Make quality
information
available
Show
consideration
Interest
Figure 8. Layout of a stakeholder map identifying how to prioritize time and communication efforts
among stakeholders (adapted from Twichell et ai, 2018a).
16
-------
Stakeholder mapping is an iterative process. For the Nutrients SDR Pilot, the communication
team collaboratively decided where each stakeholder was placed and revisited the map
multiple times before sharing with the research team and management for feedback. The
stakeholder map was updated to ensure it reflected the best assessment of the stakeholders at
that point in the research process.
The four quadrants were further broken down into different categories of stakeholders (Figure
9). Stakeholder mapping for the Nutrient SDR Pilot deviated from the traditional four-quadrant
stakeholder mapping (Twichell et al. 2018a,b) by identifying key partners and project partners
within the key stakeholders quadrant. We did this because following the more traditional four-
quadrant mapping did not fully characterize the types of stakeholders engaged in the Nutrients
SDR Pilot. Key partners are those in the box denoted by the grid-like pattern in the top-right
corner of the map that have the most interest and influence within the project. This
stakeholder category was added because of the significant difference between contributions
that a key partner (the Barnstable Clean Water Coalition) made in developing and
implementing the Nutrient SDR Pilot's research and the rest of the project partners. The other
project partners are those in the box with the dotted pattern who are also in the top-right
quadrant as they also contribute significantly to the development and implementation of the
research and are slightly less involved in the project. Any other stakeholders in the top-right
quadrant in the pattern-less blue box that are not partners are considered key stakeholders.
Stakeholders in the three remaining quadrants are categorized as other stakeholders. Each of
these three quadrants of other stakeholders requires communication to be targeted in slightly
different ways based on their level of interest and influence over the project, as denoted in
Figure 9 and in the Implementing Strategic Communication section.
17
-------
Key
Other
partner
stakeholders
Project partners
-------
project partners to keep them informed and engaged. In the Nutrients SDR Pilot, we rely on our
key partner to engage homeowners participating in an l/A septic system pilot. It is appropriate
for our key partner to engage these homeowners given our key partner's local expertise,
presence in that community, and pre-established contacts among these homeowners. In a
collaborative project, you can only control the information coming out of your own
organization. In the Nutrients SDR Pilot, we trust that our partners are communicating with
stakeholders in the most effective way possible, and when necessary, we have used weekly
meetings to discuss communication strategies, priorities, and messaging for these other
stakeholders.
0)
u
c
0)
D
_c
PARTNERS MANAGE/BUILD
INTEREST
ฆ EPA OW
Other
permitting
agencies
Local funders
ฆ Town
officials
Other federal
agencies
ฆ Local
environmental
NGOs
Local health
officials
WE PARTNER
WITH/ENGAGE/CONSULT CLOSELY
ป Key partner
Project partners
International environmental NGO
US(iS/ EPA Regional office
ฆ State environmental
agencies/departments
Other ORD agents
PARTNERS ENGAGE/INFORM
Nearby towns
Town residents
Dep. Natural History
Coastal Communities
WE PARTNER
WITH/ENGAGE/CONSULT CLOSELY
Bog
farmers
Consultancies
ฆ Community
organizatior
EPA researchers
Living observatory
ฆ University
researchers
Aquaculturists
Interest
Figure 10. Example of a stakeholder map. This stakeholder map is a generalized version of the
Nutrients SDR map as of June 2020. Placement of each stakeholder was determined among
communication team members and verified by other researchers and managers on the project.
19
-------
Table 1. Categories of stakeholders and responsible parties for engagement as represented on
the stakeholder map.
Who Engages with Stakeholders
Stakeholders
EPA (We) partners with,
engages / consults with
Key partner
Environmental NGO
USGS/EPA regional Office
State environmental agencies/departments
Other ORD agents (mid- to high-level
managers)
EPA (We) engages with,
consults closely with
Bog farmers
Consultancies
University/other researchers
Community organizations
Aquaculturists
EPA researchers
Partners manage / build interest
Other EPA offices
Other federal agencies
Other permitting agencies
Local environmental NGO
Local funders
Town officials
Local health officials
Partners engage / inform
Nearby towns
Town residents
Coastal communities
What We Learned:
When it came to stakeholder mapping, it was important to meet with each
research team to ensure all stakeholders were identified and then appropriately
placed on the stakeholder map. This map has been revisited several times and
will continue to be revisited to account for changes in stakeholder participation
and engagement.
Once stakeholders are placed on the map, we found it useful to present this map
back to project researchers to make sure that stakeholders' placement in the
map adequately reflected and represented their level of involvement from the
perspectives of project researchers.
20
-------
Limitations of Stakeholder Involvement
When identifying and mapping stakeholders, not all stakeholders need to be involved in every
part of the process. While there is a consensus on the need for public involvement in
environmental decision-making and an increasing appreciation of the value of participation for
modeling efforts, there are also limitations associated with stakeholder processes. First, being
engaged in a stakeholder process can be a significant commitment for the participant in terms
of time or financial resources, and many stakeholders may not be willing or able to make this
commitment (Beierle &Cayford, 2002; Huzzard, 2020). Depending on the length of the project,
some stakeholders may lose interest over time. Stakeholder fatigue occurs when stakeholders
feel overwhelmed by the intensity of engagement, which reduces their willingness to
participate and the quality of their engagement (Durham et al., 2014). Depending on prior
engagement and trust established between stakeholders, it can also be difficult to maintain
high levels of stakeholder engagement in research design when some stakeholders are more
interested in participating once a project produces results (Huzzard, 2020). Often, only those
who are most interested in the research project will stay throughout the duration of
stakeholder engagement efforts which can bias the input, misrepresent consensus, and risk the
legitimacy of the process (Korfmacher, 2001). Additionally, for projects with multiple working
components, different working speeds and products coming out of each component can render
stakeholder updates difficult and discourage a high degree of engagement (Huzzard, 2020). To
avoid stakeholder fatigue, we have been strategic in stakeholder engagement, providing
updates and opportunities for participation as the project progresses without inundating
stakeholders with excessive updates.
Many traditional approaches to stakeholder engagement fail because they are perceived as
one-way communication from the organizing agencies to the stakeholder groups (Stave, 2002)
or as a required effort to justify already determined policies (Beierle & Cayford, 2002). To
address this limitation, our key partner leads, co-creates, and distributes their own updates on
research and communication about the Nutrients SDR Pilot to their community. These
communications highlight the voices of local stakeholders and provide a communicator outside
of EPA for the project. Our partners have addressed these concerns through video question and
answer sessions with community members about the research project. A lack of clarity
surrounding roles, responsibilities, and rules of engagement can pose a challenge to meaningful
engagement (Mauser et al., 2013), so establishing these roles in the beginning of a project and
revisiting them periodically throughout the duration of the project will facilitate communication
and engagement processes.
The timing of a research project is also important when it comes to interdisciplinary
engagement. No matter how exciting the research might be, if stakeholders are dealing with
other prevailing priorities, pursuing a high degree of engagement could be challenging to
sustain. Being aware of the social, political, and environmental context where the project is
being conducted, and within which other stakeholders are operating, is important to determine
whether the timing of a project is correct, and ultimately the impact of a project.
21
-------
Engagement processes themselves require skilled facilitators and, depending on the effort, may
also require a number of technical experts trained to explain their work to diverse groups
(Alexander & Robertson, 2004). This is often a barrier to effective stakeholder engagement, as
many scientists are not trained in how to communicate their results to audiences outside their
discipline (Scheufele, 2013). Stakeholders' desire to participate and the quality of their
contributions can also be impacted when communication does not occur in a manner that is
relevant to a stakeholder, for example, by overwhelming non-technical audiences during
technical sessions with overly technical information.
While the sciences of communication provide in-depth research on how to communicate best
with people from various academic and social backgrounds (Dilling & Lemos, 2011; Fischhoff &
Scheufele, 2013), this has not been historically required in Western scientific training (Fischhoff
& Scheufele, 2013; Scheufele, 2013). To ensure more effective communication, we used an
external facilitator for the problem formulation workshop. The external facilitator provided
intentional structure around engagement, served as a neutral moderator, and also helped
mediate the risk of voices not being heard to ensured accurate representation of ideas.
It is not uncommon for stakeholder views and interests to conflict. In this situation, it is
important to first identify a stakeholder's expectations and why they might conflict with those
of other stakeholders. Then, identify the source of conflict (conflict of priorities, personality
conflict) and whether it is rooted in structural/organizational conflict or individual conflict
(Semeniuk, 2010). Listening to stakeholder concerns and effectively communicating with
stakeholders can help build trust which can help overcome conflicting views, interests, and
goals (Semeniuk, 2010). Power and knowledge imbalances between stakeholders can also lead
to poor representation of disempowered stakeholders and their needs, and ultimately to the
exclusion of some stakeholders (Brouwer et al., 2013). While this has not been confronted in
the Nutrients SDR Pilot, the potential for knowledge and power imbalances is particularly
relevant in projects involving historically marginalized communities. Ways to address these
imbalances fall outside the realm of this communication planning document, though strategies
and best practices to address these imbalances are detailed in Brouwer et al.'s analysis of
stakeholder power dynamics in multi-stakeholder processes (2013).
A.4. Implementing Strategic Communication ]
How have researchers communicated with these stakeholders to this point?
What do we want to achieve in communicating with each of these stakeholders?
What are anticipated communication products that each research team will
produce for stakeholders moving forward? What seem to be the most effective
methods to engage with different stakeholders based on past experiences?
The information collected during the stakeholder mapping process is used to develop a
targeted communication plan to meet the communication goals for the Nutrients SDR Pilot
(Figure 4). The stakeholder mapping process allows for the identification of which stakeholders
to target and provides an initial understanding of the appropriate frequency and extent of
communication efforts to be pursued with each type of stakeholder.
22
-------
The communication strategy builds from this initial understanding to tailor a communication
objective for each stakeholder. The Community Tool Box (2020) is a publicly available tool and a
valuable resource in guiding stakeholder-specific communication and is the source for most of
the methods used in the Nutrients SDR Pilot to set communication objectives, frame messaging,
and determine the frequency and mode of messaging (The Community Tool Box, 2020).
To develop a communication strategy, one must consider the following for each stakeholder
group: (1) the communication aim(s), (2) the messaging to be used, and (3) the appropriate
mode/frequency (Figure 11). The aims of communication describe the purpose in
communicating with a certain audience. These will likely differ with each stakeholder, or by
stakeholder type. Providing project-related updates, accessing information and different
perspectives, keeping people informed, recruiting interested parties, and enlisting assistance in
decision-making are some examples of communication objectives. Next, consider the
messaging: what information are you trying to share with stakeholder? What are you trying to
convey through this communication? The messaging will be conveyed through content, mood,
and language. In terms of mood, it is best to avoid being negative and extreme. The formality of
the language used is adjusted based on the audience. It is important to be straightforward,
simple, and clear with messaging. Finally, the modes, or the medium used to communicate, and
the frequency, or how often a mode would be used, are the final step to sketching out a
communication strategy. A diversity of modes can be used to transmit information to a target
audience. Potential modes can include workshops, town hall meetings, word of mouth, print
and news stories, press releases, posters, presentations, and events (The Community Tool Box,
2020). The best mode of communication to use will differ based on the target audience. For
example, using social media or web-related media can target younger demographics, although
older demographics are increasingly using social media (PEW Research Center, 2019). The
frequency of communication will be determined by the interest and influence a stakeholder or
partner has on the project.
23
-------
Figure 11. Steps to Formulate Communication Targeted to Each Stakeholder's Role.
With these considerations in mind, Table 2 is an example of how to define stakeholder groups'
roles and identify communication goals and modes appropriate for each role. Table 2 applies
the stakeholder categorization used in the mapping exercise above.
24
-------
Table 2. Example of Stakeholder Categories and Group-specific Communication Approaches. A starting
point for identifying different types of stakeholders with whom researchers may engage throughout an
SDR project. These groups were used for the Nutrients SDR Pilot, but other SDR projects may demand
adjusted groups, aims, and modes and frequencies to match the project's context.
Stakeholders
Role
Aims of Communication
Mode & Frequency
Key Partner(s)
Primary collaborator
throughout the research
process: development of
research needs, site
prioritization,
implementation, and
broader stakeholder
communication
Coordinate and streamline
communication that is
external to the core set of
partners, facilitate
iterative planning and
implementation
discussions, ensure
research reflects the needs
of stakeholders
Weekly calls,
participation or co-
creation in most other
communication
activities
Project
Partners
Organizations and
individuals that will be
conducting research as
collaborators to ORD
research or that will be
conducting research that is
intentionally complementary
to ORD research for the
nutrients pilot
Reduce duplication of
efforts, improve synergy of
findings and expertise
Periodic meetings
(monthly), workshops,
EPA reports
Key
Stakeholders
Organizations and
individuals that are the main
end users for research
results and who should have
a strong influence on
research prioritization
Develop research to meet
the needs of decision
makers, translate research
findings to those tasked
with making nutrient
management decisions
Calls every 4 months,
reports, annual
presentations
Other
Stakeholders
Organizations and
individuals that are the end
users for research results,
that may be affected by
implementation of research
findings, and who should
have some influence on
research prioritization
Improve transferability of
research findings to and
from the watershed,
coordinate
complementary research,
ensure interested
community members are
aware of our work
Communication and
engagement ranges for
these stakeholders
depending on level of
interest and
engagement.
After defining communication goals for each stakeholder group, the operational tactics for
communication are selected. There are numerous methods for external communication with
stakeholders. Table 3 details some of the different modes of communication that can be used to
engage with stakeholders, such as the use of a social media application like Twitter; the use of webinar;
creation of a website, newsletters or bulletins, infographic; educational outreach, and more. Table 3
also includes a description of the appropriate situation and time to employ each method.
25
-------
Table 3. Methods For Communicating with Diverse Stakeholders.
Communication Mode
Phase in Research Process
Appropriate Uses
Twitter thread
After a stakeholder event,
following initial experimental
results that can be shared with
public
Provide a brief summary of initial results
with a broad audience (more general
than stakeholders). Use hashtags to
engage specific stakeholder groups.
Facebook
At any point in the process
Provide a brief summary of initial results
with a broad audience (more general
than stakeholders).
to
o
n
2L
YouTube
During a field research project
(demonstrating equipment),
after an experiment (if results
are visually/aesthetically
interesting)
Provide an in-depth look at a small
portion of a project, or an overview of an
overall project (more general than
stakeholders).
(D
a.
53'
Instagram
During fieldwork, or with
preliminary or final results
Explain a key scientific fact that is the
basis for the project at hand, the use of
research equipment, or impact of a
research project on human health using
limited text and a singular image to reach
a general audience. Use hashtags to
engage a specific stakeholder group.
Webinars
To describe a research or
communication approach or at
conclusion of an experiment
Give live presentation to people located
near and far - allows for detailed
description of project status along with
opportunity to answer audience
questions. This can also be posted online
for later review at viewers' leisure.
Short Science
Articles
After completion of any step in
the research process (i.e., plan
development, experiment
completion, after a
stakeholder event)
Go slightly in depth to key findings from
an experiment or details of an event, sent
to a broad audience of subscribers, but
also can be sent as a link directly to
specific partners and stakeholders.
Website
After results are finalized
mainly, potentially mid-project
Present central figures or images from
the research findings, share key findings
with a broad audience.
o
3
OTQ
(D
-4-1
o
Bulletin/
newsletter
Throughout and after research
process
Give interested stakeholders an update
on the current status of the research
project, assumes some familiarity with
the project.
3
ฃ
r+
3*
OQ
Internal
Newsletter
Throughout the research
process at critical project
phases, particularly at the end
of an experiment or after an
event
Provide updates on current status of
research with fellow employees, giving
regular updates across newsletters.
Peer-reviewed
journal articles
At conclusion of research
process
Present technical findings to a scientific
audience.
Reports
At the conclusion of the
research process
Present key findings both internally and
with stakeholder audience through long
form writing.
26
-------
Communication Mode
Phase in Research Process
Appropriate Uses
Infographic
(one-pager)
Defining a problem.
After results are finalized
Present key findings to stakeholders that
would want an overview of research
without jargon - can be specific
stakeholder audience (fellow researchers,
elected officials) or a general audience.
<
5)'
c
ฃU_
"O
o
a.
c
Poster
presentations
Defining a problem; after
results are finalized
Present key findings to stakeholders that
would want an overview of research
without scientific or technical jargon - can
be specified to a narrow stakeholder
audience (fellow researchers, elected
officials, etc.) or a general audience.
n
r+
to
Story Map
After selection of research
sites to detail research plans at
each location, or after results
are finalized
Communicate research plans or finding
with local stakeholders to provide spatial
understanding of planned research and
increase contextual understanding
(should provide a brief overview of
project and location) of why work is
being done in selected locations.
I-
o
n
0)
r+
5"
3
dr
0)
to
(D
a.
m
3
OTQ
Q)
OTQ
Educational
outreach
Any stage
Mimic experimental techniques as a
demonstration, present relatable results,
targeted to specific grade level
understanding/standard.
Oral
presentation to
a nontechnical
audience
With preliminary or final
results
Mimic experimental techniques as a
demonstration, present relatable results,
targeted to specific grade level
understanding/standard.
fD
3
(D
3
r+
Conference
presentations
With any fieldwork or publicly
visible research conduct
Share research results with scientific
community.
Other
Field hand-out
(palm cards)
With any fieldwork or publicly
visible research conduct
Share research overview with passersby
when conducting fieldwork.
27
-------
The Nutrients SDR communication team established feedback mechanisms and embedded them within
the communication process to ensure communication goals were met, the team was being
accountable to stakeholders, and evaluate the efficacy of the communication strategy. A feedback
mechanism can be as simple as designating a point of contact and providing their contact information
in communication products or creating an email address where stakeholders can send feedback.
Feedback mechanisms employed by the Nutrients SDR Pilot included a feedback link in the Nutrients
SDR bulletin created specifically for the project, as well as an appointed leader for each research group
to act as a liaison between researchers and stakeholders.
Timelines
We created communication product timelines for individual research groups to help with researcher
accountability to stakeholders and the project and communication goals. Research groups listed their
initial goals for communication products that meet stakeholder needs. To accomplish this for the
Nutrients SDR communication process, members of the communication team met with the leaders of
each research team to define appropriate products and audiences for their work (Table 4). While these
timelines may change over time as communication needs change, they are a good way to keep track of
communication and engagement efforts. These timelines also help define the tactics each team is using
to move towards achieving the overall communication goals of the project. Table 4 details the timeline
for one of the sub-groups for the Nutrients SDR Pilot, with generalized descriptions of communication
efforts. This can be used as a template for organizing future SDR project communication efforts.
28
-------
Table 4. Group Timeline for SDR Project Research Team. The research group's tentative planned public-
facing and scientifically oriented products.
Stakeholder/
Partner
Public-facing activities, and
reports/scientific products
Estimated
Timeframe
Predicted
distribution/
publication date
Lead
Researcher(s) who is ultimately
responsible for the product
Science
Community
Peer-reviewed article (past
work with stakeholders)
2020
EPA researchers, stakeholders
Partners &
Stakeholders
Research and experiment
planning workshop
December 2019
EPA researchers, key partner,
stakeholders
Public
Twitter thread on workshop
February 2020
Sub-team communication lead
Stakeholders
Research stakeholder
workshop (attendance and
presentation)
March 2020
EPA researchers
Homeowners
& Press
Press release and
explanatory placard on
research site
August 2020
Sub-team communication lead
and key partner
Public
Twitter thread on
experiment updates
August 2020
Sub-team communication lead
and EPA social media staff
Homeowners
Video on experiment status
September 2020
Key partner
Community &
Research
Stakeholders
Project Bulletin
August 2020
Sub-team communication lead
Stakeholders
&
Practitioners
Public meeting
outreach/fact sheet
Fall-Winter 2021
Sub-team communication lead,
EPA researchers, stakeholders,
EPA communication staff
Restoration
Practitioners
Infographic/brochure on
public engagement practices
2022
Sub-team communication lead
Stakeholders
&
Practitioners
Interpretive signs for project
sites
2021
Sub-team communication lead
and key partner
Science
Community
Peer-reviewed article(s) on
experiments +social science
2022
EPA researchers, partners,
stakeholders
Will project-scale communication products include a project summary report? Peer reviewed
papers on the project as a whole? A stakeholder workshop on project results and their
application?
While it can be difficult to anticipate at the beginning of a project the project-level final communication
products that might be produced, these products provide a target that smaller communication
products can build towards. Larger-scale products require a significant amount of time and
organization, so anticipating what these products will look like and what will be required to achieve
them early on will lend to a more successful process. This requires guidance from project leads on the
expected outputs of the project, and as an SDR project, defining these products should include input
from key partners and stakeholders. This may evolve throughout the process, but with an output goal
in mind, smaller communication products developed throughout the process can serve as pieces that
can be integrated into the final effort. For the Nutrients SDR Pilot, we anticipate a project report and
several peer-reviewed papers, as well as several presentations on research conclusions to different
stakeholder audiences.
29
-------
B. Building an Evaluation Approach
Evaluating progress towards achieving overall project goals enables lessons learned to continually
improve a communication strategy over a project's lifespan and in future research efforts. The
Nutrients SDR Pilot was designed to identify and evaluate best practices for communicating SDR in
future research efforts. The main purpose of evaluation is to "assess or improve the merit, worth,
value, or effectiveness of a program or project and to advance the field" (CAISE, 2020). Evaluation
identifies limitations that a project may need to overcome as well as opportunities a project can
capitalize upon. The outcomes of evaluation are recommendations of strengths that can be built upon
and areas for improvement. Evaluation can be conducted throughout the course of project
implementation (formative evaluation) and after completion (impact evaluation). For the Nutrients
SDR Pilot, the results of the evaluation are intended for use by ORD in the consideration of future use
of SDR principles, as well as by other agencies and researchers who are considering implementing or
participating in SDR-based research. The evaluation process used in this report began with the
establishment of communication goals (as shown in Figure 2).
Preparing for Evaluation
Who has enough experience in evaluation and distance from the project team to objectively
assess progress towards communication and project goals and outcomes? What approach will
be used to assess progress towards and accomplishment of project outcomes and goals?
The first step in planning our evaluation was to identify who will conduct the evaluation. The evaluator
provides a neutral perspective on the achievement of project and communications goals, has
experience in either program evaluation or communication evaluation and should not be directly
involved in the research. For the Nutrients SDR Pilot, an evaluator external to the project outlined
plans for evaluating the communication goals using logic models.
30
-------
There are a range of methods that can be used for formative and impact evaluations. This report
presents the logic model of evaluation that was developed to provide insights to organizers about how
the intended evaluation components relate to one another and how they connect with evaluation
metrics and methods. At the time of this report's publication, the Nutrients SDR Pilot is in its middle
stages of implementation. The logic model provides the anticipated structure of evaluation; however,
it has not yet been applied.
Logic Models in Evaluation
Logic models are a depiction of what a program is designed to do and what it is to accomplish. The
models describe critical linkages among what is done and who is engaged with short-, medium- and
long-term impacts (refer to Figure 12). They consist of a series of "if/then" statements that, "if"
implemented as intended, "then" lead to the desired results and are often the core of program
evaluations. Importantly, a logic model:
Shows not only what is done but how it impacts important groups of people
Shows how researchers plan for short-term work to influence the achievement of
medium-term and ultimate goals far into the future
Can be used as a method for creating or refining goals
Logic models serve as a roadmap that provide the rationale for a sequence of steps (Figure 13). When
used in evaluation, logic models help connect resources (financial, time, and expertise) to anticipated
impacts. The anticipated impacts are the benefits and outcomes that result from the activities within
the overall project in the short, medium, and long term (ultimate impacts). The short- and medium-
term impacts include learning activities such as increased knowledge, changes in attitudes, or higher
motivation to act differently, as well as actions such as changes in behavior or decisions. The ultimate
impacts of the model include changes in basic conditions of the environment or related communities
(Taylor-Powell & Henert, 2008). It is important to define timeframes (e.g., six months, two years, five
years) for short-term, medium-term, and ultimate impacts. For the Nutrients SDR Pilot, the short-term
impacts are those occurring by the end of the pilot in FY2022, medium-term impacts are those that
occur between the end of the project and 2030, and ultimate impacts occur beyond 2030.
31
-------
Simplest Form of a Logic Model
Activities to
take place
Impacts: WHO benefits + HOW
they benefit
A bit more detail...
What we do
Who we reach
Short-term impacts
Med-term
impacts
And a bit more...
Short- and medium-term
impacts are:
Ultimate
impacts are:
LEARNING
& ACTION
CONDITIONS
Knowledge
Behavior
Social
Attitudes
Decision-making
Economic
Motivations
Policies
Civic
Environmental
Figure 12. Description of logic models arid their application to connecting what we do and who we
reach with intended short-term, medium-term, and ultimate outcomes. These logic models are
based on the work of the University of Wisconsin's Cooperative Extension Program.
Nutrients SDR Logic Models for Evaluation
The Nutrients SDR Pilot research team spent several months creating a logic model that identified four
short-term impact statements to represent the goals of the pilot's communication and engagement
activities over three years (Figure 13). This logic model was developed and coordinated by an
evaluation team and was completed by researchers and managers on the Nutrients SDR Pilot in a one-
day workshop (refer to Figure 16 in Appendix C for the worksheet used). The goals included impacts
researchers and managers hoped to achieve both externally (partner and stakeholder opinions and
learning) and internally (EPA staff and manager opinions and learning). A logic model ensures that
these desired short-term, or project-length impacts are in line with (1) the desired ultimate impacts
that will occur in future project evolutions, and (2) the activities occurring in the day-to-day progress of
a project, and (3) the various audiences involved in and reached by those activities. This allows for
coordinated big picture thinking with steps to make the big picture goals a reality (Kellogg Foundation,
2004). Logic models allow for coordinated thinking using desired impacts to define activities and
audiences, an approach often referred to as backwards design.
32
-------
What we do
Who we reach
Short-term impacts
Medium-term impacts
Ultimate
impacts
Figure 13. Compendious Logic Model for the Nutrients SDR Pilot. Boxes with a "*" indicate where the Nutrients
SDR Pilot communication team mapped applicable metrics. The logic model created for the Pilot includes all
three thematic areas for completeness. Additional logic model content from the Nutrients SDR Pilot can be found
in Table 6 of Appendix C.
B.1. Identify Potential Metrics
How can we measure our progress towards achieving these goals? What existing mechanisms
do we have that might track progress? How might we modify existing mechanisms to better
track progress, and what are we missing?
When establishing goals, it is important to identify potential quantitative or qualitative metrics to track
progress and evaluate whether goals have been met. This applies both to the overall project goals and
to project-specific communication goals. The metrics can take many forms and must be clear and
measurable to capture what the communication goal is trying to achieve. Deciding how many metrics
are needed to capture a goal and what degree of performance counts as success are subjective
processes and should be agreed upon when establishing monitoring and evaluation metrics. For
example, if there is a 20 percent increase in the number stakeholders attending project workshops
over a certain period, is this enough to meet the goal of increased engagement? Or would a 50 percent
increase be more appropriate? The necessary quantity of a metric to be considered successful will be
based on the context within which an SDR project is implemented. It is also important to establish a
baseline point in time and/or condition against which to evaluate the communication strategy. For the
Nutrients SDR Communication Plan, the communication team elected to use the Problem Formulation
Workshop in 2018 (Twichell et al. 2019) as a baseline from which to measure and evaluate engagement
and communication efforts.
33
-------
This report focuses on metrics relevant to the communication goals established in section A.2. focused
on communicating the Nutrient SDR Pilot's research (Figure 14). As shown, a metric can be associated
with more than one goal. While quantitative metrics may traditionally come to mind (e.g., the number
of visits to a webpage or number of people attending an event), qualitative metrics are both useful and
valid to measure things that cannot be measured quantitatively (e.g., participants expressing they have
achieved a higher level of understanding after participating in a project-related event). Metrics can also
include a checklist of yes/no questions for tracking less complex progress towards a goal. Figure 14
details metrics being tracked for two goals in the Nutrients SDR Pilot Project.
Goal 1: Encourage
and sustain partner
and stakeholder
engagement in a
targeted manner
based on each
stakeholder's role
in the Nutrients
SDR pilot
Metrics:
Number of formal quarterly meetings to discuss aspects related to the
evaluation or build relationships with stakeholders, including community
members
Number of informal (often one-to-one or impromptu) meetings to discuss
aspects related to the evaluation or build relationships with stakeholders,
including community members
Changes (positive or negative) in attitudes or behaviors of key stakeholders
regarding the impact of communications and engagement
Goal 2: Produce
and deliver
accessible products
tailored to specific
stakeholder needs
to make EPA
science more
usable on the Cape
and transferrable to
other areas
Metrics:
Was ORD invited to participate in conferences and meetings by stakeholders
on nutrient inputs and/or alternative technologies? How many?
Number of events at which information related to study is presented
Number of knowledge and information products, including name and brief
description e.g., story boards, illustrations, briefs of any type, blogs or papers
produced as part of the intervention or for documenting impact evaluation
analysis and findings
Change in institutional culture of evidence use or commitment to
communication and evaluation
Figure 14. Metrics Associated with Two Communication Goals.
For the Nutrients SDR Communication Plan, most of the metrics are based on the Bie metrics of
stakeholder engagement (International Initiative for Impact Evaluation, 2021; the link to the original
web page has since been broken), and similar information can be found by referencing the following
sources: Sandhu-Rojo, 2018; Larson & Williams, 2009; Initiative for Impact Evaluation, 2012;
MacQueen et al., 2021. The metrics were selected with a balance in mind between accurately
measuring progress towards goals and ease of collection. For SDR research projects, the data for the
identified metrics may come from numerous sources and need new tracking mechanisms. There are
various types of tracking mechanisms that can be used, but careful design is needed to ensure they
accurately capture what project members did to meet goals. While some examples of quantitative,
qualitative, and checklist-type metrics are listed above, the comprehensive list of metrics used in the
Nutrients SDR Pilot is in Appendix A.
34
-------
The metrics outlined in the Nutrients SDR Pilot Communication Plan were mapped to the boxes within
the logic model, to show the Nutrients SDR Pilot team was able to show both what they did (Table 6 in
Appendix C) and the short-term impact of their work (Table 7 in Appendix C). The data sources used by
the communication team to track metrics included a monthly report collectively generated by
individuals involved in the pilot using a semi-structured data entry form. Other sources considered by
the team included tracking web analytics, surveys, focus groups, and interviews. The communication
team planned to use those metrics and methods to evaluate the identified short-term impacts, as their
tracking will occur throughout the duration of the project. While it is important to identify and analyze
medium- and long-term impacts, this can be difficult to do when it is beyond the funding timeline of a
project. Depending on resources available for evaluation, the authors propose reporting on metrics at
least once during the span of a project (a formative evaluation) and at the culmination of a project or
end of a project research planning cycle (an impact evaluation). Reporting on metrics demands that
they are continually tracked as described in the next section.
B.2. Develop Evaluation Methods & Tracking Tools
How can we document our progress towards the chosen metrics?
Who is in charge of collecting and tracking this data?
For ORD, one way to track metrics is through an internal review process database of ORD research
publications and presentations. If an agency has a database for internal review of publications, that can
also be used to aggregate metrics. Internal interviews with federal project researchers and key
stakeholders can also be used to track changes in researcher sentiments towards stakeholder
engagement and science communication. For the Nutrients SDR Pilot, examples of mechanisms used to
aggregate data for metrics include brief monthly summaries of all communication and outreach
activities conducted, and a content review of documents produced as part of the project. These
mechanisms and the metrics they track are in Appendix E. The main data aggregating mechanism that
demands researcher effort is filling out a 'monthly report' form that tracks metrics and engagements
(Table 5). This serves the purpose of keeping stakeholders up to date on project happenings while
tracking engagements, events, products, and anticipated products. Additionally, a communication
team member and internal project researcher are conducting interviews with researchers internal to
the project at the beginning, middle, and end of the project to understand the impact of increased
stakeholder engagement on research efforts (refer to Appendix D for questionnaire). This research has
been approved through internal quality assurance and an Institutional Review Board. Parallel
interviews were also conducted with stakeholders that are a part of this project to provide insights
about engagement and communication. These interviews serve to document changes in perspective on
the value and usefulness of consistent stakeholder engagement and provide a more complete picture
of researcher effort towards and opinions about directly orienting work towards stakeholder needs.
The questions asked of stakeholders and researchers are based on the literature of public engagement
with scientific research, and on what we anticipated as challenges of stakeholder engagement for the
team. These included questions about the process of communicating with stakeholders and clarifying
roles and goals in this work. The communication team also has considered future use of web analytics
from social media and website posts to provide data for metrics.
35
-------
Table 5. Monthly report semi-structured data entry form capturing stakeholder engagement
information from pilot team members.
Date
(all
2020)
Research
category
(e.g.,
bogs)
Nature of
engagement
EPA
staff
present
External
stakeholder
present
Activity
Was the
event
submitted
or invited
to?
Total #
of
people
present
Notable
outcomes
Weekly
Project-
wide
Conf calls
Tim,
Walter,
Marty
Key partner
Joint
updates
Co-
planned
4
Increased
cooperation
March
4
Septics
Conf call
Alexie
Key partner
Planning
stakeholder
Q&A
material
invited
2
Initial draft
of Q&A for
experts to
answer
complete
March
11
Septics
Skype
meeting
Tim,
Alexie
Conservation
NGO
Conference
planning
invited
5
Finalizing
sessions for
conference
March
12
Septics
Workshop
Katie,
Kaytee,
Cathy
Conservation
NGO,
university
researchers,
state
agencies
Attendance
&
presentation
invited
30
Relationship
building,
shared
results
The complete list of identified metrics for the Nutrients SDR Pilot is provided in Appendix A. Some
metrics involve continual, mostly quantitative data collection (e.g., number of events at which
information related to the pilot is presented) and can be periodically updated throughout the course of
the pilot. Other metrics involve qualitative data collection (e.g., conversations with stakeholders on
their understanding of nutrients science after a researcher presentation) and interpretation at defined
points in time.
36
-------
Identifying & Implementing Lessons Learned
Identifying lessons learned throughout the process of planning and implementing communication
efforts for SDR projects will help refine recommended practices and identify areas for improvement in
future projects. Keeping track of communication efforts and collecting the perspectives of stakeholders
and researchers both throughout the process and at the conclusion of the research project can provide
useful data from which to identify lessons. For the Nutrients SDR Pilot, our various evaluation tracking
mechanisms and our social science study of engagement process will provide much of that data. Along
with internal efforts to identify lessons on how to best conduct SDR communication, the input of an
external evaluator can provide objective insight on avenues for future improvement. While we are
currently at the midpoint of this project, the first round of interviews with researchers and
stakeholders and the observations of internal reviewers have identified a few in-process lessons. These
lessons will be incorporated into efforts for the duration of the project where possible, putting into
practice the iterative approach we have emphasized throughout the report.
Research and communication goals should be drafted in collaboration with stakeholders. While
problem identification did occur in close collaboration with stakeholders, and key partners are aware
of the research goals, a more inclusive goal development process would have presented the drafted
goals to the stakeholders for input prior to establishing the official goals.
37
-------
Trained facilitators are valuable for large collaborative meetings. To this point in the project, we have
had both professionally and internally facilitated meetings and workshops. Interviews led to the
recommendation to hire professional facilitators for large collaborative meetings and workshops to
ensure all stakeholder voices are heard and meeting goals can be met.
Recognize researchers' past efforts communicating with stakeholders, and the existing relationships
they have built with stakeholders. While this is a pilot of using a fully solutions-driven approach, many
researchers do have past experiences in stakeholder engagement that can helpfully inform solutions-
driven communication. Additionally, researchers' existing relationships from past projects have been
key to stakeholder engagement and research in this pilot, and the value of these relationships should
be recognized.
Being transparent about how stakeholders are engaged can be difficult. This lesson specifically refers
to stakeholder mapping. While stakeholder mapping is a useful tool to begin thinking about who to
involve in a project and how, stakeholders' placement on a map according to interest and influence in
a project can be damaging if these maps are shared and taken out of context. Using different methods
to map engagement priorities should be pursued in the future to allow for greater transparency.
Stakeholder relationships do not end when a research project is officially over. A lesson already being
implemented in this project is that stakeholder relationships do not end when a research project is
officially over, and it is likely in the interests of everyone involved to maintain said relationships for
future needs and work.
Being aware of the political and economic context in communities with which you are working,
including other projects, plans, and priorities that will affect these communities, is critical for a
project's success. Where project goals may appear to distract from or contradict community plans and
projects, a project may be met with skepticism or hostility, and may be overlooked. It is important to
work with local stakeholders and officials to understand local context, how a proposed project might
interfere with these plans, and to work on ways these might be compatible. When conducting applied
research, such as an SDR approach, as opposed to theoretical research, you are taking on political and
social context. It is equally important to be aware of cultural contexts within a community, ensuring
that marginalized groups are engaged in the project.
At the end of the project, lessons learned can be identified through the methods and metrics that were
identified in the evaluation planning process. These lessons relate to whether and why the
communication goals were achieved along with identifying shortcomings and successes related to
achieving communication goals. We recommend considering the following when evaluating whether
communication goals were achieved:
Where did the initiative succeed? Why might that have been?
Where did the initiative fall short, and by how much?
What communication, engagement, or information-sharing steps could be incorporated into
the communication plan to better target the areas where the initiative fell short?
How are research outputs being used, and who is the biggest user?
38
-------
The methods and metrics outlined in the evaluation process in this report allow for an evaluation
report to formally assess performance by analyzing the metric data collected in the tracking tools. This
report should include specifying areas for future improvement in the team's approach to SDR and
additional lessons learned. Further, it should document how well goals and outcomes were achieved,
and any notable successes or choices that are captured in the tracking tools and data that aided in
those successes.
Conclusion
Communication planning is an important step in conducting an SDR project and should be conducted
early and revisited often. This report serves as a process guide for the creation of an SDR
communication plan based on the Nutrients SDR Pilot communication plan process. It presents this
process step by step, from goal development, through stakeholder identification, strategic
communication design, through to evaluation. As the communication insights in this report stem from
a pilot project, it is expected that recommended practices for EPA communication planning in SDR will
evolve as researchers move through research implementation and evaluation and learn from these
initial efforts. Additionally, recommended practices will be dependent on the nature of specific
projects based on what is appropriate. Communication planning in SDR requires flexibility to meet
stakeholder needs in a manner that is context-specific: there is no 'magic bullet' approach or single
best way to communicate SDR. The revision of SDR communication plans is important as SDR demands
iteratively modifying plans to match the evolving list of stakeholders and their needs. While the specific
EPA context of the Nutrients SDR Pilot has informed the chosen goals and communication methods,
there are transferable takeaways to communication planning. Building and maintaining relationships
with stakeholders over time, continued two-way communication, frequently revisiting the community
of stakeholders, and presenting targeted updates throughout a research project are good practices to
adopt in developing future communication plans. Moving forward, future researchers can add to the
stakeholder-centric approaches described here to further build a research program that cooperatively
meets the needs of stakeholders while protecting national land, air, and water resources.
39
-------
References
Alexander I., & Robertson, S. (2004). Understanding project sociology by modeling stakeholders. IEEE Software,
21(1): 23-27. https://doi.org/10.1109/MS.2004.1259199
Aus, R. (2017). How to set goals for your project communication plan. Workamajig. Retrieved October 15, 2021,
from https://www.workamaiig.com/blog/proiect-communication-plan-set-goals
Baghdikian, C. (2018). Solutions-Driven Research: Applying Translational Science Approaches to Research to
Address Real-World Problems. SETAC North America 39th Annual Meeting, Sacramento, CA. November 4-8,
2018.
Ballejos, L.C., & Montagna, J.M. (2008). Method for stakeholder identification in interorganizational
environments. Requirements Engineering, 13: 281-297. https://doi.org/10.1007/s00766-008-0Q69-l
Beierle, T.C., & Cayford, J. (2002). Democracy in Practice: Public Participation in Environmental Decisions.
Routledge.
Bevan, B., Barton, A.C., & Garibay, C. (2018). Broadening Perspectives on Broadening Participation in STEM.
lnformalscience.org. Retrieved October 15, 2021, from https://www.informalscience.org/sites/default/files/BP-
Report.pdf
Brouwer, H., W. Hiemstra, S. der Vugt, &d H. Walters. (2013). Analyzing stakeholder power dynamics in multi-
stakeholder processes: insights of practice from Africa and Asia. Knowledge Management for Development
Journal, 9(3): 11-31.
https://www.researchgate.net/publication/283417668 Analysing stakeholder power dynamics in multi-
stakeholder processes insights of practice from Africa and Asia
CAISE. (2020). "What is Evaluation?", lnformalscience.org. Retrieved October 18, 2021, from
https://www.informalscience.org/what-evaluation-0
Chintala, M., Mazzotta, M., Mulvaney, K.K., Merrill, N., Martin, D., Hill, T., Ayvazian, S., Lake, J., Wigand, C., and
Berry, W. (2018). Research for Nutrients Solutions on Cape Cod Series. U.S. EPA Office of Research and
Development, Washington, DC, EPA/600/F-18/051, 2018.
Commonwealth of Massachusetts. (2021). Environmental justice populations in Massachusetts. Retrieved
October 18, 2021, from https://mass-
eoeea.maps.arcgis.com/apps/MapSeries/index.html?appid=535e4419dc0545be980545a0eeaf9b53
Conservation Law Foundation v. Environmental Protection Agency, l:13-cv-12704-MLW. (D. Mass. 2013).
Dilling, L., &Lemos, M.C. (2011). Creating usable science: Opportunities and constraints for climate knowledge
use and their implications for science policy. Global Environmental Change, 21(2): 680-689.
https://doi.Org/10.1016/i.gloenvcha.2010.ll.006
Doran, G.T. (1981). There's a S.M.A.R.T. way to write management's goals and objectives. Management Review,
70(11): 35-36. https://communitv.mis.temple.edU/mis0855002fall2015/files/2015/10/S.M.A.R.T-Wav-
Management-Review.pdf
40
-------
Durham E., Baker H., Smith M., Moore E., & Morgan V. (2014). BiodiVErsa Stakeholder Engagement Handbook.
Biodiversa. Retrieved October 18, 2021 from https://www.biodiversa.org/706/download
Druschke, C.G., & Hychka, K.C. (2015). Manager perspectives on communication and public engagement in
ecological restoration project success. Ecology and Society, 20(1): 58. http://dx.doi.org/10.5751/ES-07451-
200158
EPA (U.S. Environmental Protection Agency). (2020). Exploring Solutions to Nutrient Pollution: Restoring Cape
Cod's Waters. Epa.gov. Retrieved October 18, 2021, from https://www.epa.gov/sites/default/files/2020-
08/documents/exploring-solutions-nutrient-pollution.pdf
EPA (U.S. Environmental Protection Agency). (2021J. EJSCREEN: Environmental Justice Screening and Mapping
Tool. Epa.gov. Retrieved 18 October, 2021, from https://www.epa.gov/eiscreen
Fair-Wright, C. & Juli, T. (2016). Overcoming stakeholder resistance through dialogue. Paper presented at PMI
Global Congress 2016EMEA, Barcelona, Spain. Newtown Square, PA: Project Management Institute. Retrieved
October 18, 2021, from https://www.pmi.org/learning/librarv/overcoming-stakeholder-resistance-10696
Fischhoff, B., & Scheufele, D.A. (2013). The science of science communication. Proceedings of the National
Academy of Sciences, 110(Supplement 3), 14031-14032. https://doi.org/10.1073/pnas.131208011Q
Huzzard, T. (2020). Achieving impact: Exploring the challenge of stakeholder engagement. European Journal of
Work and Organizational Psychology, 30(3): 379-389. https://doi.org/10.1080/1359432X.202Q.1761875
International Initiative for Impact Evaluation. (2021). Impact Evaluation. 3ieimpact.org. Retrieved 18 October,
2021 from https://www.3ieimpact.org/What-we-offer/impact-evaluation
International Initiative for Impact Evaluation. (2012). What is a stakeholder engagement and evidence uptake
plan. Doczz.net. Retrieved 21 October, 2021, from https://doczz.net/doc/8150865/what-is-a-stakeholder-
engagement-and-evidence-uptake-plan
Kenny, G. (2014). Five Questions to Identify Key Stakeholders. Harvard.edu. Retrieved October 18, 2021, from
https://www.hbsp.harvard.edu/product/H00PH9-PDF-ENG
Kellogg Foundation. (2004). Logic Model Development Guide. W.K. Kellogg Foundation. Retrieved October 18,
2021, from https://www.wkkf.org/resource-directorv/resources/2004/01/logic-model-development-guide
Kolaczyk, E.D., and Csardi, G. (2014). Statistical Analysis of Network Data in R. Springer: NY.
Korfmacher, K.S. (2001). The Politics of Participation in Watershed Modeling. Environmental Management, 27.
https://doi.org/10.1007/s00267001Q141
Larson, S., & Williams, L. (2009). Monitoring the success of stakeholder engagement: Literature review. In
Measham TG, Brake L (Eds.) People, communities and economies of the Lake Eyre Basin: DKCRC Research Report
45, Desert (pp. 251-298).Knowledge Cooperative Research Centre. Retrieved October 21, 2021, from
http://www.nintione.com.au/resource/DKCRC-Report-45-People-communities-and-economies-of-the-Lake-
Evre-Basin.pdf
41
-------
MacQueen, K.M., Harlan, S.V., Slevin, K.W., Hannah, S., Bass, E., & Moffett, J. (2021). Stakeholder engagement
toolkit: For HIV prevention trials. Fhi360.org. Retrieved October 21, 2021, from
https://www.fhi360.org/sites/default/files/media/documents/Stakeholder%20EngagementToolkit%20for%20HI
V%20Prevention%20Trials.pdf
Maryland Department of Information Technology (DolT). (2021). Stakeholder Management
Plan. Doit.maryland.gov. Retrieved 19 October, 2021, from
https://doit.marvland.gov/SDLC/.../Stakeholder%20Management%20Plan.doc
Mauser, W., Klepper G., Rice, M., Schmalzbauer, B.S., Hackmann, H., and Leemans, R. (2013). Transdisciplinary
global change research: the co-creation of knowledge for sustainability. Curr Opin Environ Sust. 5:420-31.
https://doi.Org/10.1016/i.cosust.2013.07.001
Measure Evaluation. (2011). Stakeholder Engagement Tool. Retrieved October 18, 2021, from
https://www.measureevaluation.org/publications/ms-ll-46-e.html
National Institutes of Health. (2020). Translational Science Spectrum. National Center for Advancing
Translational Sciences. Retrieved October 18, 2021, from https://ncats.nih.gov/translation/spectrum
Nelson, J. & Brooks, L. (2015). Racial Equity Toolkit: An opportunity to operationalize equity. The Local &
Regional Alliance on Race and Equity. Retrieved October 18, 2021, from https://racialequitvalliance.org/wp-
content/uploads/2015/10/GARE-Racial Equity Toolkit.pdf
Pew Research Center. (2021). Social Media Fact Sheet. Pewresearch.org. Retrieved October 18, 2021, from
https://www.pewresearch.org/internet/fact-sheet/social-media/
Sandhu-Rojon, R. (2018). Selecting Indicators for Impact Evaluation. Communityindicators.net. Retrieved
October 21, 2021, from https://communitvindicators.net/wp-content/uploads/2018/01/Selecting-lndicators-
for-lmpact-Evaluation.pdf
Semeniuk, M. (2010). Running with scissors: techniques for managing conflicting expectations. Paper presented
at PMIฎ Global Congress 2010Asia Pacific, Melbourne, Victoria, Australia. Newtown Square, PA: Project
Management Institute. Retrieved October 18, 2021, from https://www.pmi.org/learning/librarv/managing-
conflicting-expectations-6893
Sun, K. (2017). A goal-setting system I learned from Google (and still use at my current venture). Forbes.
Retrieved October 18, 2021, from https://www.forbes.com/sites/karlsun/2017/Q5/16/the-idea-i-stole-from-
google-or-how-i-learned-to-love-okrs/?sh=5a47cbfd6a2f
Sydney Public Relations Agency. (2019). How to set goals and objectives. CP communications. Retrieved October
18, 2021 from https://publicrelationssvdnev.com.au/how-to-set-goals-and-obiectives/
Scheufele, D.A. (2013). Communicating science in social settings. Proceedings of the National Academy of
Sciences, 110(Supplement 3) 14040-14047. https://doi.org/10.1073/pnas.121327511Q
Stave, K. (2002). Using system dynamics to improve public participation in environmental decisions. System
Dynamics Review. 18(2): 139-167. https://doi.org/10.10Q2/sdr.237
42
-------
Storksdieck, M., Bevan, B., Risien, J., Nilson,R., & Willis. K. (2018). Charting the Intersection of Informal STEM
Education and Science Communication: Results of a Social Network Study. Washington, DC: Center for
Advancement of Informal Science Education. Retrieved October 18, 2021, from
https://www.informalscience.org/charting-intersection-informal-stem-education-and-science-communication-
results-social-network-studv
The Community Tool Box. (2021). Section 1: Developing a Plan for Communication. The University of Kansas.
Retrieved 18 October, 2021, from https://ctb.ku.edu/en/table-of-contents/participation/promoting-
interest/communication-plan/tools
Twichell, J.H., Mulvaney, K.K., Hulet, C., Secunda, J., Leiby, A., Buchholtz, and ten Brink, M. (2018a). "Getting
Community Buy-in for Stormwater Funding, A Four-Session Participatory Workshop: Facilitator Manual." U.S.
Environmental Protection Agency, Office of Research and Development, National Health and Environmental
Effects Research Laboratory, Atlantic Ecology Division, Narragansett, Rl. EPA/600/R-18/214.
Twichell, J.H., Mulvaney, K.K., Hulet, C., Secunda, J., Leiby, A., Buchholtz, and ten Brink, M. (2018b). "Getting
Community Buy-in for Stormwater Funding, A Four-Session Participatory Workshop: Participant Workbook." U.S.
Environmental Protection Agency, Office of Research and Development, National Health and Environmental
Effects Research Laboratory, Atlantic Ecology Division, Narragansett, Rl. EPA/600/R-18/214.
Twichell, J.H., Mulvaney, K.K., Hubbell, B., Erban, L., Berry, W., Chintala, M., Crocker, Z., Gleason, T., Horsley, S.,
Munns, W.R. Jr., Rae, A., Reyes, S.S., and Smith, S.N. (2019). "Nutrients Translational Science Pilot Problem
Formulation Workshop Report and Evaluation." U.S. Environmental Protection Agency, Office of Research and
Development, National Health and Environmental Effects Research Laboratory, Atlantic Coastal Environmental
Sciences Division, Narragansett, Rl. EPA/600/R-19/107.
Taylor-Powell, E., & Henert, E. (2008). Developing a logic model: Teaching and Training Guide. University of
Wisconsin Cooperative Extension. Retrieved October 18, 2021, from
https://www.peerta.acf.hhs.gov/sites/default/files/public/uploaded files/Logic%20Model%20Guide.pdf
University of Arkansas for Medical Sciences (UAMS) Translational Research Institute. (2021). What is
translational research?. UAMS.edu. Retrieved October 18, 2021, from https://tri.uams.edu/about-tri/what-is-
translational-research/.
Vangen S., & Huxham, C. (2003). Nurturing Collaborative Relations: Building Trust in Interorganizational
Collaboration. Journal of Applied Behavioral Science, 39(1): 5-31.
https://doi.org/10.1177/0021886303039001Q01
Washington State Bar Association, (n.d.). Goal Setting: professional development. Wsba.org. Retrieved October
18, 2021, from https://www.wsba.org/docs/default-source/resources-services/mentorship/mentorship-
curriculum/02-goal-setting-v4.pdf?sfvrsn=9ac839fl 11
43
-------
Appendix A: Metrics to Monitor and Evaluate
Communication Plan Goals for SDR
This list provides additional metrics employed by the Nutrients SDR Pilot team to track their
communication efforts.
Were engagement events effective?
How many events were hosted by the project team, and how many people attended?
Was ORD invited to participate in conferences and meetings by stakeholders on nutrient inputs
and/or alternative technologies? How many? Which types of meetings was ORD invited to
participate in?
Number of stakeholders that received engagement products
Number of stakeholders that interacted with engagement products
How many people have used feedback mechanisms established by the SDR project team?
How much and what type of feedback was received from policymakers, practitioners, and
beneficiaries of the project?
Number of formal websites on which SDR products/results are posted
Number of social media mentions/accounts related to the SDR project
Observable/measurable change in stakeholder perspective regarding the value of community-
wide research efforts and engagement through research process
Change in institutional culture regarding evidence use or commitment to evaluation (a change
in institutional culture can be gauged by periodic interviews with internal researchers, for
example)
Number and type of members of the public that reached out with questions
Compare survey data from pre- and post-SDR workshops and events to evaluate efficacy of
engagement. This is suggested for major events like a Problem Formulation Workshop.
Did the SDR team consider and incorporate, new ideas and solutions brought up during the
problem formulation meeting? List/take note of them.
Did the project team hear from a diversity of different perspectives?
Stakeholders from different quadrants in the stakeholder map voiced their ideas and concerns
Estimate of number of people representing common and distinct interests
Did the project team hear from stakeholders that have not traditionally been involved?
Did all stakeholder groups feel comfortable sharing their ideas? (Was the atmosphere
conducive to stakeholders whose viewpoints and input has traditionally been marginalized?)
44
-------
Were engagement/communication strategies carried out according to plan?
Number of formal and informal meetings to discuss aspects related to SDR planning and
evaluation
Number of knowledge and information products produced as part of intervention (one-pagers,
signs, pamphlets, listserv emails)
Different types of knowledge and information products produced as part of intervention
Number of formal and informal meetings to discuss aspects related to stakeholder
engagement, or for building relationships
Number of participants
Number of each stakeholder type that attended (federal agents, state actors, etc....)
Did communication strategies reach intended audiences?
Change in readership of SDR communication products (i.e., increase in readership for a
project newsletter)
Number of media mentions where study was mentioned
Communication, Engagement, and Outreach Plan utility and applicability:
Have other agencies, organizations, or actors been able to apply this communication plan
in a useful manner? Cite which ones.
Are non-expert stakeholders able to follow and apply this plan without much effort?
Look at post-workshop and event survey data for SDR events to evaluate whether people
thought the engagement was valuable
General Web-based Metrics:
Website traffic: Percent increase or decrease?
Number of people using a project website at peak times or over a certain amount of time
Number of likes or comments on an online article or social media post
Increase/decrease of number of subscribers to a listserv or newsletter
45
-------
Appendix B: Social Network Analysis: An
Additional Communication Tool for SDR
Social network analysis is a method to quantify relationships between stakeholders which relies on
statistical analysis to identify the most integral members of a community for communication. It does
this primarily by identifying two types of critical stakeholders: 1) those who are important because of
their high number of collaborators or contacts, and 2) those stakeholders who connect with other
potential stakeholders that would otherwise not have social connections to the stakeholder process
(Kolaczyk and Csardi, 2014). Social network analysis also provides a way to visually depict stakeholders
and their connections. Identifying the central stakeholders is important for building an effective
communication strategy, as stakeholders that are more integral to the research will need to be more
meaningfully involved and consistently updated relative to other stakeholders (Storksdieck et al.,
2018). While this method can also help populate the list of relevant stakeholders to a project, an initial
list of informed or key stakeholders is needed to start the analysis. Defining connections among
stakeholders in this approach often involves an interview or survey that asks stakeholders with whom
they collaborate on the topic of interest or to identify other stakeholders who may have a stake in the
project (Kolaczyk and Csardi, 2014). Researchers define what constitutes a "connection" between
stakeholders. It could be as simple as two stakeholders being aware of each other or demand more
significant relationship such as having an existing or past work collaboration.
One result of this analysis is a "map" or "graph." This is a visual representation that denotes all the
identified stakeholders as a designated shape and draws lines among stakeholders that are
"connected" as it is defined by the researcher (Figure 15). This is used to ensure a comprehensive
understanding of stakeholders and their relationships to the researcher(s) and to each other.
Stakeholders that are connected with lines to many other stakeholders are more "central" to the
network and should be more heavily involved in and updated on the research relative to less central
stakeholders.
There are several programs that can be used to conduct a social network analysis. Two of the most
popular are UCINET, gephi, and the igraph package in the R programming language. Depending on the
software, the process of conducting this analysis differs. Even within the R software, there are multiple
approaches to preparing these graphs. The necessary information for any such analysis is a "node list"
of all of the nodes, or stakeholders in the network, and an "edge list" of all the connections between
individual nodes.
Conducting social network analysis once research is underway can also provide insight into whether
the stakeholders that are most connected to other stakeholders in the research, as defined in this
analysis, are being sufficiently engaged in the research process. Social network analyses can also be
conducted at the conclusion of a research project as an evaluation. One such use would be comparing
who community stakeholders perceive as central stakeholders in the project to the network of
individuals with whom researchers interacted. Such an evaluation can help inform future projects,
building researcher understanding of key stakeholders in specific research and topic areas.
46
-------
Mayor's pffice
Figure 15. Example of a Social Network Analysis Map. This is an example of the visual depiction of a
fabricated stakeholder network. Each individual node denotes an individual stakeholder, and each line
defines a connection between stakeholders. This also displays how attributes of stakeholders can be
visually explained as the size of the nodes is used to denote different levels of engagement, and
placement denotes organizational similarity (for example, town government representatives are
grouped and industry representatives are grouped).
47
-------
Appendix C: Additional Logic Model Information
What we do
1
Who we reach
Short-term impacts
What impacts are feasible by
2022?
Think about what you want to do and who might benefit from your work in the next 5 years,
then convert it to an impact statement...
e.g., hold logic
model sessions,
Evaluate and
produce report
5
e.g., ORD researchers
~
e.g., Other ORD researchers
use what we have learned
from this process to
implement their own
translational science
planning processes.
Impact statement: WHO
benefits & HOW they benefit
Consider Knowledge,
Attitudes, Motivations,
Decision-making, Policies
Med-term
impacts
2022-2030
Ultimate
impacts
2030+++
Conditions
Social
Economic
Civic
Environ-
mental
Figure 16. Worksheet for Nutrients SDR Logic Model Development,
48
-------
Logic Model Tables
Table 6. Correspondence between "What we do" boxes on the logic model in Figure 13 and Nutrients SDR Pilot goals. Metrics that map to
each goal related to communication and corresponding data sources are also included.
What We Do
Associated Goals
Proposed Metrics/Indicators
Source
Engage stakeholders
and communicate
results
Build and maintain
partnerships to
identify and evaluate
the most promising
nutrient remediation
strategies AND ensure
the usability
and transferability of
research
to stakeholders
Encourage and
sustain partner and
stakeholder engagement
in a targeted manner
based on
each stakeholder's role in
the pilot
1. Number of formal quarterly meetings to discuss any aspect
related to the evaluation or for building relationships with
stakeholders, including community members.
2. Number of informal (often one-to-one or impromptu)
meetings to discuss any aspect related to the evaluation or
for building relationships with stakeholders, including
community members.
3. Number participants in formal meetings.
4. Number of stakeholders that interact with the knowledge
products (# of downloads of policy notes, briefs,
blogs, memo articles, working papers etc. from your
website).
5. Presence of note-takers and whether they were able to
capture participant perspectives for a workshop.
6. Changes (positive or negative) in attitudes or behavior of key
stakeholders about the impact of communications and
engagement, using the 2020 internal researcher interviews
as a baseline
Monthly report
(1, 2, 3)
Web analytics (4)
Other(5, 6)
49
-------
What We Do
Associated Goals
Proposed Metrics/Indicators
Source
Produce and
deliver accessible
products tailored
to specific stakeholder
needs to make EPA
science more usable on
the Cape and
transferable to other
areas
1. Number of events at which information related to study is
presented.
2. Number of knowledge and information products, including
name and brief description e.g., cartoon story boards,
illustrations, songs, stories or theatre, briefs of any type,
blogs or papers produced as part of the intervention or for
documenting impact evaluation analysis and findings (please
specify).
3. Was ORD invited to participate in conferences and meetings
by stakeholders on nutrient inputs and/or alternative
technologies? How many?
4. Change in institutional culture of evidence use or
commitment to communication and evaluation, using the
2020 internal researcher interviews as a baseline
Monthly report (1,3)
Other(2, 4)
50
-------
What We Do
Associated Goals
Proposed Metrics/Indicators
Source
Facilitate pilot
researchers
in communicating both
scientific results and their
research efforts with
stakeholders throughout
the research process
1. Number of publicly facing knowledge and information
products, including name and brief description e.g.,
illustrations, stories or briefs of any type, blogs or papers
produced as part of the intervention or for documenting
impact evaluation analysis and findings; as well as
contributing or being interviewed for media communications
like newspaper articles (online and in print); including Tweet
threads and social media management.
2. Number of participants at formal meetings who are decision-
makers (estimate), including community members as
decision-makers as appropriate.
3. Number of events at which information related to study is
presented.
4. Number of stakeholders that interact with the knowledge
products (# of downloads of policy notes, briefs, blogs,
memo articles, working papers etc. from your website)
5. Are we incorporating a diversity of ideas, views, and
concerns shared from different members of the public?
6. Scale-up of the program to which the findings have
contributed in some verifiable way.
Other (1, 5, 6)
Monthly report (2, 3)
Web analytics (4)
Evaluate
contributions of
SDR methods to
develop
promising practices
for engagement
with stakeholders
Evaluate contributions of
SDR methods to develop
promising practices for
engagement
with stakeholders
1. Create and maintain a list of all data and information sources
we will use for this evaluation and where these items are
located.
2. Did we collect and measure the indicators we set out to use
for evaluation of each goal?
3. Number of knowledge and information products, including
name and brief description e.g., illustrations, stories, briefs
of any type, blogs or papers produced as part of the
intervention or for documenting impact evaluation analysis
and findings (please specify), media mention.
Other (1, 2, 3)
51
-------
Table 7. Correspondence between "Short-term impact" boxes on the logic model in Figure 13 and Nutrients SDR Pilot goals. Metrics that map
to each goal and corresponding data sources are also included. Short-term impacts and associated goals were developed before
communication goals were finalized and may not reflect the most recent iteration of communication goals.
Short-term Impact/Associated Goals
Proposed Metrics/Indicators
Source
Key stakeholders show change in
attitudes/behaviors about impact
of communication/engagement and
in perspective regarding values of
community-wide research efforts
and engagement
1. Changes (positive or negative) in attitudes or behavior of key
stakeholders about the impact of communication and
engagement through a survey (upon which we will rely on a partner for
dissemination and collection). We will use the pre-problem
formulation workshop as our baseline survey data. Interviews can also
be used.
2. Observable/Measurable change in stakeholder perspective regarding
the value of community-wide research efforts and engagement through
research process (interviews with people before and after
communication initiative / plan).
3. Look at survey data from pre-SDR Problem pilot workshop to
evaluate interest and compare it to a very end workshop for the project
(can BCWC send a final project evaluation survey?).
Problem
formulation survey/key
stakeholder interviews (1, 2)
Other(4)
52
-------
Short-term Impact/Associated Goals
Proposed Metrics/Indicators
Source
ORD researchers internal and
external to the Nutrients SDR Pilot
understand the value of stakeholder
communication and engagement
1. Change in institutional culture of evidence use or commitment to
communication and evaluation: track this through a yearly focus group
with project scientists and gauge enthusiasm for producing alternative
communication products.
2. Are we incorporating a diversity of ideas, views, and concerns shared
from different members of the public?
3. Is there something we tried that did not work? (topics, events,
products that received little or no interaction, or
negative interaction) The whole point of a pilot is to learn so it is
important to think critically for improvement.
4. Look at survey data from pre-SDR Problem pilot workshop to
evaluate interest and compare it to a very end workshop for the project
(can BCWC send a final project evaluation survey?).
5. Create a list and debrief on "lessons learned". What are the lessons
learned from the entire project?
6. Create a list of communication methods that project scientists used
and discuss what worked best and least.
7. List successes and failures of the communication plan.
8. Ask people if the engagement made a difference: what did you think
worked best/worst? Did you feel like this event was a valuable use of
your time? Did you communicate and engage through this specific TSP
project in a different way than you would've otherwise (getting at value
of SDR/tsp)? Do this through interview.
Problem formulation
survey (4)
Focus groups(1)
Exit interviews (2, 3, 5, 6, 7,
8)
53
-------
Short-term Impact/Associated Goals
Proposed Metrics/Indicators
Source
ORD managers internal and external
to the Nutrients SDR
Pilot understand the value
of stakeholder communication and
engagement
1. Number of written updates/reports delivered to ORD managers.
2. Number of briefings requested by ORD managers.
3. Number of pilot-related events involving stakeholders at which an
ORD manager is present.
4. Scale-up of the programme to which the findings have contributed in
some verifiable way (this metric is appropriate for some project
components but not all of them.) Consider a scale-up in terms of
whether this project has influenced interest in this research in other
places on Cape Cod; a change in the intervention scale (private ->
neighbourhood-scale; or neighbourhood scale -> regional scale).
Monthly report (1)
Other (2, 3, 4)
54
-------
Appendix D: Interview Questionnaire
Interviews with project researchers and key stakeholders are being conducted twice during the
lifespan of the Nutrients SDR Pilot to track how the perspectives of internal researchers toward
communication evolve during the project's duration. This is the questionnaire that was used for
interviewing internal EPA researchers and stakeholders. These questions can be adapted to meet your
organization's needs and evaluate and improve internal communication processes.
Scripted Project Description
You are being asked to take part in a research study. The purpose of this study is to understand how
stakeholder engagement has influenced your research in the Nutrients Solutions-Driven Research Pilot,
and to identify best practices and lessons learned for stakeholder engagement in solutions-driven
research based on the pilot. The results of this study will be aggregated to inform an EPA report
offering lessons learned from this pilot and recommended practices in solutions-driven research at
EPA.
If you agree to be in this study, this confidential interview will take 30-60 minutes of your time today,
and then again in one year and at the conclusion of the project. Questions will address your personal
perspectives on conducting solutions-driven research on Cape Cod and will invite you to reflect on your
stakeholder engagement and communication experiences in the project. Participation in this interview
is entirely voluntary. You are welcome to skip any questions you do not want to answer and end the
interview at any time without affecting your relationship with the investigators of this study or EPA.
There are no known risks, benefits, or compensation. Additionally, you have the right to request that
the researchers not use any of your responses.
Your responses will be strictly confidential. We will immediately replace your name with a pseudonym
in our files following the interview, and the recording of this interview along with the form coding your
given name with the pseudonym will be saved in a password-protected folder on an EPA device.
Questionnaire for ACESD Researchers
How do you think your research is beneficial to the communities on Cape Cod?*
What do you see as your role in the Nutrients SDR Pilot?*
What do you see as markers of success for your research in this project?
Stakeholder Engagement in Research
What type of things do you think of as stakeholder engagement?*
What kind of stakeholder engagement has been a part of your past work prior to this pilot?
What motivated you to engage with stakeholders in past projects?
What kind of impediments have you experienced in trying to engage with stakeholders in your
past research?
How have you previously identified relevant people to engage?
55
-------
Nutrient SDR Engagement
How does this project's approach to engagement differ from past projects?*
Which stakeholders you have personally interacted with for your research on this project?*
How frequently do you interact with these stakeholders?*
Have you worked with these stakeholders before?*
Are you satisfied with the contributions from your stakeholders?*
Are there areas where there could be better quality/frequency of contributions?*
How has your team benefitted from working closely with stakeholders on your work in this
project (overall/for your team specifically)?*
Any specific examples?*
What are the drawbacks?*
What kind of impediments have you experienced in trying to engage with stakeholders in your
research for this pilot?*
Why do you think stakeholder engagement is a part of this project (overall/for your team
specifically)?*
How do you think stakeholders should be involved in the Cape Cod project overall?*
How about for your specific project (should they be more or less involved)?*
A variety of stakeholders are currently a part of the project in many different ways - do you
think this is adequate (to reach project/your research goals)? (Is it excessive?) If not, what
might need to change for adequate stakeholder involvement across various relevant groups?*
Whose job is it to engage with stakeholders for this project? Why?*
Has engagement with stakeholders changed how you did your research? How?*
Has stakeholder engagement affected the anticipated products you're making? How?*
As a percent of your overall research time, how much of that time has been focused on this SDR
Pilot?
Of that time, what percent of your time do you dedicate to stakeholder communications?*
Solutions-Driven Research at EPA and Moving Forward
How would you define (describe) solutions-driven research?*
What do you think about expanding EPA's application of a solutions-driven approach to
research?*
Does it make sense to expand this approach to apply it to the majority of ORD's projects?*
Would adopting an SDR approach be conducive to your other projects and future research?
How?*
How would that impact the quality of research your team conducts?*
56
-------
Lessons Learned
What would you do differently next time in engaging with stakeholders?*
Are there ways you could improve how you engage with stakeholders?*
What have you learned from the SDR Pilot that will shape your efforts to engage with
stakeholders moving forward?*
Are there any tools/trainings that would make you feel more comfortable with stakeholder
engagement?*
Do you have any advice on how to improve ACESD's/ORD's approach to solutions-driven
research based on your experience with this project?*
Anything else you'd like to add about stakeholder engagement in solutions-driven research?*
Questionnaire for Stakeholders
How do you think EPA's research is beneficial to the communities on Cape Cod?*
What do you see as EPA's role in the nutrients SDR Pilot?
What do you see as your role in the nutrients SDR Pilot?*
What do you see as markers of success for your work on this project?
Stakeholder Engagement in Research
What type of things do you think of as stakeholder engagement?*
What kind of stakeholder engagement/inter-organization collaboration has been a part of your
past work prior to this pilot?
What kind of impediments have you experienced in trying to engage with stakeholders
(researchers) in your past work?
Nutrient SDR Engagement
Why do you think stakeholder engagement is a part of this project (overall/for your team
specifically)?*
How does this project's approach to engagement differ from past projects?*
Which stakeholders (researchers) have you personally interacted with for your research on this
project?*
How frequently do you interact with these stakeholders?*
Have you worked with these stakeholders before?*
Are you satisfied with the contributions from these stakeholders?*
Are there areas where there could be better quality/frequency of contributions?*
How has your organization benefitted from working closely with EPA on your work in this
project (overall/for your team specifically)?*
Any specific examples?*
57
-------
What are the drawbacks?*
What has gone particularly smoothly in working with EPA on this project?
What kind of impediments have you experienced in trying to engage with EPA and other
stakeholders for this pilot?*
What has been difficult about working with EPA on this project?
Are there any notable challenges to working with EPA that go beyond completion of the
research?
What do you think of the balance of roles played by EPA and other stakeholders in the project
overall?*
How about for your piece of the project (should they be more or less involved)?*
A variety of stakeholders (including researchers) are currently a part of the project in many
different ways - do you think this is adequate (to reach project/your research goals)? (Is it
excessive?) If not, what might need to change for adequate stakeholder involvement across
various relevant groups?*
Whose job is it to engage with stakeholders and the EPA for this project? Why?*
Has engagement with EPA changed how you proceeded with this project? How?*
Has stakeholder engagement affected the anticipated research products you're making? How?*
As a percent of your overall work time, how much of that time has been focused on this SDR
Pilot?
Of that time, what percent of your time do you dedicate to stakeholder communications?*
Lessons Learned
What would you do differently next time in working with EPA?*
What do you think EPA should do differently next time in building a research partnership?
In engaging with researchers?
What have you learned from this pilot that will shape your efforts to engage with research
institutes moving forward?*
Do you have any advice on how to improve ACESD's/ORD's approach to solutions-driven
research based on your experience with this project?*
Anything else you'd like to add about stakeholder engagement in solutions-driven research?*
58
-------
Appendix E: Nutrients SDR Pilot Tracking Tools
for Metrics
Table 8. Metrics for the Nutrients SDR Pilot organized by tracking tool type.
Tracking Tool
Metrics
Monthly
Report
Number of formal quarterly meetings with stakeholders
Number of events at which info related to study is presented
Number of written updates/reports delivered to ORD managers
Number of informal meetings with stakeholders (approximately)
Number of participants/decision-makers in formal meetings with stakeholders
Was ORD invited to participate in conferences and meetings by stakeholders?
How many?
Number of pilot-related events involving stakeholders with an ORD manager
present (specify that the column for EPA staff also includes managers)
Presence of note-takers. Number of briefings requested by ORD managers
Internal
Review
Database
Number of knowledge and information products including name and brief
description (we are capturing Number of but not necessarily name or description
in monthly report)
Internal
Interviews
Create list of communication methods that project scientists used and discuss
what worked best (list of methods from monthly report but discussion from
interviews at end of project with each team about communications)
Change in institutional culture
Are we incorporating a diversity of ideas? (how did researchers incorporate
feedback from stakeholders into their research and communication of results?)
Ask people if engagement made a difference
Key
Stakeholder
Interviews
Ask people if engagement made a difference etc.
Changes in attitudes or behaviors of key stakeholders about impacts of
communication
Observable/measurable changes in stakeholder perspective regarding value of
community-wide research efforts and engagement through research process
Scale-up of the program
Pre and post survey data to evaluate interest
Web
Analytics
Number of actors that interact with knowledge products
Reflexive Self-
assessment &
Internal
Interviews
Is there something we tried that didn't work?
Create a list of "lessons learned"
List successes and failures of comms plan
Did we collect and measure indicators?
Other
List of all data and information sources used for evaluation and where they are
located (up to evaluator)
59
-------
Appendix F: Summary of Communications Plan
The U.S. Environmental Protection Agency's (EPA) Office of Research and Development (ORD) is
working closely with stakeholders on a Nutrients Solutions-Driven Research (SDR) Pilot to design and
implement research to inform watershed-scale solutions to nutrient pollution. A primary goal of this
pilot project is to support our stakeholders in meeting their nutrient reduction goals. This pilot is
focused on excess nutrients issues on Cape Cod, Massachusetts, that have caused impairments in more
than 30 small estuaries and have resulted in reductions in water clarity, increases in macroalgae, fish
kills, and more. These impacts can also affect coastal home values and recreation, which are of high
social and economic value to the residents and visitors of Cape Cod.
Nutrients SDR Pilot Communication Plan Goals
1. Encourage and sustain partner and stakeholder engagement and trust in
a targeted manner based on each stakeholder's role in the Nutrients
SDR Pilot.
2. Produce and deliver accessible products tailored to specific stakeholder
needs to make EPA science more usable to Cape Cod communities and
transferable to other areas.
3. Facilitate two-way communication between pilot researchers and
stakeholders throughout the research process.
4. Evaluate communication and engagement methods to identify
promising practices for future research.
5. Facilitate and encourage internal and external communication with
researchers and managers about the Nutrients SDR pilot to foster a
better understanding of the value of stakeholder communication.
6. Develop a project-specific SDR Communications Plan to act as a
communications model for future SDR projects in ORD.
Another major goal of this Nutrients SDR Pilot is to serve as a test case for designing and enacting
extensive and thoughtful communication efforts throughout ORD's research process.
"Communication" in this Nutrients SDR Pilot Communications Plan encompasses both traditional
efforts focused on research updates and more explicit stakeholder engagement strategies to
cooperatively design and implement research projects to provide the information needed by our
stakeholders for their management efforts. This Nutrients SDR Pilot Communications Plan is designed
to be a living document for the researchers and managers to track communications activities and
evolving stakeholder needs, as well as to evaluate the pros and cons of various communication
approaches. There is an accompanying report, "Communications Planning in Solutions-Driven
Research," that provides more extensive information on the methods for designing the Nutrients SDR
Pilot Communications Plan, so the methods are not described in detail here.
60
-------
This Nutrients SDR Pilot Communications Plan details the past, existing, and anticipated
communication efforts for the pilot within the context of the project's general and communication-
specific goals (refer to inset box), in line with the Nutrients SDR goals, this Nutrients SDR Pilot
Communications Plan emphasizes a stakeholder-centric approach and provides detailed guidance on
targeted communication strategies for effectively engaging with different stakeholders and partners in
research and communications. These goals emphasize the importance of integrating two-
way stakeholder communication throughout the research process and serve to define the major
outcomes desired from both research and communication efforts. As is expressed in these goals, the
underlying momentum behind the project implementation and communication efforts has been built
through trust building, consistent two-way communication, targeted updates, flexible timelines,
and openness to iteratively updating the stakeholder network. This Nutrients SDR Pilot
Communications Plan also recommends an evaluation of the communication efforts for use in
guiding future ORD research efforts as they incorporate SDR elements in their work.
To determine the appropriate communication pathways, the communications team mapped potential
stakeholders based on researchers' understandings of each stakeholder's interest in and influence over
the project to categorize communication efforts by the different types of stakeholders. The
communications team met with each of the sub-teams on the Nutrients SDR Pilot project to
appropriately capture the degree of engagement of partners and stakeholders by researchers. We
developed a project timeline to track major and recurring communication and engagement events and
outputs for each team and across the project. An array of communication methods was implemented
and is proposed as a part of this pilot by the SDR researchers. These range from traditional academic
journal articles for communicating research outcomes to several informal conversations with
stakeholders and many methods in between.
|^| |j "I" ri "I" c ChD D i I <^+-
Partners
"client," "wireless network," and
"Planning" icons by Gilbert Bages
from the Noun Project
61
-------
The diversity of communication methods is designed to ensure that the products of this research are
easily understood and useful to the full range of stakeholders and partners. These alternative
communication methods were compiled into a table, which identifies production processes,
appropriate uses and audiences, and points of contact for each method.
There are several communication efforts that have been implemented successfully to date. One
notable initial effort was a problem formulation workshop with more than 50 Cape Cod
stakeholders from local, state, and federal agencies, the research community, and environmental
NGOs to define the problem and identify research needs. Communication efforts
from the workshop included publishing the workshop proceedings, sharing the proceedings with all
attendees, and presenting the results at numerous conferences and webinars.
These communication efforts have resulted in targeted research activities for a suite
of technologies including, innovative and alternative septic systems (source control), bioreactors and
cranberry bog restoration (to intercept groundwater nitrogen), and shellfish aquaculture and wetland
restoration (to remove nitrogen that reaches the estuary). A second workshop was held to
look specifically at the use of retired and active cranberry bogs as test beds for nutrient reduction
technologies such as bioreactors and wetland restoration. The outcomes of the second
workshop were communicated via a workshop report and a Twitter thread. As a result, we now
have multiple working groups of diverse stakeholders collaborating on research to
evaluate nitrogen remediation in bogs. Additional ongoing communication efforts include weekly calls
with the key partner and project partners, contributions to community outreach efforts related to
alternative septic system adoption, online monitoring data, and more.
Solutions-driven research calls for intensive and intentionally designed communication and
engagement efforts with an array of stakeholders to ensure the usability of the science. This Nutrients
SDR Pilot Communications Plan outlines the extensive existing and proposed communication methods
that have been designed carefully by ORD researchers and the communications team for the Nutrients
SDR Pilot. While targeted communication outputs and stakeholders are specific to this pilot, we have
crafted this plan as an initial model for future SDR communications plans at ORD and EPA. To
implement this degree of communication requires careful coordination and organization to not
overburden stakeholders and to allow for the synergy of efforts. The required dedicated effort on the
part of the Nutrients SDR Pilot researchers and the communications team to plan for and execute
communication efforts is extensive. This Nutrients SDR Pilot Communications Plan provides
documentation of these novel efforts and a tool for researchers and managers to use in implementing
the Nutrients SDR Pilot.
Range of Communication Methods
Story
Maps
Twitter
threads
+++++
62
-------
v>EPA
United States
Environmental Protection
Agency
PRESORTED STANDARD
POSTAGE & FEES PAID
EPA
PERMIT NO. G-35
Office of Research and Development (8101R)
Washington, DC 20460
Official Business
Penalty for Private Use
$300
------- |