National Estuary Program Program Evaluation Guidance


NATIONAL

H ST IJ A R Y

PROGRAM

National Estuary Program

Program Evaluation Guidance

Final

August 3, 2016

-------
National Estuary Program
Program Evaluation Guidance

Contents

Section I: Purpose, Background, Goals, and Framework

A.	Purpose

B.	Background

C.	Goals

D.	Framework

Section II: Scope and Format of Documentation

A.	Program Evaluation Package

B.	On-site Visit

C.	PE Findings

Section III: Program Evaluation Process

A.	Program Evaluation Team Structure

B.	Program Evaluation Team Responsibilities

Section IV: NEP Groups and Program Evaluation Schedule

A.	NEP Groups

B.	Program Schedule

Attachments:

Attachment 1:

Attachment 2:

Attachment 3:

Attachment 4:
Attachment 5:

NEP Program Evaluation Logic Model (page 12)

Standardized Performance Measures for Program Management Core
Elements (pages 13-19)

Annual Funding Guidance Requirement for Leveraged Resources
Reporting (page 20)

Program Evaluation Schedules (pages 21-27)

Responsibilities for the Parties Involved in the Program Evaluation
Process (page 25)

-------
National Estuary Program
Program Evaluation Guidance

Section I: Purpose, Background, Goals, and Framework

A.	Purpose

The primary purpose of the Program Evaluation (PE) process is to help the U.S.
Environmental Protection Agency (EPA) assess how the implementation of
Comprehensive Conservation and Management Plans (CCMPs) by the 28 programs
included in the National Estuary Program (NEP) are making adequate progress in
achieving programmatic and environmental results. Documented progress merits
continued funding under §320 of the Clean Water Act (CWA). Continued funding for
each NEP under §320 of the CWA is contingent upon Congress appropriating sufficient
funds to the EPA for the purpose of supporting the NEPs and their CCMP
implementation.

The PE process is also useful for:

•	providing case studies of NEP leadership in collaborative and adaptive
management;

•	documenting the value added to environmental management by the national
program and individual NEPs, including their role in convening stakeholders and
interpreting science for management;

•	demonstrating continued stakeholder commitment;

•	assessing the progress of the NEP as a national program; and

•	transferring lessons learned within EPA, among NEPs, and with other watershed
programs.

B.	Background

The EPA began a NEP Implementation Review (IR) process in 1997 to determine which
NEPs with approved CCMPs qualified for continued funding. The IR process was
initially conducted every two years. In 2000, the process was streamlined and the review
cycle was extended from every two to every three years for those programs that had
already undergone the biennial review. In 2003, the IR process was revised with the
intent to make IRs less burdensome to the NEPs while still collecting sufficient
information to evaluate NEP progress and technical transfer. The IR cycle remained a
three year cycle.

In 2006, the IR process was reevaluated due to increased federal program accountability,
e.g., the Program Assessment Rating Tool (PART). An IR Reassessment Team composed
of EPA Headquarters (HQ) staff, EPA NEP Regional Coordinators, and NEP Directors
participated in the reevaluation process that led to the 2007 NEP Program Evaluation

3

-------
Guidance. In 2011, the PE process was reassessed to identify further streamlining
opportunities. The PE framework laid out in the 2007 PE Guidance was maintained, but
the Tracking/Reporting sub-element under the Program Management element was
eliminated with the exception of two of the criteria that were transferred to the Outreach
and Public Involvement sub-element. The PE cycle was also changed from a three year
cycle to a five year cycle (four consecutive years with the fifth year to be spent producing
a findings report).

In 2016, the PE process was reassessed again to identify opportunities for improvement.
The changes are describe below:

• More specific guidance has been added regarding the format of the documentation
for the PE package. Specifically, the work plan summary elements and EPA's
expectations are better defined.

• As such, a number of items have been deleted, added, or clarified. NEPs no longer
need to re-submit work plans with the PE package. Electronic NEP work plans
submitted annually to EPA HQ will be use for the NEP PE.

• The element Tools from the work plan core elements is eliminated because EPA
has is already collecting such information through the element Direct Assistance
based on reviews of past PE letters.

• NEPs will no longer be required to report on CWA implementation. Instead EPA
will extract information on use of CWA tools through the work plan summary
itself.

• For the years covered in the PE cycle, the EPA HQ will provide cumulative report
on habitat restored and protected and leveraged funds data (primary and
significant role) using NEPORT. For PE purposes only, reporting on leveraging
support role was eliminated. This data (support role) will continue to be collected
in NEPORT and Regions will have access to it.

• More emphasis on Management Conference engagement during on-site visit has
been added as a means to foster greater understanding of the NEP and its
particular challenges by both HQ and Management Conference members.

• The role of an ex-officio NEP Director volunteering to serve on the PE team has
been better defined and clarified to respond to comments from the NEPs and EPA
Regions.

• Updated timeline for on-site visit and final PE letter to expedite communication of
PE findings to the NEP Director.

• The due date for PE submittal package is extended to March 15.

-------
The PE cycle remains a five year cycle. Four consecutive years followed by one year off
to prepare a PE finding report of all 28 NEPs.

C.	Goals

The goals of this PE Guidance are to:

•	ensure objective and consistent PEs among the different NEPs;

•	further align the PEs with individual NEP CCMP priorities and related NEP
annual work plans; and

•	measure progress in achieving programmatic and environmental results over the
short-term, intermediate, and long-term, including documenting NEP
contributions to improving or reducing pressures on coastal watersheds.

D.	Framework

This PE Guidance is based on a NEP Program Evaluation Logic Model Framework (NEP
PE logic model) which incorporates the Pressure-State-Response (PSR) model (see
Attachment 1). The NEP PE logic model is designed to help guide reporting on stages of
NEP progress (otherwise referred to as environmental milestones and targets) toward
restoring and maintaining the ecological integrity of each nationally designated estuary.
The NEP PE logic model shows causal links among pressures, activities, partnerships,
outputs, and short-term, intermediate, and long-term outcomes. It is being used to help
the NEPs address the challenges of accounting for external factors such as changes in
social and economic norms. The NEP PE logic model allows the NEPs to be recognized
for their activities toward improving conditions or reducing pressures in their estuary
since a connection between the NEP activities and ultimate environmental change might
be difficult to establish.

Section II: Scope and Format of Documentation

The PE consists of several phases: A) development and submission of a package of
required information, B) PE team site visit to each NEP under evaluation, and C)
documentation of PE findings via formal letter from EPA HQ.

A. Program Evaluation Package

1) Standardized Performance Measures Responses

For the years covered within the PE cycle, the NEP should assess its performance
against the standardized measures laid out in Attachment 2. This assessment
must be substantiated by supporting documentation (e.g., State of the Bay

Reports, finance plan, adaptive management strategies, etc).

The standardized performance measures address program management practices
common to all NEPs and cover the following elements:

5

-------
•	Financial Management

•	Program Planning and Administration

•	Outreach and Public Involvement

•	Reporting

•	Assessment and Monitoring

•	Research (optional)

In many instances, the standardized performance measures will reflect work plan
activities and therefore should also be describedfurther in the NEP work plan
narrative summary described below.

2) NEP Work plan Narrative Summary

The EPA is not specifying a page limit for the NEP work plan narrative
summary. The document should be concise and fully address each topic listed
below with supporting documentation. For the years covered within the PE cycle,
the NEP should summarize the following:

•	Key NEP work plan goals and activities, stages of progress toward
achieving the NEP's environmental milestones and targets, and role of key
stakeholders supporting the activities. The following elements describe
generally what is in an NEP work plan to achieve CCMP goals:

1)	Habitat

2)	Water Quality

3)	Living Resources

4)	Healthy Communities

5)	Trainings

6)	Direct Assistance

To the extent you are able, use these elements to organize your NEP's key work
plan goals and activities by using the logic model components; if correlation is
not possible, use your existing work plan structure.

Note: NEP outcomes are understood to be sensitive to changes in climate.

Reports on relevant climate change activities should be incorporated in these
narratives.

•	A brief summary of how each challenge identified in the previous PE has
been addressed.

•	A budget summary (pie charts or other graphics are acceptable) with an
accompanying brief narrative showing funding use during the PE cycle;
summary should include a breakdown of funds used for program staff as
well as funds spent on specific projects and other activities.

•	A description of external factors (e.g., institutional barriers, emerging
issues) affecting the NEP work plan goals and/or progress, and related

6

-------
adaptive management strategies. Indicate ways EPA can support efforts to
address these factors.

The EPA HQ will provide to the PE team leader the following information
from NEPORT to further demonstrate accomplishments to work plan goals
and activities for the PE cycle:

• A cumulative total of the funds leveraged for the years covered in the PE
cycle, and breakdown by primary and significant leveraging roles (see
Attachment 3).

• A cumulative total of the acres protected and restored, and breakdown by
habitat type for the years covered in the PE cycle.

If there insufficient information in the PE package, the NEP will be required to provide
supplemental documentation addressing questions or information gaps identified by the
PE team.

B. On-Site Visit

After the PE package is submitted, the PE team will plan and schedule an on-site visit.
The on-site visits are typically one to three days in length. The intent of the visit is to
foster greater understanding of the NEP and its particular challenges by both EPA HQ
and Management Conference members and to share initial evaluation findings with the
NEP and its key members. On-site visits also provide opportunities for the PE team to
view on-the-ground projects. The PE team will work with the NEP Directors to determine
the best time to schedule the visit.

The NEP should use the PE on-site visit to:

• demonstrate successes and accomplishments, especially those that are innovative
and have technical transfer possibilities;

• expand upon work plan progress summarized in the work plan narrative
summary; and

• demonstrate how external factors may be influencing progress toward
environmental milestones and targets.

The PE team members should use the on-site visit to:

• meet and build relationships among EPA and NEP partners;

• discuss any questions or issues with submission of PE materials;

• discuss preliminary findings (strengths and challenges); and

• identify and discuss potential program recommendations with NEP, if feasible.

C. PE Findings

Each NEP will receive a final PE letter with a determination of "pass, " "conditional

-------
pass" or "fail" (see Figure 1). The letter from EPA will formally document the
determination based on the standardized performance measures responses, work plan
narrative summary, and information provided by the NEP through discussion and site
visit. The letter will include details about program strengths, challenges, and
recommendations for improvements, including timeframes, as needed. In the case that an
NEP "fails" the evaluation review, EPA will work closely with the NEP to improve its
performance. The EPA will decide on a case-by-case basis the status of the annual §320
allocations for any NEP that does not merit continued funding.

Figure 1: Thresholds for Final Determination

Pass

Conditional Pass

Fail

The Program:

The Program:

The Program:

S shows progress

S does not show progress toward

S does not show

toward

environmental milestones and targets,

progress toward

environmental

but meets all baseline expectations

environmental

milestones and

for Fully Performing in all sub-

milestones and

targets;

elements;

targets;

and

or

and, either:

S meets all

S does not show progress toward

S is at the level of

baseline

environmental milestones and targets,

Minimally

expectations

and has not met all baseline

Performing in

for Fully

expectations for Fully Performing in

three or more sub-

Performing in

up to three sub- elements;

elements;

all sub-

or

or

elements.

S shows progress toward environmental

S receives repeated





milestones and targets, but has not

"conditional



met all baseline expectations for

passes " on the



Fully Performing in up to three sub-

same challenge(s).



elements.



Section III: Program Evaluation Process
A. Program Evaluation Team Structure

The PE teams for each NEP will include the EPA HQ NEP coordinator (PE team leader),
the EPA Regional NEP coordinator, and if possible an ex-officio NEP Director. The
purpose of the NEP ex-officio Director in the PE process is to allow an exchange or
sharing of information from one NEP to another (e.g., provide technical transfer
assistance to the NEP undergoing the PE and take lessons learned back to their own
NEP). Although, the ex-officio NEP Director is expected to review and comment on the
draft PE letter, he/she cannot be involved in EPA's final determination. Responsibilities

8

-------
for the PE team members are outlined in Section III.B. below and in Attachment 5.

The EPA HQ and Regional Coordinators are expected to include PE on-site visits for
their NEPs in their annual travel budget plans; however, travel for on-site visits is
dependent on the availability of funds. Alternative arrangements in the absence of travel
funds include webinars or video-conferences. The schedule and groupings for upcoming
PEs and their associated on-site visits is presented in Attachment 4.

B. Program Evaluation Team Responsibilities

The PE team is responsible for reviewing materials and their consistency with this
guidance, planning the site visits, and developing initial findings, ideally to present to the
NEP during the site visit; evaluations will be based on the following.

• Responses and supporting documentation related to standardized performance
measures (see Attachment 2). The PE team will use the standardized
performance measures as a tool to objectively identify a program's strengths,
challenges, and areas for improvement in the program management area. The
standardized performance measures are based on a descriptive scoring system of
four levels: Excellent, Good, Fully Performing, and!or Minimally Performing. All
the core elements will have the same weight of importance in terms of overall
evaluation conclusions.

• Work plan narrative summary related to NEP specific work plan goals and
activities (see Attachment 1). A qualitative assessment will be done on the work
plan narrative summary that includes discussion of key NEP work plan goals and
activities as they implement the CCMP and contribute to environmental
improvements. This assessment will include attention to the details as described in
Section II. A.2. above.

• Observations made during the on-site visit, and team discussions with NEP
staff Management Conference, stakeholders and partners. The on-site visit will
be used to discuss any questions or issues with the PE submission, discuss
preliminary findings, and explore possible recommendations. It is a chance to
visit project sites and meet with NEP Policy and/or Management Conference
members, stakeholders and partners to listen to their insights and concerns about
CCMP implementation.

The PE team will document its findings in writing based on comments provided by each
PE team member (electronic comments will be submitted to the PE team leader by each
PE team member). The findings will reflect the PE package, the on-site visit, and
discussions with the NEP. The PE team will develop a final PE letter for EPA
management based on the PE team's documented findings as described in Section II.C.
above.

-------
Section IV: NEP Groups and Program Evaluation Schedule

A. NEP Groups

NEPs are distributed in four groups (A, B, C, and D) as shown in Figure 2. The PE
Schedule for each group is shown in Attachment 4. If there are any changes to this
schedule, the EPA will notify the NEPs in a timely manner.

Figure 2: PE Groups A-B-C-D

PE Group A
PE: 2017

PE Group B
PE: 2018

PE Group C
PE: 2019

PE Group D
PE: 2020

(7 Programs)

(6 Programs)

(8 Programs)

(7 Programs)

B ar atari a-

Tampa Bay

Buzzards Bay

Charlotte Harbor

Terrebonne







Casco Bay

Coastal Bend Bays

Partnership for the
Delaware Estuary

Morro Bay

Indian River Lagoon

Albemarle-Pamlico
Sounds

Long Island Sound

Columbia River

Massachusetts Bay

Delaware Inland
Bays

Santa Monica Bay

Barnegat Bay

Peconic Bay

Narragansett Bay

New York/New
Jersey Harbor

Piscataqua Region

Estuaries

Partnership

San Juan Bay

Sarasota Bay

Puget Sound

San Francisco Bay

Tillamook Bay



Maryland Coastal
Bays

Mobile Bay





Galveston Bay*



* Galveston Bay will have its PE one year later.

B. Program Evaluation Schedule

1)	The PE team leader should hold a conference call with members of the team
and the NEP Director undergoing the PE at least three months prior to the
deadline for submitting the PE package. The purpose of the conference call
will be to discuss logistics on the preparation and submission of the PE
package. Discuss process timeline and clarify roles and responsibilities. The
NEP Director may choose to include Management Conference members or
other stakeholders.

2)	Program Evaluation packages will be due to EPA HQ PE team leader on
March 15. The PE team leader will see that electronic copies of NEP work
plans for the years covered within the PE cycle (see Attachments 5) are
sent to the ex-officio NEP Director by March 15. If March 15 falls on a
weekend, the packages will be due on the following Monday.

10

-------
3)

The PE team members should review the PE package and submit written
electronic comments to the PE team leader within three weeks after receiving
the PE package.

4)	The PE team leader should hold a conference call with the PE team members
one week after receiving comments from the PE team. The purpose of this
conference call is to:

•	discuss the PE findings; and

•	identify follow-up questions or information gaps requiring the NEP to
submit additional documentation.

5)	The PE team leader should schedule a conference call between the NEP
Director and the PE team within two weeks after conducting the PE team
conference call. The purpose of this conference call is to:

•	discuss strengths and challenges of the NEP;

•	discuss additional documentation needed to address any information gaps
identified by the PE team. Such documentation should be provided for
EPA review prior to the on-site visit or demonstrated during the on-site
visit; and

•	schedule and discuss the agenda for the on-site visit.

6)	Conduct on-site visits within two months after receiving the PE package.

7)	The PE team should hold a conference call with the NEP Director within two
weeks after the on-site visit to allow the NEP Director the opportunity to
address any concerns raised during the on-site visit.

8)	The PE team should prepare a final draft PE letter within six weeks after the
on-site visit and should provide the NEP Director an opportunity to review the
letter before it is finalized.

9)	The PE Team will finalize the formal findings letter for signing by the NEP
HQ Branch Chief.

10)	The HQ NEP Branch Chief will review and sign the PE letters within three
weeks of receipt from the PE team leader.

11)	If the NEP Director request the PE team can discuss the final PE letter with
the Management Conference in order to discuss and/or clarify findings.

11

-------
Attachment 1: The NEP Program Evaluation Logic Model

EVALUATION
PLANNING

From
CCMP

CCMP and
Work plan
Goals

Operational
Strategy

Reporting
on Impact

Action Plan/
NEP Work

¦>

NEP Core Elements

PROGRAM
MANAGEMENT

Program

Implementation and
Reporting

> Financial

Management

> Program Planning
and Administration

> Outreach and
Public Involvement

Ecosystem Status
and Trends

> Research

> Assessment and
Monitoring

> Reporting

WORK PLAN

Ecosystem
Restoration and
Protection Projects*

> Habitat

> Water Quality

> Living Resources

> Healthy
Communities

* Assumes human and
built environments are
part of ecosystem

Technical Assistance
and Capacity
Building

> Training

> Direct Assistance

RESPONSE

Activities

From Work plan

Partnerships

Outputs

"Deliverables"

. in

5
-------
Attachment 2: Standardized Performance Measures for Program Management Core
Elements

-------
Core Element: Program Implementation and Reporting
Sub-element: Financial Management

Program Implementation and Reporting

Financial Management

Program Planning
and Administration

Outreach and Public
Involvement

LEVEL

PERFORMANCE MEASURES

Excellent

The Program demonstrates Excellent performance because:

o The Program researches, identifies, and tracks prospective donors and funding opportunities
(applicable for non-profit organizations).

o Program staff, Management Conference members, and volunteers have received finance/fundraising
training if appropriate.

o The majority of the Program's outreach materials contain funding information (e.g., thanking donors,
acknowledging project funding, including a membership form, etc.).

Good

The Program demonstrates Good performance because:

o The Program has a current finance plan (approved by the Management Conference within the past

six years) that includes estimated costs, funding sources, goals, responsibilities, and milestones,
o The Program integrates finance planning into its annual workplan (i.e., an assessment of funding

obtained in the previous year, current funding, and funding to be pursued in the coming year),
o The Program has a monthly revenue and expenditure tracking system.

o The Program has a case statement (a brief statement outlining accomplishments and results that
could occur with additional resources).

Fully Performing

Baseline expectations.

o The Program meets its non-federal match obligation and provides detail in the annual workplan

submittal to the EPA about match funding sources and uses (e.g., workplan tasks),
o The Program has a plan for diversifying and augmenting funding sources that is approved by the

Management Conference and includes estimated costs, goals, responsibilities, and milestones,
o The Program has the partnerships and strategic alliances to identify and secure resources to
implement its CCMP.

Minimally
Performing

The Program does not meet all of the performance measures in the Fully Performing level.

The EPA expects that, in order to be a Fully Performing Program, all baseline expectations are met. Performance measures in the Good
and Excellent levels are not required. They are benchmarks for what the Program can do to improve performance given the Program's
priorities and organizational capacity.

-------
Core Element: Program Implementation and Reporting
Sub-element: Program Planning and Administration

Excellent

The Program demonstrates Excellent performance because:

o The Program encourages professional development opportunities for staff members,
o The Program is a leader in the transfer of lessons learned in watershed management.

Good

The Program demonstrates Good performance because:

o The Program has a Management Conference that:

¦ has a written vision statement and/or mission and goals;

¦ is fully engaged in developing and implementing the workplan;

¦ assists in building active partnerships;

¦ ensures broad stakeholder representation in priority setting and Program oversight;

¦ provides a clear and transparent decision-making process that includes the public (e.g.,
operating procedures, agreements and/or bylaws for committees, etc.); and

¦ has a mechanism for identifying existing and emerging issues,
o The Program is seen as a leader in watershed management.

Fully Performing

Baseline expectations.

o The Program has a Management Conference that:

¦ is fully staffed;

¦ provides Program direction;

¦ oversees development and approves annual budget and workplan;

¦ ensures sufficient Program resources;

¦ sets a framework for bringing together diverse interests in a collaborative fashion (e.g., develop
synergy among various organizations);

¦ ensures communication between Program committees;

¦ ensures Program actions are based on both stakeholder priorities and good science;

¦ communicates about and supports the Program; and

¦ has a process for reevaluating its priorities.

o The Program staff coordinates and supports Management Conference responsibilities,
o The Program has human resources principles in place (e.g., staff members have position

descriptions and periodic performance reviews),
o The Program office has autonomy with regard to the host entity (e.g., sets and follows its own
priorities, exhibits visibility in the watershed, etc.).

Minimally
Performing

The Program does not meet all of the performance measures in the Fully Performing level.

-------
Core Element: Program Implementation and Reporting
Sub-element: Outreach and Public Involvement

Program Implementation and Reporting

Financial Management

Program Planning
and Administration

Outreach and Public
Involvement

LEVEL

PERFORMANCE MEASURES

Excellent

The Program demonstrates Excellent performance because:

o The Program supports citizen recommendations by implementing/supporting priority projects via the
annual workplan.

o The Program has a media/marketing campaign underway, such as a social marketing campaign, with

a specific behavior change message related to a CCMP priority issue(s).
o The Program has a brand/image and related graphics, tag lines, etc. that effectively promote and

create widespread recognition of the Program,
o The Program has socio-economic indicators to monitor and report on the impact of outreach and

public involvement activities,
o Efforts exist to achieve and document behavior change.

Good

The Program demonstrates Good performance because:

o The Program has an active CAC or analogous structure that proposes workplan projects and is

represented during Management Conference or executive committee meetings,
o The Program, through the communication plan, actively conducts outreach through such things as

signage, radio/TV spots, special events, public presentations, topic-specific workshops, etc.
o The Program supports efforts to develop and implement such things as environmental education
curricula, teacher training, ecotourism programs, small grant programs, estuary celebrations, and/or
citizen recognition programs,
o The Program shares innovations and lessons learned at regional and national meetings (e.g.,

Estuarine Research Federation (ERF) biennial meeting, The Coastal Society (TCS) biennial meeting,
Coastal Zone (CZ) biennial meeting, NEP national meeting, etc.).
o The Program reports annually programmatic results to the public and stakeholders (via the Program's
website, public database, hard copies, and/or other media) as specified in the NEP Funding
Guidance and describes progress linked towards annual workplan goals and milestones.

Fully Performing

Baseline expectations.

o Citizens are involved in Program decision-making and implementation (e.g., Citizens Advisory
Committee (CAC) or analogous structure, system for public input, open meetings, public notice of
meetings and events, and/or opportunities for reviewing and prioritizing outreach and public
involvement projects, etc.).
o The Program has a multi-year, strategic communication plan that includes needs, target audience(s),

objectives, project descriptions, deliverables, and deadlines,
o The Program has multi-media communication tools (e.g., newsletters, annual reports, fact sheets,

website, listserves, and/or videos/CDs, etc.) that are updated as needed,
o The Program reports programmatic results to the public and stakeholders (via the Program's website,
public database, hard copies, and/or other media) as specified in the NEP Funding Guidance.

Minimally
Performing

The Program does not meet all of the performance measures in the Fully Performing level.

-------
Core Element: Ecosystem Status and Trends
Sub-element: Research*

Ecosystem Status and Trends

Research*

Assessment and Monitoring

Reporting

LEVEL

PERFORMANCE MEASURES

Excellent

The Program demonstrates Excellent performance because:
o Research is used to change policy.

o The Program shares its science and technology research and findings at regional and national
meetings (e.g., Estuarine Research Federation (ERF) biennial meeting, The Coastal Society (TCS)
biennial meeting, Coastal Zone (CZ) biennial meeting, NEP national meeting, etc.).
o Scientific and technical reports produced by the NEP are peer reviewed,
o Program staff sits on state and national science boards and committees.

Good

The Program demonstrates Good performance because:
o Research is conducted by appropriate partners.

o Research identifies significant, missing data that warrant additional monitoring or sampling,
o The Program uses research results to develop management options and implement solutions,
o Results from research are combined and translated into plain English for reporting to the public,
o The Program or its partners have established a process to regularly reevaluate its research needs.

Fully Performing

Baseline expectations.

o The Program or its partners has a process to identify research needs,
o The research needs are consistent with CCMP goals and actions,
o The Program's research needs are approved by the Management Conference.

Minimally
Performing

The Program does not meet all of the performance measures in the Fully Performing level.

*The Program has the option to report a "not applicable" for the Research sub-element. However, if not applicable, the Program
must include justification that either (1) research is not a priority for the Management Conference, or (2) lack of resources does
not allow the Program to conduct or support research efforts.

-------
Core Element: Ecosystem Status and Trends
Sub-element: Assessment and Monitoring

Excellent

The Program demonstrates Excellent performance because:

o The monitoring plan produces sufficient data to support a comprehensive and integrated analysis of
environmental conditions.

o The Program or its partners seeks more efficient and cost-effective technologies for monitoring as
appropriate.

o The Program trains volunteer groups to improve the quality of data collection.

Good

The Program demonstrates Good performance because:

o The Program uses monitoring data to assess and re-direct management actions and programs

implemented under the CCMP as necessary,
o The monitoring plan has a schedule for review/updates that is approved by the Management
Conference.

o The Program uses monitoring data to identify gaps in knowledge,
o Available data is analyzed for ecosystem status and trends.

o The Program promotes the establishment of volunteer monitoring groups to supplement NEP
monitoring efforts.

Fully Performing

Baseline expectations.

o The Program has a Scientific and Technical Advisory Committee (STAC) or analogous structure to

ensure that Program decision-making is tied to good science,
o The Program has indicators in use that are recognized by the Management Conference,
o The Program has a monitoring plan in use that is recognized and/or approved by the Management
Conference and:

¦ meets QA/QC requirements;

¦ identifies various parties' roles and responsibilities for monitoring;

¦ has a timetable for collecting and reporting on data; and

¦ identifies funding needs and/or commitments for the monitoring program.

o The monitoring plan produces data to support an analysis of specific environmental conditions.

Minimally
Performing

The Program does not meet all of the performance measures in the Fully Performing level.

-------
Core Element: Ecosystem Status and Trends
Sub-element: Reporting*

Ecosystem Status and Trends

Research

Assessment and Monitoring

Reporting*

LEVEL

PERFORMANCE MEASURES

Excellent

The Program demonstrates Excellent performance because:
o Reports discuss adaptive management strategies.

o Reports recognize new and emerging issues to be considered in updates or revisions to the CCMP.

Good

The Program demonstrates Good performance because:

o The Program has an environmental progress report that communicates ecosystem status and trends
to the public every three to five years (e.g., "State of the Bay" report, Environmental Report Card,
significant newspaper insert, newsletters, websites, etc.).
o Major reports:

¦ discuss the Program's goals and priorities, indicators in use, ecosystem status and trends, and
maps of study area;

¦ discuss the health of the estuary (i.e., habitat, water quality, and living resources); and

¦ include conceptual models that represent the best understanding of current ecosystem
processes.

Fully Performing

Baseline expectations.

o The Program has an environmental progress report that communicates ecosystem status and trends
to the public on a periodic basis (e.g., "State of the Bay" report, Environmental Report Card,
significant newspaper insert, newsletters, websites, etc.).
o Major reports:

¦ are linked to CCMP actions, goals, priorities, indicators, and monitoring systems;

¦ feature a narrative description of the Program's study area in plain English explaining the
relationship between human activities and impacts on resources; and

¦ are approved by the Management Conference.

Minimally
Performing

The Program does not meet all of the performance measures in the Fully Performing level.

*Refers to Reporting of Ecosystem Status and Trends in the Program study area.

-------
Attachment 3: Annual Funding Guidance Requirement for Leveraged Resources Report

WHAT: As part of CCMP implementation, each NEP works to ensure its long-term financial
sustainability by pursuing leveraging opportunities, i.e., financial or in-kind resources committed
above and beyond the Federal funding provided under the Section 320 grant. Leveraged resources
include resources that are administered by the NEP and those that are not.

FOR MORE INFORMATION: Please contact Craig Alvord at (202) 566-1275.

Definitions of Leveraging Roles and Examples:

Primary role indicates that the NEP played the central role in obtaining leveraged resources. For
example, the NEP:

• convened a workgroup that created a stormwater utility;

• wrote a grant proposal that helped fund the implementation of a CCMP action;

• solicited funds and in-kind support for NEP operations (e.g., office space); or

• provided funds to partners for use as match for grants that fund CCMP implementation.

Significant role indicates that the NEP actively participated in, but did not lead the effort to obtain
additional resources. For example, the NEP:

• wrote parts of a grant proposal or identified lands for habitat restoration;

• identified lands for habitat restoration that were restored using other sources of funding

• directed other non-NEP resources (e.g., SEP money) to projects;

• established a program such as a local land trust that raised money for CCMP implementation;

• convened or actively participated in a stormwater utility workgroup that subsequently raised
funds for CCMP implementation; or

• provided seed money to support a larger project, e.g., a public event.

-------
Attachment 4: Program Evaluation Schedules

2017: Group A (Period covered: July 1. 2011 - June 30. 2016)

Barataria-Terrebonne, Casco Bay, Indian River Lagoon, Massachusetts Bay, Peconic Bay, San Juan Bay,
Tillamook Bay

November 4, 2016

November 11, 2016
December 15, 2016

March 15, 2017

April 14, 2017

May 12, 2017

NEP Directors should determine whether they can volunteer to serve on a PE
team and notify to the PE Coordinator at EPA HQ.

EPA HQ will set up PE teams for Group A NEPs.

Deadline for PE team leader to hold a conference call with members of the
team and the NEP Director to discuss logistics on the preparation and
submission of the PE package among other issues.

Due date for PE submittal package. A total of three electronic copies are
needed (one for each EPA members of the PE team and one for the ex-
officio NEP Director). EPA HQ Coordinator is responsible for making a file
copy.

The PE team leader sends electronic copies of NEP work plans for FYs 12,
13, 14, and 15 to the ex-officio NEP Director.

Deadline for PE team leaders to hold a conference call for the PE team
members to compare notes after reviewing the PE package and submitting
written comments to the PE team leader.

Deadline for PE team leaders to hold a conference call with the NEP Director
and the PE team to discuss additional documentation needs, schedule the on-
site visit, and identify issues that should be addressed during the on-site visit.

May 15 - July 14, 2017 Period for on-site visits.
LETTER DEVELOPMENT* *

2 weeks after visit

4 weeks after visit

6 weeks after visit
9 weeks after visit
September 29, 2017

Period to resolve any concerns rose during the on-site visit.

Deadline for team leader to prepare draft letter documenting the PE team's

findings, recommendations, and rating.

Deadline for PE team (Regional Coordinator and ex-officio NEP Director) to
review and provide comments on draft letter.

Deadline for NEP Director to review and provide comments on draft letter.

Deadline for concurrence and signature by CMB Branch Chief.

All seven PE letters should be signed and sent out to the respective NEPs.

** The schedule for the PE letter development assumes no major issues arise from the PE that requires
extensive consultation.

-------
2018: Group B (Period covered: July 1, 2012 - June 30, 2017)

Tampa Bay, Coastal Bend Bays, Albemarle-Pamlico Sounds, Delaware Inland Bays, Narragansett Bay*,
Sarasota Bay

November 10, 2017

November 17, 2017
December 15, 2017

March 15, 2018

April 13, 2018

May 11,2018

NEP Directors should determine whether they can volunteer to serve on a PE
team and notify to the PE Coordinator at EPA HQ.

EPA HQ will set up PE teams for Group B NEPs.

Deadline for PE team leader to hold a conference call with members of the
team and the NEP Director to discuss logistics on the preparation and
submission of the PE package among other issues.

The PE team leader sends electronic copies of NEP work plans for FYs 13,
14, 15, and 16 to the ex-officio NEP Director.

EXEMPTION:

*Narragansett Bay review period covered (July 1, 2013 - June 30, 2017).
Work plans to be reviewed: FYs 14, 15, and 16.

Deadline for PE team leaders to hold a conference call for the PE team
members to compare notes after reviewing the PE package and submitting
written comments to the PE team leader.

May 14 - July 13, 2018 Period for on-site visits.
LETTER DEVELOPMENT* *

2 weeks after visit

4 weeks after visit

6 weeks after visit
9 weeks after visit

Period to resolve any concerns rose during the on-site visit.

Deadline for team leader to prepare draft letter documenting the PE team's

findings, recommendations, and rating.

Deadline for PE team (Regional Coordinator and ex-officio NEP Director) to
review and provide comments on draft letter.

Deadline for NEP Director to review and provide comments on draft letter.
Deadline for concurrence and signature by CMB Branch Chief.

September 28, 2018 All six PE letters should be signed and sent out to the respective NEPs.

** The schedule for the PE letter development assumes no major issues arise from the PE that requires
extensive consultation.

-------
2019: Group C (Period covered: July 1, 2013 - June 30, 2018)

Buzzards Bay, Partnership for the Delaware Estuary, Long Island Sound, Santa Monica Bay, New York /

New Jersey Harbor, Puget Sound, Maryland Coastal Bays, and Galveston Bay *

November 9, 2018 NEP Directors should determine whether they can volunteer to serve on a PE

team and notify to the PE Coordinator at EPA HQ.

November 16, 2018 EPA HQ will set up PE teams for Group C NEPs.

December 17,2018 Deadline for PE team leader to hold a conference call with members of the

team and the NEP Director to determine if existing reports fully address the
PE questions and identify questions that call for additional documentation
among other issues.

March 15, 2019 Due date for PE submittal package. A total of three electronic copies are

needed (one for each EPA members of the PE team and one for the ex-
officio NEP Director). EPA HQ Coordinator is responsible for making a file
copy.

The PE team leader sends electronic copies of NEP work plans for FYs 14,
15, 16, and 17 to the ex-officio NEP Director.

EXEMPTION:

* Galveston Bay will have PE one year later than would have in the previous
PE process. Therefore, its review period covered (July 1, 2012 - June 30,
2018). Work plans to be reviewed: FYs 13, 14, 15, 16, and 17.

April 19, 2019 Deadline for PE team leaders to hold a conference call for the PE team

members to compare notes after reviewing the PE package and submitting
written comments to the PE team leader.

May 17,2019 Deadline for PE team leaders to hold a conference call with the NEP Director

and the PE team to discuss additional documentation needs, schedule the on-
site visit, and identify issues that should be addressed during the on-site visit.

May 20 - July 19, 2019 Period for on-site visits.

LETTER DEVELOPMENT* *

2 weeks after visit

4 weeks after visit

6 weeks after visit
9 weeks after visit

Period to resolve any concerns rose during the on-site visit.

Deadline for team leader to prepare draft letter documenting the PE team's

findings, recommendations, and rating.

Deadline for PE team (Regional Coordinator and ex-officio NEP Director) to
review and provide comments on draft letter.

Deadline for NEP Director to review and provide comments on draft letter.
Deadline for concurrence and signature by CMB Branch Chief.

September 30, 2019 All eight PE letters should be signed and sent out to the respective NEPs.

** The schedule for the PE letter development assumes no major issues arise from the PE that requires extensive consultation.

-------
2020: Group D (Period covered: July 1, 2014 - June 30, 2019)

Charlotte Harbor, Morro Bay, Columbia River, Barnegat Bay, Piscataqua Region Estuaries, San

Francisco Estuary, Mobile Bay

November 8, 2019 NEP Directors should determine whether they can volunteer to serve on a PE

team and notify to the PE Coordinator at EPA HQ.

November 15, 2019 EPA HQ will set up PE teams for Group D NEPs.

December 15,2019 Deadline for PE team leader to hold a conference call with members of the

team and the NEP Director to determine if existing reports fully address the
PE questions and identify questions that call for additional documentation
among other issues.

March 16, 2020 Due date for PE submittal package. A total of three electronic copies are

needed (one for each EPA members of the PE team and one for the ex-
officio NEP Director). EPA HQ Coordinator is responsible for making a file
copy.

The PE team leader sends electronic copies of NEP work plans for FYs 15,
16, 17, and 18 to the ex-officio NEP Director.

April 13, 2020 Deadline for PE team leaders to hold a conference call for the PE team

members to compare notes after reviewing the PE package and submitting
written comments to the PE team leader.

May 15,2020 Deadline for PE team leaders to hold a conference call with the NEP Director

and the PE team to discuss additional documentation needs, schedule the on-
site visit, and identify issues that should be addressed during the on-site visit.

May 18 - July 17, 2020 Period for on-site visits.

LETTER DEVELOPMENT* *

2 weeks after visit

Period to resolve any concerns rose during the on-site visit.

4 weeks after visit

6 weeks after visit
9 weeks after visit

Deadline for team leader to prepare draft letter documenting the PE team's
findings, recommendations, and rating.

Deadline for PE team (Regional Coordinator and ex-officio NEP Director) to
review and provide comments on draft letter.

Deadline for NEP Director to review and provide comments on draft letter.
Deadline for concurrence and signature by CMB Branch Chief.

September 30, 2020 All seven PE letters should be signed and sent out to the respective NEPs.

** The schedule for the PE letter development assumes no major issues arise from the PE that requires
extensive consultation.

-------
Attachment 5: Responsibilities for the Parties involved in the Program Evaluation Process

EPA HQ:

• HQ Program Evaluation (PE) Program Manager

oversee PE process
set up PE teams
distribute NEP PE guidance
prepare PE finding report

• HQ NEP coordinators should provide assistance to NEPs, such as help interpreting the
PE Guidance and/or feedback on the draft PE package, upon request

• PE team leader

schedule conference calls with members of the team and the NEP Director

send electronic copies of the NEP work plans for the years covered within the PE

cycle to the members of the team

review and comment on the NEP PE package

collect electronic comments from members of the team

coordinate and conduct the on-site visit

draft the PE letter for review and signature by EPA management
send final PE letter to the NEP

EPA Regions:

• provide assistance to NEPs, such as help interpreting the PE Guidance and/or feedback
on the draft PE package, upon request

• PE team member

participate on conference calls

review the NEP PE package

submit electronic comments to the PE team leader

participate in the on-site visit

help draft the PE letter

review, comment and concur with the final draft PE letter

Ex-officio NEP Director:

• participate on conference calls

• review the NEP PE package

• submit written comments to the PE team leader

• participate in the on-site visit

• provide technical transfer assistance to the NEP undergoing the PE

• review and comment on the draft PE letter

NEPs undergoing the PE:

• prepare and submit the PE package to EPA HQ and Regions by March 15

• participate on conference calls

• address the PE team comments and provide any additional information requested by the
PE team

• host the NEP on-site visit

-------