&EPA
United States
Environmental Protection
Agency
Office of Policy
(1807T)
July 2013
EPA-100-F-13-002
Evaluation of the National Water
Program Climate Change Adaptation
Strategy
Fact Sheet
http://www.epa.gov/evaluate
For more information on
completed evaluations at
EPA or the Evaluation
Support Division, visit the
above link.
July 2013
Introduction
The National Water Program Strategy: Response to Climate Change was first published in
2008. OW created one of the first climate change adaptation strategies in the Agency, and is
considered a model for other emerging efforts within the federal government. The 2008
Strategy describes the likely effects that climate change will have on water resources and
implications for EPA's Office of Water (OW) and the National Water Program (NWP). In 2012,
OW developed an updated strategy that describes NWP's long-term goals for sustainably
managing water resources in light of climate change. It is intended as a roadmap to guide
future program work and inform the Agency's annual planning process. In late 2012, OW's
Immediate Office (10), which is responsible for coordinating OW climate change work,
requested support from EPA's Evaluation Support Division (ESD) to conduct a retrospective
evaluation of lessons from the 2008 Strategy, and to develop a prospective measurement
framework to track the progress on the 2012 Strategy.
Evaluation Questions
The evaluation was framed by the following questions:
Evaluating the implementation of the 2008 Strategy:
A How well is climate mainstreamed into OW programs? What are the barriers to
mainstreaming and how might this be better accomplished?
^ What goals, implementation experience, or lessons from the 2008 Strategy could be useful
to guide implementation of the 2012 Strategy?
^ What goals and strategic actions in the 2012 Strategy should EPA headquarters (HQ) and
regional programs prioritize?
Developing a prospective measurement approach for the 2012 Strategy:
^ What is the measurement approach that can be used to measure adaptation progress in
five areas: infrastructure, watersheds and wetlands, ocean and coastal waters, water
quality, and working with tribes?
^ What specific elements need to be applied to the phased approach to tracking progress
outlined in the 2012 Strategy, to make it a robust measurement framework?
^ What, if, any, revisions should EPA make to its baseline data collection process to ensure
that data collected are meaningful and objective?
A How can lessons learned from the evaluation of OW's measurement approach inform
measuring progress in the EPA-wide Adaptation Plan, and inform development of the next
Agency Strategic Plan?
-------
Evaluation Methods
This study draws on multiple data sources to inform the lessons learned related to our research questions. Key sources of
information include:
A Interviews with EPA representatives from 10 and from all four OW program offices and several regional offices, as
well as external organizations. In total the evaluators interviewed 26 EPA staff and 4 non-EPA staff. An EPA focus
group to discuss lessons from the 2008 Strategy and recommendations for implementation of the 2012 Strategy.
Most focus group participants are also among the interviewees discussed above.
A Review of documents related to the NWP climate change strategies and their implementation, including EPA
memoranda and other policy documents.
^ Review of baseline data collected by 10 to assess progress towards the strategic actions included in the 2012
Strategy.
^ Review of existing literature and online publications that address climate change adaptation strategies and activities.
Key Findings
Implementation of the 2008 Strategy
^ How well is climate mainstreamed into OW programs? Overall, evaluation participants felt that the degree of
integration of climate change into the NWP is low but improving. There are a few NWP programs in which substantial
integration progress has been made, but most divisions, offices, and programs are in the early stages.
^ What are the barriers to mainstreaming and how might this be better accomplished? The evaluators
found a range of barriers preventing the integration of climate change considerations into OW's daily operations. Key
barriers include competing priorities, lack of resources, characteristics of climate change as an issue, organizational
and structural aspects of OW, and external influences.
^ What goals, implementation experience, or lessons from the 2008 Strategy could be useful to guide
implementation of the 2012 Strategy? The evaluation team distilled several findings and associated lessons on
the overall effectiveness of the 2008 Strategy, as both a document and a process, for affecting change in and
through the NWP. These include findings related to strategy use, influence, and value; communications and
outreach; management support; staffing and resources; training and capacity building; and measurement.
^ What goals and strategic actions in the 2012 Strategy should EPA headquarters (HQ) and regional
programs prioritize? 10 is currently developing prioritization criteria in order to effectively allocate resources
toward climate related activities. Key criteria under consideration include: urgency, risk, geographic scale,
programmatic scale, and probability of occurrence. The evaluators encountered a broad range of perspectives on
prioritization of the strategic actions described in the 2012 Strategy. Interviewees did not advocate for prioritization of
particular strategic actions; instead, they provided a suite of prioritization schemes for OW consideration.
Prospective Measurement Approach for the 2012 Strategy
^ What is the measurement approach that can be used to measure adaptation progress in five areas:
infrastructure, watersheds and wetlands, ocean and coastal waters, water quality, and working with
tribes? The 2012 Strategy contains 53 strategic actions in the five vision areas and cross-cutting areas of program
support. In the 2012 Strategy, OW adopted a seven-phased approach to adaptive management to measure progress
towards achieving stated goals and strategic actions, which the evaluators agree is a reasonable framework. At this
stage of climate change adaptation, the evaluators suggest that OW programs should be accountable for outputs, and
learn from outcomes. Thus, we suggest that OW apply the seven phases to track progress towards goals articulated
the 2012 Strategy by measuring both outputs and priority outcomes.
^ What specific elements need to be applied to the phased approach to tracking progress outlined in the
2012 Strategy, to make it a robust measurement framework? 10 needs to develop objective criteria for each
phase of adaptive management to facilitate consistent measurement. 10 should identify priority outcomes using a
logic model approach. For each priority outcome, 10 will need to identify if the phase of adaptive management has
been met, based on the specific criteria defined. The evaluation team recommends piloting a measurement approach
with a few priority goals.
A What, if, any, revisions should EPA make to its baseline data collection process to ensure that data
collected are meaningful and objective? The evaluation team documented several data quality and consistency
-------
challenges with baseline data report. Moving forward, we recommend that 10 take steps to adhere to key tenets of
data quality and consistency. Core among these, 10 will need to develop data reporting templates, clear instructions
for reporting, and institute a quality control plan.
^ How can lessons learned from the evaluation of OW's measurement approach inform measuring
progress in the EPA-wide Adaptation Plan, and to inform development of the next Agency Strategic
Plan? Other EPA offices could go through a similar process of using logic models to select priority outcomes, and
developing measures, criteria, and data collection strategies relevant to those priority outcomes. Alternatively, offices
could to continue to use existing measures, and track progress on them in light of climate change. Whatever
approach other offices choose to take in measuring progress on climate change adaptation, they will need to weigh
the merits of measuring outputs vs. outcomes; be transparent about assumptions and weighting; and ensure data
quality. Given the priority actions identified in the Adaptation Plan, and EPA's key strengths and institutional
capabilities, three key areas of EPA expertise may include providing data, decision-support tools, and training to
partners related to climate change adaptation. Further work is needed to define a few specific measures that are
relevant across the agency, plan for data collection associated with these measures, and determine the degree of
EPA's contribution to outcomes achieved.
Recommendations Summary
Reinvigorate NWP management and staff commitment to the Climate Change Strategy.
1. Clarify the purpose of the 2012 Strategy.
2. Clarify commitments and roles associated with the 2012 Strategy.
3. Seek buy-in for the 2012 Strategy among OW management and staff.
Create management practices that keep climate change integration front and center.
4. Schedule regular management-level strategic discussions adaptation and Strategy implementation.
5. Ask key climate change questions relentlessly up and down the management chain.
6. Shift the balance of implementation focus toward "customer service" and learning.
7. Recognize and reward climate change integration progress.
Empower EPA staff and state, tribal, and local partners.
8. Focus education and training support on connecting climate change to practical work.
9. Expand engagement on the strategy with State, Tribal, and local partners.
10. Attract and plan for resources.
Clarify the purpose of measurement and pilot a meaurement approach that includes
outcomes.
11. Seek buy-in for measuring progress on Strategy implementation.
12. Consider adopting an outputs plus priority outcomes measurement approach.
13. Within the measurement approach selected, acknowledge the iterative and evolving nature of this work.
14. Ensure data quality and consistency in collecting measurement data.
Contacts
• Britta Johnson, EPA Evaluation Support Division, Johnson.britta@epa.QOv
Carl Koch, EPA Evaluation Support Division, koch.carl@epa.gov
Joel Corona, EPA Office of Water, corona.ioel@epa.aov
Report Link: http://www.epa.gov/evaluate/pdf/water/eval-water-climate-chanqe-adapt-strateqy.pdf
Date Completed: July 2013
------- |