NCEI
NATIONAL CENTER FOR
ENVIRONMENTAL INNOVATION
Measuring and Evaluating Stewardship
and Innovation Programs
Learning From the PART
A REVIEW
Scope and Purpose of this Study
The Environmental Protection Agency's National
Center for Environmental Innovation sponsored
a review of how federal agencies measure the
performance results of programs engaged in
stewardship and innovation. The goal of this
research is to provide insights that will help EPA
stewardship and innovation programs
understand how they can better measure and
evaluate their performance. This research
draws on best practices of programs from a
variety of federal agencies that have undergone
one or more Program Assessment Rating Tool
(PART) reviews conducted by the Office of
Management and Budget.
September2007
Key Definitions for this Study
Stewardship programs cultivate both an
ethic and a practice of responsibility for
the long-term wellbeing of a resource.
Environmental stewardship, in
particular, is where all those whose
actions affect the environment -
individuals, businesses, communities,
and government organizations -
actively take responsibility to improve
environmental quality and achieve
sustainable results.
Innovation programs identify and
increase the adoption of more effective
programs (those that achieve better
results) and more cost-effective
programs (those that achieve the same
or better results for less investment of
resources) compared to traditional
approaches.
Key Findings from the Analysis
A number of programs across the federal
government are designed to foster stewardship
and innovation. Many successful programs do
not set innovation goals, perse, but rather set
goals in terms of outcomes which drive them to
innovate. These programs have found that
they can be more effective by identifying
program goals and performance measures that
focus on outcomes or the program's results, and then using those goals and measures to
make routine and strategic decisions. In addition, this research finds that:
1) The best outcome measures are those that are directly tied to a program's
strategic goals. This finding applies both to intermediate outcomes such as
changes in attitudes and behavior, as well as to long-term outcomes such as
reductions in pollution. Aligning performance measures across subprograms and
having them all roll up into overall measures that show progress toward strategic
plan goals is a good strategy for long-term performance measurement.
2) Measurement approaches that will be most effective for a given program
depend on the program's objectives. For example, when a program's goal is to
reduce unwanted events (e.g., oil spills or accidents), counting and characterizing
the events in terms of location, time, and causal factors is extremely useful for
improving outcomes. Alternatively, when a program's goal is to change attitudes or
behavior, it is helpful to create a mechanism (e.g., a survey) to assess changes in
those people whose attitudes or behavior the program is trying to influence.
-------
3) Performance measurement works best when measurement data are regularly analyzed
and shared, and when program strategies or practices are continually revised based on the
evidence. For example, innovative programs may compile and analyze performance data from a
variety of sources (e.g., EPA, states, industry, and academic articles) in order to identify performance
trends, factors that may be influencing changes in performance, and best practices for improving
performance. Innovative programs can then share their findings with program staff and target
audiences to help them assess their own performance and identify ways to make improvements. It is
also important to offer flexibility in how program staff and target audiences conduct routine operations,
so that if individuals discover better or more cost effective approaches for meeting the outcome goals
and targets, they are able to adopt them.
4) Having a few priority targets, some of which are "stretch" targets, fosters innovation. This
applies both to intermediate targets, such as changes in attitudes or behaviors, and to long-term
targets, such as reductions in pollution. Stretch targets are best applied to a few, strategic priorities,
where breakthrough progress and innovation is sought. In other areas, more modest targets in line
with past rates of progress are more appropriate. Programs will be better prepared for PART reviews if
they periodically re-evaluate ambitious long-term goals in light of experience and make adjustments if
new information shows that the existing goals are unreasonable.
5) Program evaluation is useful for confirming or refuting causal relationships. This is
particularly true for stewardship and innovation programs with program goals of a relatively abstract
nature, where many contributing factors could potentially cause a desired change in behavior. Program
evaluation can help stewardship and innovation programs demonstrate their impact to key audiences,
such as senior management, Congress, and the public.
Recommendations for Stewardship
and Innovation Programs
The findings from this research suggest that programs
seeking to foster stewardship and innovation should:
•s Develop outcome-focused goals and measures
s Invest in collecting outcome-based performance data
s Integrate performance measures and results data into program
management decisions whenever possible.
•/ Tailor performance measures to reflect program goals and a significant
portion of a program's activities.
s Select a few intermediate and long-term targets for improving program
performance, including a few stretch targets.
s Arrange for periodic evaluation of the program by external, independent
evaluators.
------- |