NCEI
NATIONAL CENTER FOR
ENVIRONMENTAL INNOVATION
September 2007
Measuring and Evaluating
Stewardship and
Innovation Programs
Learning from the PART
Summary Report
Evaluation Support Division
Promoting Environmental Results Through Evaluation
-------
Acknowledgements
This report was prepared for the U.S. Environmental Protection Agency's (EPA) National Center for
Environmental Innovation under Contract EP-W-04-023 with Industrial Economics, Incorporated (lEc).
The lEc study team included Dr. Shelley Metzenbaum, Visiting Professor, University of Maryland
School of Public Policy and performance management consultant, and Tracy Dyke-Redmond and Eric
Ruder of lEc. Brewster Boyd and Tim Larson of Ross & Associates Environmental Consulting also
provided research assistance. Katherine Dawes and John Heffelfinger, of EPA's Office of
Environmental Policy Innovation, Evaluation Support Division (ESD), provided direction and technical
advice during the course of the study.
-------
Measuring and Evaluating Stewardship and Innovation Programs: Learning
from the PART
The Environmental Protection Agency's (EPA) National Center for Environmental Innovation
(NCEI) has sponsored a review of how United States (US) federal agencies measure the
performance results of programs engaged in stewardship and innovation. "Measuring and
Evaluating Stewardship and Innovation Programs: Learning from the PART" focuses on the
Program Assessment Rating Tool (PART) assessments reported by the US Office of
Management and Budget (OMB) as part of the federal budget process. The PART assessments
provide insight into how to measure and evaluate performance. Benchmarking PART
assessments may assist EPA in its own performance management efforts for environmental
stewardship and innovation programs.
This effort was completed in two stages. The first stage of research, completed in October 2005
and updated in December 2006, reviews federal stewardship and innovation programs outside of
EPA to analyze the approaches to measurement and evaluation that result in the strongest PART
assessments. The second stage of the research, completed in December 2005 and updated in
December 2006, presents three detailed case studies of successful approaches to demonstrating
results in federal agencies other than EPA. The second stage also analyzes PART summaries for
selected EPA programs that focus on stewardship and innovation, and highlight insights from
other agencies that could be applied in EPA programs. This paper provides an overview of the
studies' conclusions and concisely summarizes findings from the entire body of research.
Overarching Conclusions about How to Measure and Evaluate
Stewardship and Innovation Programs
The findings from this analysis show that stewardship and innovation programs can design
measures and evaluations that meet OMB's criteria for demonstrating results.l This key
challenge emerges, however: stewardship and innovation programs must be well-defined, and
designed based on an explicit program theory (i.e., the assumptions underlying how a program
and its component practices are likely to reduce problems or advance opportunities for gain).
Then the program theory must be framed in the PART in ways that show clear, causal links
between program actions and outcomes. It can be helpful (though not essential) to map the
program theory on a logic model2 and to understand the role of "stewardship" or "innovation" in
the program logic. It can also be helpful to count and characterize the problems a program seeks
to reduce and identify preventable causal factors. Likewise, it can also be helpful to identify the
benefits a program seeks to provide and to identify contributing factors.
Both stewardship and innovation programs face a challenge distinguishing between the ends and
means of their efforts. For example, does a given program treat stewardship as a strategy to
Please see the Appendix for this study's working definitions of stewardship and innovation, as well as a
description of the study's methodology.
2
A logic model is a diagram and text that describes and illustrates the logical (causal) relationships among
program elements and the problem to be solved, thus defining measurements of success.
-------
achieve long-term outcomes (i.e., increasing an ethic of responsibility that may ultimately lead to
improved environmental outcomes), or as an outcome in its own right (i.e., a long-term
commitment to protect a particular resource without explicitly defining outcomes that protection
of that resource leads to)? Stewardship programs have been rated well under the PART, but it is
important to clarify the role that stewardship plays in the program definition. Likewise, what
role does innovation play in a program's logic? Innovation programs that have done well under
PART treat innovation as a means to an end, not as an end in and of itself. Quite a few agencies
undertake innovation programs - both to increase the effectiveness and the cost-effectiveness of
their programs. However, the term "innovation" rarely appears. Instead, the need for innovation
is implied in expectations for outcome targets, cost-effectiveness, and program efficiency.
The PART reviews analyzed in this report suggest that it is vitally important that stewardship
and innovation programs clearly identify and communicate the desired ends of their programs,
both in terms of the short-term and long-term outcomes they are seeking to achieve, and orient
all program activities and performance measurement toward those outcomes. Once the role of
stewardship and/or innovation is defined for a particular program, performance measures must
track progress toward intermediate and long-term outcomes. Well-rated PART programs pay
attention to program efficiency, as well. This may include measures of timeliness or program
results for a given level of investment, for example. It may also include continuing efforts to
achieve the same overall program effectiveness for lower public and/or private costs, even if per
unit metrics are not feasible.
The specific challenge of performance measurement for stewardship programs is that they often
seek to influence the attitudes and behaviors of a broad range of organizations or individuals. In
order to successfully measure progress towards stewardship, it is important to be able to
specifically define the desired attitudes and behaviors the programs are seeking to engender, as
well as the target audience, and articulate how these behaviors are expected to lead to
achievement of long-term outcomes. It is also necessary to be able to clearly identify and
communicate how the programs' activities will reach the intended audience, and the way in
which the activities will motivate this audience to change its attitudes and behaviors.
Stewardship programs can use a variety of strategies to measure progress toward their stated
goals, including:
• Identifying measurable indicators of understanding, attitudes, and behaviors that can be
tracked regularly and consistently over time;
• Establishing a baseline for indicators of understanding, attitudes, and behaviors among the
target audience;
• Routinely collecting data on key indicators of understanding, attitudes, and behaviors in
order to be able to establish trends over time; and
• Comparing understanding, attitudes, and behaviors of the target audience to one or more
comparison groups to establish whether observed changes can be attributed to the program's
activities.
The particular challenge of performance measurement for innovation programs is determining
how to foster innovation that improves outcomes, not innovation for its own sake. Overall, it
appears that readily accessible, consistent, and accurate outcome-focused performance data can
be used as an accountability tool to ensure that flexibility provided to enable innovation is
actually used to improve performance and meet outcome goals (rather than providing an
unintended opportunity for backsliding, which may be a concern in some instances).
-------
As with stewardship programs, innovation programs also need to define who they are seeking to
influence (the target audience), and to set performance goals for each target audience.
Innovative program may use a variety of strategies to foster innovation while maintaining
accountability, including:
• Clearly communicating outcome goals for each target audience, and establishing
expectations about meeting those goals;
• Analyzing and compiling performance data in order to identify performance trends, factors
that may be influencing changes in performance, and best practices for influencing
performance;
• Sharing performance data widely among target audiences, in both packaged and raw forms,
so they can use these data to assess their own performance and identify ways to make
improvements;
• Offering flexibility in how target audiences conduct routine operations, so that if individuals
discover better or more cost effective approaches for meeting outcome goals, they are able to
adopt them;
• Sharing aggregated performance data with the public; and
• Serving as a collector and clearinghouse of performance data from a variety of sources (e.g.,
EPA, states, industry, other countries, and academia) in order to identify and share innovative
and effective practices.
For both stewardship and innovation programs, performance measures can also be used in
program evaluations that are designed to test the assumptions and hypotheses underlying the
program design. This can ultimately help a program demonstrate the extent to which its work
has contributed to positive outcomes, by allowing others to ask, "Without this program's
contributions, would these outcomes have occurred? Or are independent factors, unrelated to the
program, responsible for key outcomes?" Therefore, stewardship and innovation programs
should encourage objective, independent program evaluations that provide supporting evidence
for program theory and validate program results. Short of full-scale evaluations, the ability to
replicate program results in other locations or with other demographic groups or business sectors
can also enhance confidence in a particular program's effectiveness.
-------
Key Findings Regarding Performance Measurement for Stewardship and Innovation
Several themes and insights related to performance measurement for stewardship and innovation
programs emerge from a review of the PART summaries of EPA and other federal agencies.
Some are relevant and instructive regarding how to frame a stewardship or innovation program
and how to demonstrate results in ways that meet OMB's PART criteria. These themes can be
organized under four iterative steps that occur over the lifecycle of a project:
Step 1: Set goals and select
performance measures
Step 2: Track progress
Step 3: Evaluate effectiveness
Step 4: Use results for
program improvement
Each of these topics is discussed below.
Step 1: Setting Goals and Selecting Performance Measures
Stewardship and innovation programs are most effective when they identify program goals and
performance measures that focus on outcomes or the program's results. OMB greatly prefers
outcome and efficiency measures rather than output measurement. The best outcome measures
are those that are directly tied to a program's strategic goals. This finding applies both to
intermediate outcomes such as changes in attitudes and behavior, as well as to long-term
outcomes such as reductions in pollution. While outcome measures should relate to a significant
portion of a program's activities, it is not necessary to measure the results of every program
activity. Directly measured outcomes are preferred. However, creative approaches to measures
(i.e., a mix of intermediate measures and indirect indicators) may be accepted in the PART as
long as the program uses well-established measurement practices and has the support of strong
program theory.
Programs that are new to developing performance measures or in a "start-up" phase deal with
special challenges in a PART review. However, there are strategies that can earn stewardship
and innovation programs credit as their performance metrics evolve over time:
• Aligning performance measures across subprograms and having them all roll up into overall
measures that show progress toward strategic plan goals is a good strategy for long-term
performance measurement.
• A suite of indicators can be well received, and in some cases outputs may be accepted.
- PART credit may be given for outputs relative to building public understanding of
stewardship concepts, but these outputs must have a logical connection and coordination
with other program goals.
- Credit may be given for milestones to develop outcome measures in the future.
• Demonstrating consistent progress toward goals that have only recently been developed is
important, but it is often difficult. In these instances, programs should strive to develop
performance measures for which it is possible to get historical information and show a trend
line that illustrates performance prior to and subsequent to the program's intervention.
-------
• Having program performance measures that contribute directly to strategic plan goals - even
measures that are new and relatively untested - can help demonstrate program value.
Programs must balance having ambitious goals with having realistic ones. Having a few priority
targets, some of which are "stretch" targets, fosters innovation. This applies both to intermediate
targets, such as changes in practices or attitudes, and to long-term targets, such as reductions in
pollution. Stretch targets are best applied to a few, strategic priorities, where breakthrough
progress and innovation is sought. In other areas, more modest targets in line with past rates of
progress are more appropriate. Programs will be better prepared for PART reviews if they
periodically re-evaluate ambitious long-term goals in light of experience and make adjustments if
new information shows that the existing goals are unreasonable. If the goals remain a strategic
priority, adjustments should occur to make goals more realistic, but not eliminate their "stretch"
potential. When setting targets and selecting performance measures, programs may wish to
consider the experience of other programs that have undergone PART reviews:
• Targets for annual improvements can be incremental when coupled with the appropriate
long-term performance measures.
• Targets can be chosen based on intermediate outcomes, as long as programs have identified a
logical and causal link between the intermediate outcomes and the long-term outcomes.
• Annual performance measures can include intermediate as well as long-term outcomes.
• Annual performance measures may include subjective indicators, e.g., relevancy and quality
of research projects, as well as performance indicators. These subjective indicators should be
assessed by an appropriate group (e.g., stakeholders or peer reviewers).
• Not all programs need to be assessed for targets each year; rather, a certain percentage of
programs can be evaluated for targets each year.
• Targets should reflect a limited number of ambitious goals.
• When a target has been met, programs can change targets by picking more ambitious
outcome targets or by selecting a different problem to address with new targets.
• Output targets may be acceptable in very limited cases as annual performance targets, if
supported by program logic and clearly connected to long-term outcomes and strategic goals.
Step 2: Tracking Progress
Measurement approaches that will be most effective for a given program depend on the
program's objectives. For example, when a program's goal is to reduce unwanted events (e.g., oil
spills or accidents), counting and characterizing the events in terms of location, time, and causal
factors is extremely useful for improving outcomes. Alternatively, measuring changes in
attitudes and behaviors of program participants over time is critical to measuring the results of
stewardship programs. Specific strategies for measuring the results of stewardship programs
include:
• Measuring attitude or behavior before and after a program intervention to identify any short-
term changes in attitude or behavior, and to gauge the intervention's initial effects.
• Measuring long-term outcomes using surveys (whether opinion or observational surveys) that
track long-term participants' progress, assess target populations' participation rates, and
measure sustained attitude or behavior changes of participants and the target population.
This is a strong complement to measurement of short-term changes.
-------
Some effective stewardship programs track outcomes among the entire population of individuals
or organizations they are seeking to influence, not just voluntary participants directly involved in
the program. For example, it may be important for stewardship programs to track changes in a
target sector or the economy as a whole, in addition to achievements made by voluntary
participants. These programs often describe their outcomes in the context of the behaviors and
performance of the population as a whole, showing how the program contributed to overall
changes or trends in the population.
Having a centralized, automated system for performance measurement enables virtually
continuous tracking of progress towards goals, facilitates data quality assurances, and allows for
data analysis that leads to program improvements.
Step 3: Evaluating Effectiveness
Stewardship and innovation programs need to be able to measure and demonstrate progress in
terms of outcomes, including changes in attitudes and behaviors. Having a track record of
performance with established outcome targets and measures helps programs establish
performance trends and demonstrate program effectiveness to OMB.
Successful programs analyze and compile performance data in order to identify performance
trends, factors that may be influencing changes in performance (e.g., time of day of incidents, or
changes in technology), and best practices for influencing performance. For example, the Coast
Guard routinely analyzes the performance measurement data it collects from three perspectives.
First, local managers use routinely collected performance data to make risk-based decisions at
the local level. Second, staff at the Coast Guard headquarters study data from across the agency
to identify and understand patterns, pattern variations, and relationships to better understand the
frequency, severity, and causes of problems and to inform the design of effective problem
reducing actions. Finally, the Coast Guard studies exceptions, anomalies, and extreme events
(such as collisions) to better understand what happened, why, and what might be done to avert
the problem in the future.
"Independent" program evaluation (i.e., external to the program being assessed) is useful for
confirming or refuting causal relationships. This is particularly true for stewardship and
innovation programs with program goals of a relatively abstract nature (e.g., "peaceful, friendly
and sympathetic relations between the people of the United States and other countries"). While
OMB prefers evidence of program effectiveness drawn from objective third parties, OMB has
accepted evidence from internal evaluations as long as they are conducted by a separate part of
the agency in accordance with sound evaluation procedures. Key insights from a review of
federal stewardship and innovation programs assessed under the PART include:
• External, independent review can take a variety of forms, including program evaluations,
"Committee of Visitors," external advisory boards, or formal external peer reviews.
• Multi-year program evaluation plans - where programs have been evaluated at least once and
are re-evaluated periodically - can receive credit.
• Multiple program evaluations may be "bundled" to collectively demonstrate results and
receive credit.
• External, independent program evaluations can provide access to professional evaluators who
are trained in this discipline.
• Internal evaluation and review can be used to supplement external evaluation and input.
-------
• Independent review can also be used to:
- Justify intangible or difficult-to-quantify program contributions and validate results.
- Address attribution/causality issues using comparison groups.
- Help draw reasonable conclusions about impact on a program's target population, even
when direct attribution or causality is difficult to measure.
- Validate a research-identified link between intermediate and long-term outcomes.
Programs may be given credit for outcomes achieved by partnering with other entities, but only
if programs can demonstrate their contribution to observed results. Coordination and partnership
with other entities, including other federal agencies, state governments, NGOs, or private entities
can be important to achieve stewardship and innovation programs' goals. However, when
multiple partners work together to achieve similar goals, attribution of outcomes to one particular
program may be difficult and therefore requires careful description of the program approach and
how the partners work together.
• Cross-agency and cross-office work requires evidence of intentional management and
leveraged value, i.e., the program logic should support the use of partnerships and distinguish
unique roles and contributions to the outcome.
• Formal Memorandums of Understanding and Memorandums of Agreement are helpful
evidence for demonstrating intentional coordination with partners.
• Contractors and grantees are not expected to meet agency goals; but are expected to share the
general goal of accelerated implementation by target groups, and to report milestones.
• Reports submitted by key participants (e.g., voluntary program participants or grantees) can
be used to demonstrate commitment towards annual and long-term goals.
• Demonstrating leverage of private funds helps justify program expenses and is highly valued
in PART scoring.
• Being able to measure the degree to which a program leverages state or private funds
supports the program's investments.
Managers should be able to show how their programs' performance and cost-effectiveness
justifies the size and nature of the public investment made. OMB expects program performance
and cost effectiveness to inform program budgets. In particular, OMB looks to see how budget
requests and resources are linked to outputs and/or outcomes and how additional resources
would affect them. OMB also expects programs to be able to demonstrate that they hold federal
program managers and partners accountable for cost, schedule, and performance results.
• Quantifying program costs, public benefits, and relative priority of particular program
investments can help justify the program.
• Cost-effectiveness measures do not have to mimic intermediate and long-term outcomes:
e.g., cost-effectiveness may be gauged in terms of participation rates and leverage of non-
governmental funds, whereas outcome measures focus on improved public health conditions.
• Despite studies showing net benefits for regulations, programs have been rated low if OMB
suspects that more cost-effective, less burdensome standards could be developed.
• Even if the statute(s) require programs to implement less cost-effective practices, OMB has
sometimes downgraded the score.
Step 4: Using Results for Program Improvement
-------
Successful programs look at performance measurement as more than just a reporting
requirement. This is one of the most robust findings in looking at well-rated stewardship and
innovation programs. These programs integrate performance measurement into program
management and have a philosophy of commitment to ongoing program improvements. Highly
rated programs tend to use measurements as an integral part of all aspects and all levels of the
program operations.
Outcome-focused goals, regular tracking of performance, and flexibility to change practices
drives innovation. Well-rated programs did not set innovation goals, per se, but rather set goals
in terms of outcomes, which drove them to innovate. The programs reviewed suggest that
communicating outcome goals and expectations clearly to staff shifted their perspectives about
the best ways to do their jobs and helped them innovate. In addition, allowing staff flexibility to
make program changes produced positive performance results.
Programs can also use performance measurement data to identify innovative approaches. For
example, by scanning the literature for approaches that produced desired changes in key
indicators, programs can identify the most effective strategies for achieving desired outcomes.
In addition, programs can analyze the impacts of different potential innovations on comparison
groups or conduct controlled experiments to identify which innovative approaches are most
effective. EPA could consider using its grants programs to test innovations by awarding funding
to grantees that are willing to test promising innovative approaches. The Agency could then
compare results from grantees using different innovative approaches to each other and to results
from programs using traditional approaches to identify which approaches are most effective and
should be scaled-up. By making sure that grantees are reporting results using comparable
measures, EPA could efficiently evaluate which innovations are most effective.
Federal programs support effective partnerships with states/locals by aiding, informing, and
engaging them in an ongoing search for more effective and more cost-effective ways to achieve
the outcome goals. Federal agencies often need to work through other levels of government or
external partners in order to change attitudes and behaviors and must find effective mechanisms
for doing so. By measuring and analyzing results, and reporting them in an easy-to-understand
manner, programs can not only improve their own effectiveness, but also educate their partners
about which innovative strategies work best.
Innovation programs under the PART face specific challenges in demonstrating that investments
in alternative policy development and research are effective and cost-effective. However, if
innovation is well integrated into program logic, these programs can fare well.
• Credit can be given for staged research of alternative policy approaches, including
identification of lessons learned, and implementation of successful protocols based on those
lessons.
• Credit can be given for reviewing prior research of policy alternatives and partnering with
field organizations, researchers and others (e.g., business, communities) to test intervention
effectiveness, with emphasis on specific outcomes (e.g., air, water, etc. outcomes.)
• New programs are challenged by the need to demonstrate immediate measurable progress. It
can be helpful to test the program on a smaller scale, perhaps limiting initial implementation
to a few locations, measuring whether or not changes occurred as hoped before rolling out
nationwide. It can also help to use performance measures for which it is possible to get
historical information and show a trend line. In programs that are best launched at the
-------
national level, initiate the program with plans and sufficient funding for concurrent outcome
measurement.
-------
Four Illustrations: Stewardship and Innovation Programs with Strong PART Scores
The following four programs were rated as "Effective" or "Moderately Effective", and therefore
may provide models for how to measure and evaluate stewardship and innovation in the PART
process. Moreover, the themes illustrated by these programs are consistent with findings from
the other PART analyses reviewed.
Stewardship of a Natural Resource: Department of Agriculture Forest Legacy Program
The Forest Legacy Program (FLP) is a competitive grant program that provides funding to
protect private forest lands from development. The program is designed to conserve forest areas
that contribute to important scenic, cultural, fish, wildlife, and recreation resources; riparian
areas; and other ecological values. FLP works with private landowners that volunteer to
participate through the use of conservation easements and fee-simple acquisitions of property.
To qualify for FLP, landowners are required to prepare a forest management plan and commit to
maintaining their land as a working forest. FLP embodies the concept of stewardship of a
resource in that it encourages and enables landowners to take responsibility for the
environmental quality and long-term sustainability of their forest lands. OMB rated FLP as
"Moderately Effective" (the second highest rating). Key aspects of FLP that contributed to a
positive PART assessment include:
• A well-defined, unique program purpose that is authorized by federal statute.
• Methods of prioritizing resources according to program goals.
• Demonstration of cost-effectiveness, including the role of leveraging non-federal investment.
• Outcome and efficiency measures that are directly tied to the program's strategic goals.
FLP has been able to define the natural resource it is seeking to protect and what constitutes
stewardship of that resource in simple, quantitative terms that facilitate performance
measurement. In this case, measuring stewardship is made "easier" by the fact that FLP is
charged with protecting only a single type of resource (private forest lands), rather than the
multiplicity of resources for which EPA is seeking to encourage stewardship (air, ecosystems,
energy, land, materials use, and water). Nevertheless, individual programs at EPA may be more
successful in their performance measurement efforts if they can focus on specific, clearly defined
priorities. In addition, FLP shows the importance of demonstrating cost-effectiveness, as well as
progress toward aggressive long-term goals. Finally, the example of FLP suggests that it is
important to measure progress not only in terms of the outcomes from those who choose to
participate (e.g., number of perpetual conservation easements), but also in terms of the total
universe of entities the program is trying to influence (e.g., percentage of private forest acres in
priority areas that are protected from conversion to non-forest uses by FLP).
Stewardship of a Natural Resource: Coast Guard, Marine Environmental Protection
Program
The Marine Environmental Protection Program focuses on preventing and responding to marine
oil spills, chemical spills, nuisance species, and other environmental risks. OMB rated the
Marine Environmental Protection Program as "moderately effective." Key elements of the
program that contributed to a positive PART assessment include:
• Using a risk-based approach to set priorities.
• Allowing managers flexibility to meet outcome goals.
10
-------
• Integrating performance measurement into program management.
The Coast Guard tries to identify the greatest risk of adverse outcomes, and then focuses
prevention efforts in that area. At the field level, when one risk area is successfully addressed,
field managers re-direct attention and resources to the next area of highest risk. This not only
encourages innovation, it also helps the agency allocate its limited resources in more cost-
effective ways. This is important to OMB, which expects programs to regularly measure and be
able to demonstrate their cost effectiveness.
The Marine Environmental Protection Program encourages local managers to analyze trends in
performance measurement data to find out what is causing problems in their jurisdiction. For
example, in Texas, local managers, aided by a data system that allowed time-of-day analysis,
discovered that oil spills were occurring at night while inspections occurred during the day. By
changing the timing of the inspections to coincide with the timing of the spills, they were able to
reduce the incidence of the spills. The Coast Guard also uses performance measurement data to
conduct periodic program evaluations, and to analyze exceptions (such as extreme events), to
understand what factors led to unwanted incidents. OMB values program managers' attention to
regular monitoring of program performance, and also expects programs to undergo periodic
independent, third-party review.
For EPA programs designed to prevent risks or harmful events, the Marine Environmental
Protection program provides a model for using performance measurement not only to
demonstrate a program's effectiveness, but also improve its effectiveness over time.
Stewardship as an Ethic/Change in Attitudes and Beliefs: Department of State, Global
Educational and Cultural Exchanges
The Global Educational and Cultural Exchange (EGA) programs are a collection of competitive
grants that are designed to strengthen international relations and foster mutual understanding
between the U.S. and other countries. The EGA programs are designed to improve
understanding of and support for U.S. policy, encourage and empower moderates, and discredit
intolerance abroad. While EGA programs are not substantively related to the natural
environment, they do illustrate how programs that are designed to foster changes in attitudes and
behaviors can measure progress towards their goals. OMB rated the EGA programs as
"Effective" (the highest rating). Aspects of these programs contributing to a positive PART
assessment include:
• Surveys that track changes in attitudes and behaviors.
• Findings from multiple independent program evaluations that validate program results.
• Well-established performance measures that hold staff and program partners accountable for
results.
The EGA is rigorous in measuring its effectiveness in changing attitudes and beliefs among
program participants. This rigor of the analysis may be directly related to the abstract nature of
the program goals. Since it is very difficult to define "peaceful, friendly and sympathetic
relations between the people of the United States and other countries," EGA has had to work
doubly hard to define quantifiable performance measures that bear a logical relationship to the
program's overall goals. For EPA programs that seek to develop an environmental stewardship
ethic, EPA should seek to define specific aspects of stewardship it seeks to promote among
11
-------
specific audiences, and use surveys to measure the degree to which the target audiences actually
change their attitudes and behaviors over time.
Identifying and Disseminating Innovation: National Highway Transportation Safety
Administration
As part of the U.S. Department of Transportation (DOT), the National Highway Transportation
Safety Administration (NHTSA) is focused on DOT's priority goal of transportation safety:
reducing traffic-linked fatalities and accidents. NHTSA has a grant program which provides
block grants to states for programs to reduce highway deaths and injuries, reduce traffic safety
problems resulting from driving while under the influence of alcohol, and improve data quality
and availability to set traffic safety priorities. OMB rated NHTSA as "Moderately Effective" (the
second highest rating). Key aspects of NHTSA that contributed to a positive PART assessment
include:
• A data-intensive process that is used to develop appropriate performance targets.
• Effective partnership with the states, particularly for data sharing.
• Clear efficiency measures aided by data collected at the federal, state and local level.
• An ongoing process to identify and test innovations, nationally and internationally.
NHTSA selects performance targets based on an analysis of national data, as well as information
it gathers about international practices. It then examines traditional and innovative practices to
find the most effective programs for reducing traffic-linked fatalities. NHTSA continually scans
internationally to find practices and changes in practice that might shed light on the effectiveness
of its interventions. For example, its innovative "Click It or Ticket" campaign evolved from a
project in South Carolina, which was in turn based on a Canadian program. NHTSA tested the
replication of that program with a controlled study, using states as the experimenters with some
implementing the NHTSA-suggested program, a group of states implementing their own
program, and a control group not implementing any changes. To assess program effectiveness,
NHTSA had to measure behavioral change, so it paid for observers to stand at street corners and
count people wearing seat belts before and after the campaign. This allowed NHTSA and the
states to assess the effectiveness of what looked like a promising innovation in an objective
manner. As another example, NHTSA took advantage of natural experiments in the states to
identify effective interventions by funding studies of the severity and costs of motorcycle-linked
accidents before and after the elimination of motor cycle helmet laws. This research provided
evidence that has been introduced into state legislatures and Congress related to debates about
whether or not to require motorcycle riders to wear helmets.
NHTSA offers EPA a model for how to run performance-focused, information-driven regulatory
programs that continually search for innovative practices to improve the effectiveness and cost-
effectiveness of its programs, while maintaining support for existing programs that operate more
effectively and cost-effectively than proposed innovations. NHTSA's mission description may
have relevance for EPA generally, and more specifically for stewardship and innovation.
Specifically, it is not hard to envision how NHTSA's approach to assessing traffic-related health
care and other economic costs in a stewardship and innovation context could be a model for how
EPA addresses "environmentally-related" health care and other economic costs. Also, note that
NHTSA treats education as central to its mission.
12
-------
Example Measures for Stewardship and Innovation
Three types of measures - annual, long-term, and efficiency - are important to the PART. While
both outputs and outcomes are noted throughout the PART analyses, OMB's guidance document
"Examples of Performance Measures" strongly favors outcomes (only three of approximately 60
exemplary measures are outputs). Consequently, the measures listed below focus on annual and
long-term stewardship-related outcome measures. Efficiency measures are also important to
provide evidence in several PART questions. Exemplary efficiency measures consider outcomes
per unit of input. Inputs could be anything that leads to a specific outcome, but most often
relates to costs or number of staff (or staff time). OMB is particularly looking for improvement
in the efficiency measures from year-to-year.
The table below provides example performance measures from stewardship and innovation
programs rated "Effective," "Moderately Effective," or "Adequate," by OMB on a PART
review. Readers should be aware of the following considerations when reviewing the example
performance measures:
• OEM does not rate individual measures, and therefore it is not possible to point to
specific measures that OMB most prefers for stewardship and innovation programs.
However, readers are encouraged to use this list of example measures as a starting point
to thinking about how to measure performance in their own programs, and how to
develop clear, creative, and relevant indicators of performance.
• Measures are specific to the programs reviewed. Other programs would need to develop
performance measures tailored to their own goals and intended outcomes.
• Many of the programs reviewed have continued to refine and improve their measures
over time, in response to OMB feedback and in light of newly available data and greater
experience in performance measurement. Program managers will likely find that there is
no perfect measure or set of measures, but they can work to enhance measurement over
time to better capture progress towards their goals.3
• Selecting good performance measures alone is not sufficient to meet OMB's
requirements. Data must be available to track progress on each measure, and agencies
must collect and analyze that data. Therefore, data availability and the practicality of
collecting data in a timely fashion should also be considered when developing
performance measures.
3 Information on updated measures, and program improvement plans, can be found online at
www.expectmore.gov.
13
-------
Selected Performance Measures for Programs Rated Adequate or Better under PART
Type
Annual
Long-
Term
Efficiency
Program
Department of Agriculture -
Forest Service: Forest
Legacy Program
Department of Agriculture -
Environmental Quality
Incentives Program (EQIP)
Department of Commerce
NOAA Protected Areas
EPA - Nonpoint Source
Grants
EPA - Civil Enforcement
Department of State- Global
Educational and Cultural
Exchanges
Department of Agriculture -
Forest Service: Forest
Legacy Program
Department of Agriculture -
In House Research:
Economic Opportunities for
Producers
EPA - New Chemicals,
Sustainable Futures, Green
Chemicals
EPA - Climate Change
Programs
EPA - Nonpoint Source
Grants
Department of State- Global
Educational and Cultural
Exchanges
Department of Agriculture -
Environmental Quality
Incentives Program (EQIP)
Department of Agriculture -
Forest Service: Forest
Legacy Program
Program Rating
(Assessment Year)
Moderately Effective
(2004)
Moderately Effective
(2004)
Adequate
(2004)
Adequate
(2004)
Adequate
(2004)
Effective
(2004)
Moderately Effective
(2004)
Moderately Effective
(2004)
Moderately Effective
(2003)
Adequate
(2004)
Adequate
(2004)
Effective
(2004)
Moderately Effective
(2004)
Moderately Effective
(2004)
Measure
Acres of land adjustments to conserve the
integrity of undeveloped lands and habitat
quality
Number of acre-feet of water conserved
Increase in the ocean literacy of students
participating in National Marine Sanctuary
Education Programs as compared to the
general student population (at a single point
in time)
Reduction in phosphorous loadings
(millions of pounds)
Change in behavior as measured by the
percentage of entities making improvements
in management practices
Percentage of exchange participants who
initiate or implement a positive change in
their organization or community within five
years of their exchange, based on
knowledge gained from their exchange
Parcelization of forests avoided (parcels
prevented)
Consumption of biomass-based fuel as a
percentage of total transportation fuel
consumption
Cumulative conservation of millions of
BTUs of energy and gallons of water
Million metric tons of carbon equivalent of
greenhouse gas emissions reduced for a
given sector (sectors include building,
industry, and transportation)
Number of water bodies identified by states
(in 2000 or subsequent years) as being
primarily NPS-impaired that are partially or
fully restored
Percent of administrative costs in relation to
program costs (Administrative Efficiency)
Acre-feet of water conserved per million
dollars of financial assistance payments for
application of irrigation water management
practices4
Total program cost or expenditure per acre
protected from conversion
1 This efficiency measure was added in February 2007.
14
-------
Selected Performance Measures for Programs Rated Adequate or Better under PART
Type
Program
EPA - New Chemicals,
Sustainable Futures, Green
Chemicals
EPA - Nonpoint Source
Grants
Department of Interior,
Partners for Fish and
Wildlife
EPA - Brownfields Program
Program Rating
(Assessment Year)
Moderately Effective
(2003)
Adequate
(2004)
Adequate
(2002)
Adequate
(2003)
Measure
Annual number of pre-screened new
chemical alternatives generated through
industry's participation during the earliest
stages of research and development5
Section 319 funds ($ million) expended per
partially or fully restored waterbody6
Acres of wetlands established, reestablished,
rehabilitated, enhanced, or protected per $1
million7
Dollars leveraged at brownfields properties
5 The PART review for this program explains that EPA provides industry with the same tools that it uses to
screen chemicals, so that industry can forecast risk-related issues prior to pre-manufacture notification submission.
This approach is expected to improve efficiency by increasing submissions of chemicals that require less intensive
reviews by EPA. The program is commended for developing a long-term efficiency measure of decreased costs per
chemical reviewed from the Sustainable Futures program (training of companies in EPA's chemical risk screening
tools).
6 The PART review for this program notes that while the program has used an average figure as an
efficiency measure, EPA expects that the costs to restore waterbodies will vary greatly, perhaps by orders of
magnitude in some cases. Factors contributing to such variability include watershed size, climatological and
hydrological factors, and the types of nonpoint sources contributing to the water quality impairment.
Editor's note: As with the efficiency measure for the Nonpoint Source Grants program, costs to establish,
reestablish, rehabilitate, enhance, or protect wetlands may vary based on the type of wetland or type of impairment
to the ecosystem. As performance measures evolve over time, it may be productive to consider segmenting the
measure to account for these different factors that affect costs of restoration.
15
-------
APPENDIX: Working Definitions and Methodology
A Working Definition of Stewardship
For the purposes of this benchmarking study, stewardship is defined as both an ethic and a
practice of responsibility for a resource. Stewardship programs: (1) cultivate an ethic by
increasing a sense of responsibility or fostering key values and attitudes among individuals,
businesses, communities, and government organizations; or (2) are dedicated to achieving
improved stewardship practices. Programs designed to foster an ethic of stewardship include:
attitude change programs, behavioral change programs, voluntary programs, and
communication/information sharing programs. Programs focusing on achieving stewardship
practices include: a variety of approaches to motivate stewardship behaviors (e.g., voluntary and
regulatory approaches), including beyond-compliance behaviors, where key actors in society
(e.g., businesses, individuals, and governments) are encouraged to reduce their impacts beyond
what is required.
A Working Definition of Innovation
For the purpose of this study, innovation is defined as identifying and increasing the adoption of
more effective programs (those that achieve better results) and more cost-effective programs
(those that achieve better results for the same investment of public or private resources or the
same results for less investment). For the purpose of this review, innovation programs are those
that are designed to: a) search among existing practices to find those that are more effective/cost-
effective in achieving agency objectives, b) develop and test new practices that are more
effective/cost-effective in achieving agency objectives, and/or c) promote adoption of more
effective/cost-effective programs.
Benchmarking Methodology
This report focuses primarily on measures of stewardship and innovation that have been
successfully used to track program results. In this case, "success" means that OEM considers
that the program is performing, and specifically that OMB has given the program a rating of
"Effective," "Moderately Effective," or "Adequate."8 In addition, this report reflects insights
from 10 EPA programs selected by the Agency based on the program's relationship to
environmental stewardship, inclusion in EPA's 2003 Innovations Strategy, themes from the
report for the EPA Administrator, "Everyday Choices: Opportunities For Environmental
Stewardship," and endorsements from EPA's Innovation Action Council.
The analysis in this report relied on recent PART summaries available to the public on OMB's
website.9 In addition, interviews with representatives from three programs were an important
source of insights for this report: the Department of State's Global Educational and Cultural
For a more detailed explanation of how OMB defines programs that are performing, see
http://www.whitehouse.gov/omb/expectmore/rating/perform.html.
9
OMB PART website, . originally accessed
on various dates in August and September 2005, and updated in December 2006. The most recent PART reviews
are available at http://www.whitehouse.gov/omb/expectmore/index.html.
16
-------
Exchanges Program, the Department of Transportation's National Highway Transportation
Safety Administration Operations and Research Program, and the Coast Guard's Marine
Environmental Protection Program. We selected these programs based on their relatively high
scores under the PART review and their relevance to the concepts of stewardship and innovation.
This study mainly focuses on programs in the natural resources area (e.g., Agriculture, Energy,
Interior, and EPA), supplemented by information from other federal programs known to engage
in a high level of innovation-driving or stewardship activities, such as policy programs and pilot
programs. The types of programs benchmarked include competitive grant, regulatory, direct
federal, and research and development (R&D) programs. Within R&D programs, the review
considers only those that engaged in programmatic R&D, rather than scientific and technology-
focused R&D. A full list of EPA programs reviewed is included in Exhibit 1, and a list of
programs reviewed from other federal agencies is included in Exhibit 2.
Note that the programs reviewed here are continuing to improve and refine their programs and
the methods they use to measure their results. Therefore, programs may already be addressing
suggested improvements identified in this report. Moreover, the PART process itself has
continued to evolve since the initiation of this study. For example, OMB now makes available
information on Program Improvement Plans, which document actions programs are taking to
respond to OMB recommendations in the PART review. Finally, readers should understand that
as OMB's experience with the PART continues to evolve, the results of PART reviews reflected
in this report may vary from what programs experience in future reviews. Programs are
encouraged to initiate face-to-face dialog with OMB prior to and during the PART review to
help OMB examiners understand agency performance and inform them of supporting evidence.
Exhibit 1: List of EPA Stewardship and Innovation Programs Reviewed
Office of Prevention, Pesticides and Toxic Substances: New Chemicals, Sustainable Futures, Green Chemicals
Office of Enforcement and Compliance Assurance: Civil Enforcement
Office of Water: Nonpoint Source Grants
Office of Solid Waste and Emergency Response: Brownfields
Office of Solid Waste and Emergency Response: RCRA Base Programs, Permits, and Grants
Office of Air and Radiation: Climate Change Programs
Office of Enforcement and Compliance Assurance: Criminal Enforcement
Office of Research and Development: Ecological Research
Office of Research and Development: Pollution Prevention and New Technology
Office of the Administrator: Environmental Education
17
-------
Exhibit 2: List of Programs Reviewed from Other Federal Agencies
Stewardship Programs:
Department of Agriculture, Forest Service: Forest Legacy Program
Department of Agriculture, Environmental Quality Incentives Program (EQIP)
Department of Commerce NOAA Protected Areas
Department of Interior, Habitat Restoration Activities
Department of Interior, National Park Service Natural Resource Stewardship
Department of Interior, Partners for Fish and Wildlife
Department of Labor, OSHA
Department of State, Global Educational and Cultural Exchanges
Department of State, Educational and Cultural Exchange Programs in Near East Asia and South Asia
Homeland Security, Marine Environmental Protection, Coast Guard
Innovation Programs:
Department of Agriculture, In House Research: Economic Opportunities for Producers
Department of Commerce Manufacturing Extension Program
Department of Transportation, National Highway Transportation Safety Administration Operations and Research
Health and Human Services, Agency for Health Care Research and Quality: Patient Safety Program
Health and Human Services, Translating Research into Practice
Housing and Urban Development, Policy Development and Research
Six programs had an initial review not included in this analysis because they: 1) were not related to stewardship
and innovation, 2) related primarily to grants programs or scientific research and development, or 3) had programs
goals and measures that were redundant with the other programs reviewed:
• Department of Agriculture, Soil Survey Program
• Department of Agriculture, Research/Extension Grants: Economic Opportunities for Producers
• Department of Agriculture, APHIS Plant and Animal Health Monitoring Programs
• Department of Energy, Biological and Environmental Research
• Department of Housing and Urban Development Lead Hazard Grants (Lead Hazard Control)
• Housing and Urban Development, Community Planning and Development Initiative
18
------- |