t — - J
5
%
Office of Inspector General
SEZJ Program Evaluation Report
r^°
PRO^
Design for Objective 8.4
Could Be Improved by
Reorienting Focus on Outcomes
Goal 8 - Sound Science and Innovative Approaches
EPA will develop and apply the best available science for
addressing current and future environmental hazards, as well as
new approaches toward improving environmental protection.
Objective 8.4 - Pollution Prevention
and New Technologies
By 2006, develop and verify improved tools, methodologies, and
technologies for modeling, measuring, characterizing,
preventing, controlling, and cleaning up contaminants
associated with high priority human health and environmental
problems.
Report No. 2002-P-000002
November 21, 2001
-------
Inspector General Office
Conducting the Review:
Office of Program Evaluation
Program Office Involved:
Office of Inspector General
Evaluation Team:
Office of Research and Development
Manju Gupta
James Hatfield
Dale Pahl
Kevin Ross
Terry Simpson
-------
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, DC 20480
OFFICE OF THE
INSPECTOR GENERAL
November 21, 2001
MEMORANDUM
SUBJECT: Program Evaluation Report;
Design for Objective 8.4 Could Be Improved by Reorienting Focus on Outcomes
Report No. 2OO2-P-0OGO2
FROM: Kwai-Cheung Chan
Assistant Inspector General ' }
Office of Program Evaluation
TO: Henry L. Longest II
Acting Assistant Administrator
Office of Research and Development
We are pleased to provide the results of our program evaluation of the research portion of
the Environmental Protection Agency's Goal 8 program, with special emphasis on the design of
Objective 8,4. This pilot evaluation was conducted by the Office of Inspector General, in
collaboration with the Office of Research and Development (ORD). The report presents our
observations regarding our primary evaluation objective - to determine whether the designs of
the Goal 8 and, more specifically, Objective 8.4 programs were consistent with the intent of the
Government Performance and Results Act, i.e., conducive to achieving outcome-related results.
As agreed to by both parties at the outset, the pilot was to determine whether program
evaluation techniques are appropriate for measuring progress in accomplishing Government
Performance and Results Act goals, and to document and evaluate the program, designs for
Goal 8 and Objective 8.4.
We held several meetings with ORD managers to discuss the pilot, from an initial
meeting in July 2000 through the preparation of this report. We provided results in briefings and
a draft report to ORD managers. We have incorporated the response to the draft report in the
main body of this report, as appropriate. ORD's comments are attached as Exhibit 3.
We wish to express our appreciation for the cooperation extended by ORD during the
pilot. The ORD managers, scientists, and engineers responsible for research in pollution
prevention and new technologies were uniformly cooperative and pleased to discuss their
research. Their cooperation contributed significantly to the success of this pilot evaluation.
If you or your staff have any questions regarding this report, please contact me at f2u2)
260-2347.
-------
-------
Executive Summary
Purpose of Pilot Evaluation
The Environmental Protection Agency's (EPA's) Office of Inspector General, in
collaboration with EPA's Office of Research and Development (ORD), conducted
this pilot evaluation. As agreed to by both parties, this pilot sought to determine
whether program evaluation techniques are appropriate for measuring the progress
in accomplishing Government Performance and Results Act goals. This pilot
specifically sought to document and evaluate the program designs for Goal 8, with
a focus on Objective 8.4 (Pollution Prevention and New Technologies) research.
We concluded that the evaluation process was appropriate. The approach
provided a better understanding of the programs, answered key questions, and
provided a partnership approach that was beneficial in developing meaningful
observations about the designs for Goal 8 and Objective 8.4. We provided results
in briefings and a draft report to ORD managers.
Results of Evaluation
This report represents our observations on whether the designs for Objective 8.4
and the research portions of Goal 8 were consistent with the intent of the
Government Performance and Results Act, i.e., conducive to achieving outcome-
related results. As part of this evaluation, we also evaluated whether the program
design was clear and how the performance goals were being achieved.
The 1997 EPA Strategic Plan recognizes that Goal 8 research is important, and
ORD is developing multi-year plans for some research areas. Each multi-year plan
for research details Annual Performance Goals and Measures designed to lead to
achievement of long-term goals. We found that ORD's Goal 8 research priorities
are consistent with National Research Council recommendations for core research,
but identified opportunities for improvement that may facilitate EPA's
accomplishing its mission of improving the environment and protecting human
health. Specifically:
Focus Can Be Placed on Outcomes. ORD has developed a multi-year plan
for Objective 8.4 that links the research strategy to major milestones and goals.
The Annual Performance Goals and Measures focus primarily on outputs (such
as developing new methods, models, and tools) rather than on achieving
outcomes (the effects resulting from the acceptance and use of these new tools
and technologies). Placing greater focus on potential outcomes could result in
more valuable benefits being realized.
-------
Technology Transfer Can Be Expanded. Success for Objective 8.4 and,
ultimately, Goal 8 depends on effective transfer of technology so that
customers can understand and use research results to accomplish desired
environmental and health benefits. Technology transfer currently generally
consists of publication in journals and presentation of papers at scientific
conferences. While these types of activities are customary and necessary,
further actions should be taken to more effectively transfer research products
to customers so that successful application of products occurs.
Program Design Can Be More Consistent. ORD created several documents
between 1997 and 2001 that describe strategies and plans for accomplishing
the Pollution Prevention and New Technologies research goal. A comparison
of these documents and the implemented program showed program design
inconsistencies that can limit successful implementation. Resolving these
inconsistencies may enhance ORD's ability to achieve its objectives.
Planning System Can Be Enhanced. ORD has developed a research
planning system to establish priorities that links the "resources" for research
with ORD "activities." There is opportunity to enhance the planning system by
linking each research area to outcomes; providing more detailed
documentation for multi-year research; and collecting and providing actual
performance data in a time frame so it can be used for diagnosing and
remedying problems with implementation.
Two-Tiered Strategy for Goal 8 Can Be Made Clearer. The 1997 EPA
Strategic Plan provides a two-tiered strategy for Goal 8: develop and apply the
best available science, as well as new approaches for improving environmental
protection. While both aspects are important, the Goal 8 description does not
provide an approach for integrating the two-tiered strategy toward one goal.
Opportunity exists to provide a clearer explanation of how the two-tiered
strategy is intended to work in order to accomplish EPA's environmental
outcomes.
Externalities Can Be Better Addressed. Agency managers need to
recognize externalities (factors beyond a Program's control that can help or
hinder its success) and their potential impact on program outcomes. Planning
documents do not take into sufficient account the role played by externalities
for Goal 8 or Objective 8.4 research. EPA and ORD can benefit by
anticipating externalities such as budget decisions, as well as by coordinating
with other organizations doing environmental research so that efficiencies can
be leveraged.
ORD has already taken several actions based on our evaluation to improve Goal 8
and Objective 8.4 programs.
ii
-------
Table of Contents
Executive Summary i
Introduction 1
Purpose of Program Evaluation Review 1
Background 1
Scope and Methodology 2
Summary of Observations 5
Focus Can Be Placed on Outcomes 5
Technology Transfer Can Be Expanded 6
Program Design Can Be More Consistent 7
Planning System Can Be Enhanced 8
Two-Tiered Strategy for Goal 8 Can Be Made Clearer 10
Externalities Can Be Better Addressed 12
Exhibits 13
1: Logic Models for Goal 8 and Objective 8.4 13
2: Office of Research and Development Actions Taken
in Response to Evaluation 15
3: Office of Research and Development Response
to the Draft Report 17
-------
-------
Introduction
Purpose of Program Evaluation Review
A new strategic direction for the Environmental Protection Agency's (EPA's)
Office of Inspector General is performing program evaluations to provide
Congress and EPA with best practices, analyses, and recommendations to address
serious management challenges, accomplish environmental objectives, and achieve
Government Performance and
Results Act (GPRA) goals.
OIG began pilot-scale program
evaluations in Fiscal Year 2000.
EPA's Office of Research and
Development (ORD) volunteered
for this pilot. Our objective was to
document and evaluate the design
of ORD's portion of EPA Strategic
Plan's Goal 8 program, with
special emphasis on Objective 8.4,
to determine whether program
evaluation techniques are
appropriate for measuring the
progress in accomplishing GPRA
goals.
Background
GPRA, enacted in 1993, challenges Federal agencies and managers to restructure
their programs around answers to several basic questions: What is our mission?
What are our goals and how will we achieve them? How can we measure our
performance? In essence, the Act seeks to shift Federal planning, management,
and decision making away from a traditional focus on resources and activities to a
focus on results and the outcomes of those results on the lives of citizens. To
make the government more outcome oriented, the Act requires agencies to: specify
the outcomes they are trying to achieve, identify specific goals and measures, and
evaluate performance against those goals and measures.
A centerpiece of the Act's outcome-oriented focus is that Agencies prepare
strategic plans that reach at least 5 years into the future and describe the critical
linkages between current activities and long-terms goals. In essence, an agency's
strategic plan is its road map for achieving long-term outcomes.
GPRA and Inspectors General
"The nation's Inspectors General have the
capability individually and collectively to review
and analyze what the Federal government now
does and ask such questions as: How can we
improve the organization and management of
the Federal government in order to maximize
productivity, effectiveness, and accountability
for performance results .... I'm encouraged that
many Inspectors General are taking the lead to
ensure that the Results Act is fully implemented
throughout the Federal government."
S Senator Fred Thompson, Chairman,
Senate Committee on Government Affairs,
Journal of Public Inquiry, Fall/Winter 2000
1
-------
The 1997 EPA Strategic Plan describes the agency's strategy for the 5-year period
of Fiscal Years 1997-2001 and includes 10 GPRA goals. Goal 8 is as follows:
Goal 8 - Sound Science and Innovative Approaches
1 will develop and apply the best available science for addressing
current and Inlin e environmental hazards, as w ell as new approaches
toward improving environmental protection.
EPA's strategy for Goal 8 identifies three performance measures: quality,
relevance, and impact on environmental decision making. Goal 8 includes
10 objectives. ORD implements core research in Objectives 8.1 through 8.4 and is
responsible for a significant part of EPA's Goal 8 resources. ORD is also
responsible for conducting research under Goals 1 through 7. We focused our
review on Objective 8.4:
Objective 8.4 - Pollution Prevention
and New Technologies
/>V develop ami verify improved tools, methodologies, and
technologies for modeling, measuring, characterizing, prevailing,
controlling, and cleaning up contaminants associated ivilli high
priority human health anil environmental problems.
The Pollution Prevention Act of 1990 established certain requirements for
pollution prevention and a national policy that pollution should be prevented or
reduced at its source. Within EPA, the Office of Prevention, Pesticides, and Toxic
Substances leads the Agency's response to requirements of the Act. Within its
GPRA structure, EPA established research for pollution prevention as
Objective 8.4 under Goal 8. ORD is responsible for developing, implementing,
and managing pollution prevention research. Objective 8.4 research is
accomplished through a combination of in-house and extramural mechanisms.
Scope and Methodology
We documented and evaluated the overall design of GPRA Goal 8, and evaluated
in more detail the design of Objective 8.4 and its program components.
Objective 8.4 was selected based on ORD input that indicated it was a well-
documented program and would be a good choice for program design evaluation.
To help determine whether EPA's designs for Objective 8.4 research are consistent
with Congressional intent, we developed questions from the U.S.
2
-------
General Accounting Office
recommendations to Congress
about how GPRA strategic
plans can be improved to
better support Congressional
and agency decision making.
Those questions are shown in
the box at right. This report
presents only the most
significant observations related
to the questions.
To interpret information, we
relied extensively on program
design and evaluation theory,
literature, and techniques.
We reviewed EPA's Goal 8
strategy, and focused on
research in Objective 8.4 funded through EPA's Science and Technology
appropriation and managed by ORD. To develop answers to the questions raised,
we gathered and interpreted information from the following:
Interviews and Workshops: We met with various program managers,
particularly those involved with Objective 8.4 research. We conducted several
workshops with ORD managers to address various aspects of the Objective 8.4
research program.
Documents: We examined a large number of documents that describe EPA's
mission as well as GPRA planning, goals, intended results, and program logic.
We used the 1997 EPA Strategic Plan as a primary source of information
through which EPA communicated to Congress and the public about GPRA
strategies, goals, and management within EPA for Fiscal Years 1997-2001.
We reviewed various EPA planning and budget documents, ORD strategic
plans, and reports from external agencies such as the U.S. General Accounting
Office and the National Research Council.
We did not consider other customers or stakeholders beyond ORD who may fund,
support, or participate in programs designed to achieve similar outcomes or goals.
We used logic models1 to identify and illustrate Goal 8 and Objective 8.4 research
program designs, and to discuss the designs with ORD managers. Figure 1 that
follows illustrates a generic logic model for an environmental program. The figure
depicts the elements of program design, and the arrows between these elements
1 "Logic Models: A Tool for Telling Your Program's Performance Story," McLaughlin, J.A.; and
Evaluation and Program Planning 22 (1999) 65-72, Jordan, G.B.
Questions for Evaluating the Design of
EPA's Goal 8 Research
• Is the GPRA structure (i.e., the objective
statement and performance measures) for
Objective 8.4 focused on outcomes?
• Are the program designs for Goal 8 and Objective
8.4 clear? Are the elements of the design, and the
links between these elements, clear?
• How are the performance goals to be achieved?
Are the strategies logically linked to the goals and
the day-to-day activities of the managers and
staff? Do the managers monitor and assess
external factors that can help or hinder achieving
outcomes?
3
-------
represent critical links in the program's design and functioning. In this figure, the
program design theory is presented in the narrative text within the rectangles.
When used this way, logic models can facilitate communication about the elements
of a program's design, the links between these elements, explicit and implicit
assumptions about program operation, and expectations about results and
accountability. Exhibit 1 contains logic models we developed for the Goal 8 and
Objective 8.4 programs.
Figure 1: A General Logic Model for an Environmental Program
Resources
Human and
financial
resources, and
other inputs
needed to
support the
program.
Activities
The specific
actions and
tasks needed to
produce the
program's
outputs.
Outputs
Products,
goods, and
services
provided to the
program's
customers.
Customers
Reached
Users of the
program's
outputs.
Short-Term
Outcomes
Changes in
customer
knowledge,
attitudes, skills,
and aspirations
(KASA)...
...followed by
changes in
customer actions
intermediate
Outcomes
Environmental
changes resulting
from customer
. actions ...
... such as
reduced
emissions,
improved ambient
conditions, or
reduced exposure
to contaminants.
Long-Term
Outcomes
Desired program
impacts...
... such as
improved
environmental
health or restored
ecosystems.
Externalities
These are factors outside of the program's control that may influence (help or hinder) the success of the
program and the accomplishment of its results.
Although this review is a program evaluation and not an audit, we followed
applicable Government Auditing Standards to the extent practical - particularly in
regard to qualifications, due professional care, quality control, planning,
supervision, evidence, and reporting. Because of significant and complex issues on
the extent to which environmental research can be managed under GPRA, the
Office of Inspector General determined that, for this pilot program evaluation, a
deviation from the Government Auditing Standards about organizational
independence was necessary. The team conducting this pilot included
professionals from ORD as well as the Office of Inspector General. The pilot team
leader was detailed to the Office of Inspector General from ORD for the duration
of the pilot. Before this detail, the team leader did not have organizational or
research responsibilities associated with ORD research in Objective 8.4.
4
-------
Summary of Observations
The 1997 EPA Strategic Plan recognizes that Goal 8 research has an important
role in EPA's achieving the long-term outcomes of protecting human and
environmental health (see box). ORD is conducting a pilot program to develop
multi-year plans for some of its most
important research areas. Each Multi-
Year Plan for research details Annual
Performance Goals and Measures
designed to lead to achievement of
long-term goals.
Based on our evaluation, we observed
that ORD has made significant
progress in its GPRA efforts but can
place more focus on outcomes. We
also noted various ways in which ORD
can make its Objective 8.4 program
design clearer by expanding on ways it transfers technology, being more consistent
with its design, and making planning system improvements. Furthermore,
performance goals for Goal 8 can be better achieved by making the two-tiered
strategy clearer and taking steps to anticipate externalities that can impact on
outcomes.
Value of Research
EPA's research program will measurably
increase our understanding of
environmental processes and our capability
to respond to and solve environmental
problems .... Research will lead to
greater certainty in assessing and
comparing environmental risks.
- 1997 EPA Strategic Plan, page 53
Focus Can Be Placed on Outcomes
Analysis of the Annual Performance Goals and Measures indicated that ORD
focuses primarily on outputs, such as developing new methods, models, tools, and
technologies. Placing greater focus on potential outcomes rather than the steps
taken to achieve outcomes could result in more valuable benefits being realized.
For example, one Objective 8.4 Annual Performance Measure is described as
"update the Coating Alternatives Guide (CAGE) to show the chemical constituents
of paint formulation and identify the most significant amounts of VOCs (volatile
organic compounds) and HAPs (hazardous air pollutants) in commercially
marketed paint formulations." Focusing this measure more on outcomes being
sought would enable managers to better assess how successfully the actions
accomplish the desired results.
Research outputs may result in short-term outcomes, such as reduced scientific
uncertainty and improved environmental decision making, if those outputs are
5
-------
transferred effectively to customers. These customers (which include EPA, states,
and industry) and the desired short-term outcomes are represented by the three
shaded boxes in the middle of the logic model in Exhibit 1. Further, customer
application of the outputs may result in intermediate outcomes, such as reduced
emissions or improved environmental quality, which will enable EPA to achieve its
long-term outcomes of improved environmental and human health. By improving
the Objective 8.4 program design to include performance measures related to
short-term outcomes, ORD can measure and know the extent to which customers
use their research products and, consequently, determine the research's impact on
intermediate- and long-term environmental outcomes.
Objective 8.4 program managers agreed research program designs generally do not
focus on measuring the application of research products by customers to achieve
outcomes. Program managers acknowledged that using logic model concepts to
perform a matrix analysis similar to that found in Exhibit 1 would be helpful in
understanding the critical linkages in managing for results.
i
Potential Area
for Improvement
Technology Transfer Can Be Expanded
The success of Objective 8.4 and, ultimately, Goal 8, depends on effective transfer
of technology and the communication of research in proper formats that are
accessible to and usable by a variety of audiences (see box). Both internal and
external customers use ORD's research products. Internal customers include
decision makers and regulators in EPA
Program Offices and Regions. External
customers and stakeholders include state and
local government regulators, industries, other
researchers, and the general public.
Review of the research area descriptions for
Objectives 8.1 through 8.4, as well as
interviews with ORD managers, indicated
that EPA technology transfer activities
consist primarily of:
Impact on Decision Making
The performance goal is for EPA's
research organization to transfer
information, findings, and results
effectively to users, partners, and
the public.
1997 EPA Strategic Plan, page 56
Publication of methodologies, technologies, and models in technical and trade
journals and on internet web sites.
Presentation of papers at scientific conferences.
These types of technology transfer are customary and necessary practices for
research organizations and should be continued. However, further actions should
be taken to more effectively transfer research products to internal and external
6
-------
customers so that successful application of products occurs. This conclusion was
also reached in a National Academy of Sciences study, Strengthening Science at
EPA, which states that EPA needs to "substantially increase its efforts to
disseminate actively ORD's research products and . . . assist others inside and
outside the agency in applying them."
The 1997 EPA Strategic Plan, as well as ORD's research planning documents, do
not identify a systematic approach for transferring research products to customers,
but such an approach is needed. Effective outreach by ORD to customers and
stakeholders is a critical element in the designs for Goal 8 and Objective 8.4
programs, as shown in Exhibit 1. Program documentation should specifically
describe the resources, activities, outputs, and performance measures related to
technology transfer and effective customer outreach.
Furthermore, organizational responsibility for technology transfer should be clearly
outlined. Objective 8.4 program managers said that although there is a
Technology Transfer and Support Division within ORD funded by Objective 8.4,
managers involved with Objective 8.4 are not clear on the level of support this
division can provide for the transfer of their research products. Program managers
said this Division provides technology transfer support to all of ORD and
estimated there is approximately one full-time equivalent of technology transfer
support for Objective 8.4. Also, ORD's general workforce consists of highly
trained scientists, engineers, and researchers, whereas the function of customer
outreach may benefit from the use of disciplines in addition to scientific research.
Outreach and technology transfer activities are key to ensuring that customers
understand ORD's research products and are able to apply them. These activities
represent a critical link between the "outputs" and "customers" elements in the
logic model. If this link does not function properly, the desired changes in
customer knowledge and behavior (i.e., short-term outcomes) leading to
improvement in environmental conditions (i.e., intermediate and long-term
outcomes) may not occur or occur to the level desired.
m
I
Potential Area
for Improvement
Program Design Can Be More Consistent
The long-term goal of Objective 8.4 is to develop and verify new tools,
methodologies, and technologies for modeling, measuring, characterizing,
preventing, controlling, and cleaning up contaminants with high priority human
health and environmental problems by 2006. ORD created several documents
between 1997 and 2001 that describe strategies and plans for accomplishing this
Pollution Prevention and New Technologies research goal. We compared the
program design as described in the ORD documents and the program as it is
7
-------
actually implemented, and noted that resolving certain inconsistencies may result in
improvements. Examples of inconsistencies include:
• The Pollution Prevention Research Strategy and the Fiscal Year 2002
Sub-Objective Summary identify an "active" risk assessment research area,
with allotted resources, but this research area has not been implemented.
One research area described in the Objective 8.4 Sub-Objective Summary -
the Persistent Bio-accumulative Toxics Initiative - was not mentioned in the
Pollution Prevention Research Strategy or the multi-year plan. Further, the
Initiative's research objectives were not consistent with the strategic goals
identified in the Pollution Prevention Research Strategy.
Although 30 to 40 staff responsible for technology transfer activities were
supported by the Objective 8.4 budget, their activities and outputs were not
described in any Objective 8.4 strategies or plans. Furthermore, program
managers estimated that only one full-time equivalent was devoted to
providing support to research managers in Objective 8.4.
Collectively, such inconsistencies can limit the ability of ORD managers to
understand and implement the elements of the Objective 8.4 program. Resolution
of these inconsistencies may enhance ORD's ability to achieve its objectives.
m
I
Potential Area
for Improvement
Planning System Can Be Enhanced
The multi-media and multi-disciplinary nature of EPA's Objective 8.4 research
requires ORD to design, manage, and integrate many individual research projects
over multiple years. A major
challenge for EPA's core
research program is determining
which research issues are
important enough to support, as
noted by the National Research
Council (see box). ORD has
developed a research planning
system to establish priorities to
help meet this challenge. This
system represents a critical link
in the Objective 8.4 program
design because the priorities it
identifies are used in making
resource decisions. ORD's
research planning system links
EPA's Environmental Research
Challenge
Clearly, EPA cannot conduct or sponsor research
on every issue of concern to the public, the
Congress, or even its own program offices ....
Because EPA's task of protecting the environment
and human health is so vast and difficult, and
because resources to undertake the necessary
research are very limited, choices will have to be
made among many worthwhile projects.
National Research Council
Building a Foundation for Sound
Environmental Decisions, pages 10, 60.
8
-------
the "resources" for research with the "activities" of EPA's research organization,
as illustrated by the arrow in Exhibit 1 (Figure 2).
ORD program managers indicated that ORD requests input from agency-wide
customers and stakeholders in identifying potential research priorities. Further,
ORD involves senior agency scientists from program and regional offices on
Research Coordination Teams, which recommend research priorities and establish
new priority rankings each year. It may take several years to complete the
individual research projects in a research area, and from 5 to 10 years to complete
all the research areas needed to achieve an objective. With this in mind, we
concluded that EPA's planning system could better address how research should
contribute to achieving environmental outcomes. Specifically:
~ ORD's research planning system focuses on the resources, activities, and
outputs associated with each research area without also addressing how the
research relates to achieving environmental outcomes. When annual priorities
are established by the Research Coordination Teams for each research area,
important GPRA information associated with that year's research - such as
performance goals, measures, and progress to achieve outcomes - is not
available because performance data are collected after the annual research
priority setting is completed. Consequently, ORD managers responsible for
implementing Objective 8.4 research programs and projects may not have
information essential for diagnosing and remedying problems with
implementation that may be needed to achieve GPRA results.
~ Currently, ORD is conducting a pilot program to develop multi-year plans for
some of its most important research areas. According to ORD planning
guidance, the multi-year plans are draft documents that "link the research
strategy and research plans to show how ORD conducts research in an
integrated fashion to reach major milestones and end points." Review of the
Draft Multi-Year Plan for Objective 8.4 disclosed that it includes activities,
outputs, and annual performance measures and goals, but does not sufficiently
consider outcomes. While the draft multi-year plan demonstrates how ORD
has taken an important step toward linking research strategies to research
plans, opportunities exist to improve the multi-year planning system by
orienting the multi-year plan toward outcomes.
~ It is difficult to measure the annual accomplishments of scientific research,
since research typically takes several years to complete. However, it is
important that plans for multi-year research projects include milestones that
can be used to periodically measure and report on progress. The Agency has
developed a reporting mechanism that rolls up the accomplishments for all
Strategic Goals, but there are restrictions on how much information can be
included for research projects still in progress. ORD managers stated it is
difficult to communicate on a year-to-year basis the progress ORD is making
toward achieving outcomes.
9
-------
There is opportunity to enhance the planning system by linking each research area
to environmental outcomes; providing more detailed documentation (including
milestones) for multi-year research; and collecting and providing actual
performance data in a time frame so it can be used by the managers in diagnosing
and correcting problems in the next annual planning cycle.
The 1997 EPA Strategic Plan indicates that the strategy for Goal 8 is two-tiered,
as discussed in the following box:
It is clear that, in the future, environmental problems will be dealt with in a different manner
than they are today. In moving towards this future, EPA has adopted a two-tiered strategy.
On one level, EPA is working to strengthen those features of the current system that have
been proven to be effective. At the second level, EPA is designing and testing
fundamentally new tools and approaches that take advantage of new scientific knowledge
and technological advancements, a growing ethic of environmental stewardship, the need to
cut waste, and increase efficiency and similar opportunities as they arise in society.
- 1997 EPA Strategic Plan, page 53
The description of the two-tiered strategy for Goal 8 includes the development and
application of:
/ Best available science.
/ New approaches for improving environmental protection.
According to the FY 2001 Annual Performance Plan and Congressional
Justification, Objectives 8.1 through 8.4 address research related to best available
science while Objectives 8.6 though 8.10 address new approaches (Objective 8.5 is
reserved for future use). Specifically:
Potential Area
for Improvement
Two-Tiered Strategy for Goal 8
Can Be Made Clearer
Two-Tiered Strategy
10
-------
Objective
Objective Title
Best
Available
Science
8.1
Ecosystem assessment and restoration
8.2
Human health risk assessment
8.3
Detecting emerging risk issues
8.4
Pollution prevention and new technologies
iV.V.V.V.
8.5
(Reserved for future use)
New
Approaches
8.6
Increase integrated holistic partnership approaches
8.7
Increase sector based approaches
8.8
Regional enhancement of ability to quantify envir. outcomes
8.9
Science Advisory Board peer reviews
8.10
Incorporate innovative approaches to envir. management
Responsibility for the scientific research-related Objectives 8.1 through 8.4 rests
with ORD, while responsibility for Objectives 8.6 through 8.10 rests with the
Office of the EPA Administrator.
In addition, our analysis indicated that Goal 8 activities are funded through
different Congressional appropriations:
/ Objectives 8.1 through 8.4 are funded primarily through EPA's Science and
Technology appropriation.
/ Objectives 8.6 through 8.10 are funded primarily through EPA's
Environmental Program and Management appropriation.
Goal 8 resources identified for Fiscal Year 2001 totaled $328.8 million, of which
about $250.7 million (76 percent) was identified for Goal 8 research in
Objectives 8.1 through 8.4. This budget information indicates that ORD, with its
Objectives related to scientific research, has significant responsibility for achieving
Goal 8 results.
Additional information could be added to the Goal 8 description to enable
customers or stakeholders to better understand how this two-tiered strategy should
be combined to accomplish the long-term environmental outcomes of the goal.
The multiple levels of responsibility and sources of funding adds to the complexity.
ORD managers noted that, as currently designed, it will be difficult to determine
whether EPA is achieving the intended Goal 8 outcomes when Objectives 8.1
through 8.4 are managed and funded by ORD while Objectives 8.6 through 8.10
are managed and funded by other EPA offices. Opportunity exists to provide a
clearer explanation of how the two-tiered strategy is intended to work in order to
accomplish EPA's environmental outcomes
11
-------
Potential Area
for Improvement
Externalities Can Be Better Addressed
The 1997 EPA Strategic Plan does not sufficiently consider the impact of
externalities that can affect the success of Goal 8 research. Externalities are
factors beyond a program's control that can help or hinder its success. In advising
Congress about GPRA strategies, the U.S. General Accounting Office emphasizes
in its report, Agencies' Strategic Plans Under GPRA: Key Questions to Facilitate
Congressional Review, that it is important for program managers to recognize
externalities and their potential impact on program outcomes.
We identified a number of externalities that may limit ORD's ability to accomplish
Objective 8.4 research outcomes and results. These include:
~ Budget decisions.
~ Economic conditions.
~ Other research organizations.
~ Other federal agencies involved in environmental research.
~ Planning guidance.
Although ORD may not be able to control externalities, such as budget reductions
and economic downturns, ORD can benefit by recognizing such potential areas in
advance and taking preparatory actions accordingly. Furthermore, by working
together with other research organizations and federal agencies conducting
environmental research, efficiencies can be leveraged by coordinating the various
efforts so that duplication is avoided and more positive results are achieved.
12
-------
Exhibit 1:
Logic Models for Goal 8 and Objective 8.4
Figure 2: Logic Model Depicting the Goal 8 Research Program Design
Outreach
Performance Measurement (Desired)
Annual Research Planning System
Resources
EPA:
FTEs
$s
Research
Scientists &
Instruments
Non-EPA:
FTEs
$s
Equipment
Stakeholder
Guidance
Strategic
Plans
Activities
Outputs
Plan
Tods
Acquire
Technologies
Coordinate
Databases
Conduct
Research
Programs
Projects
Tasks
Ecosystem
Research
-*¦
Methods
Models
Assessments
Publications
Human Health
Risk Assessment
Research
Research to
Detect Emerging
Risks
Pollution
Prevention &New
Annual
Performance
Goals and
Measures
k
Customers Short-Term Short-Term
Reached Outcomes Outcomes
(Knowledge. (Actions)
EPA Procinms Attitudes. Skills
& Aspirations) Pollution
Intermediate
Outcomes
Long-Term
Outcomes
EXTERNALITIES
Figure 2 above presents a logic model that illustrates the general design of EPA's Goal 8 core
research program. The items identified in each rectangular "element" of the logic model present
relevant information about Goal 8 core research. The white rectangles and solid arrows represent
actual program design. The shaded rectangles and dotted arrows identify those elements in
Goal 8 plans that are not currently part of the actual program design. The program design
depicted in this model is based on: information presented in EPA's description of its Goal 8
strategy; ORD's research strategy and plans; interviews with Goal 8 program managers in ORD
and the Office of the Chief Financial Officer; and workshop discussions with ORD managers.
Figure 3 on the next page presents a logic model that illustrates the general design of EPA's
Pollution Prevention and New Technologies research program in Objective 8.4. The information
is based on: EPA's description of its Goal 8 strategy, as well as the Pollution Prevention and New
Technology Strategic Plan, ORD's draft multi-year plan, interviews with ORD Goal 8 and
Objective 8.4 program managers, and interviews and workshop discussions with ORD managers.
The design of Figure 3 is similar to Figure 2, and differs mainly in the details.
13
-------
Figure 3: Logic Model Depicting the Objective 8.4 Research Program Design
Outreach
Performance Measurement (Desired)
J
Annual Research Planning System
Resources
FY 01 EPA Resources
$58.6 million and
168.7 FTE (2001 enacted
as reported in 2002
Congressional
Justification)
Non-EPA Funding
DOD, DOE, NSF
EPA Research
Equipment:
Laboratory equipment &
supplies
Computer hardware &
software
Stakeholder Input
Industry technical
associations
Environmental groups
OSHA
Labor groups
Communities
State & Federal Agencies
Academia
Congress
Research Strategy &
Plans
Activities
Tools
(development of risk
management tools)
Green Chemistry and
Engineering
- In-House
(new technobgy
devebpment)
- Grants - new
technology development)
- Sectors
(demonstratbn of new
technologies)
Environmental
Technology Verification
(verification of the
performance of new
technologies)
Environmental Systems
Management
(develop and assemble
strategy for environmental
systems management)
Persistent
Bioaccumulative Toxics
(devebpment of better
methods for identification
and testing)
Outputs
Quality Management
Plans & Reports
Evaluation Project Plans
Evaluation Reports
Verification Reports
Verification Protocols
Demonstration Reports
Journal Publications
Presentations and
Workshops
Technology Transfer
Capsule Reports
Website Information and
Databases
Computer Models
Protocols & Methods
ORD Guidance
Documents
Annual Performance
Goals and Measure
(Current)
Customers
EPA Programs
::
OSWER, OIA OR
EPA Regions
State & Local -
Agencies
Other Federal ;
Agencies
-
Universities
Industry
Industrial Sectors -
Verification
drganlzations
i Municipalities &
Communities
Short Term
Outcomes
(Knowledge. Attitudes.
Skills. Aspirations)
I Customers gain - •.' ~
knowledge about new
technology performance,
cost, and reliability
I Custcmers:ga'iri'
knowledge of alternative
I afid less;p.dluti.og
processes
I Customers .become/-; '•••'
aware of and learn to use
new and improved risk
I management .fools"
Customer attitudes
change (i.e.. uncertainty
and risk impediments
decreased) in regard to
feasibility of using new
I ps&ilutj on prevention
I t^hnology .
I Cu'stomfs an.d . v.>
stakeholders support
I development aiVd
I application of.pollution >
prevention tools and
I technologies- ' ¦ 71'"'4;
Short Term
Outcomes
(Actions)
Increased use of new
I technologies....
Increased
development of new
I technologies .. . . .
I Pollution prevention •'
tools used to make
more quantitative
I decisions about the.
I mpstpreferred "./
I pollution prevention,.
I options •••...-• .
j Development bfless
I pcjiutirii] indbsfiai
processes
Development of new
EPA and state
I standards govepng
use of new
I te.chn.409!es' -'--: '-"'C
I F.or. Envi tonrnental
I Technology . . ; .
Verification - EPA
and Congress
determine whether
the technology
verification function
Intermediate
Outcomes
Decreased air
emissions and
aquatic discharges
Improved ambient
I conditions -,.v
Reduced human
and ecosystem
exposure to
I p.cjl.ution
LongTerm
Outcomes
Improved human
and ecosystem
health
EXTERNALITIES
14
-------
Exhibit 2:
Office of Research and Development Actions Taken
in Response to Evaluation
ORD managers, scientists, and engineers responsible for research in pollution prevention and new
technologies were cooperative and pleased to discuss their research. Their cooperation
contributed to the success of this pilot evaluation. In addition, ORD has undertaken several
actions that respond to our February 2001 briefing. These actions include:
A mid-year fiscal 2001 review that involves ORD executive managers and considers
how application of the logic model tool can improve the designs and outcome
orientation of ORD's research programs.
Guidance for ORD executives and managers to improve the outcome orientation of
annual performance goals and measures as EPA prepares its initial Fiscal Year 2003
budget and strategic plan.
Consideration of revisions to the annual research planning system that focus
discussions and decisions on performance information, including outcomes. These
revisions will allow ORD and its internal customers to understand how annual funding
decisions affect ORD's ability to achieve the short-term outcomes referenced in
Exhibit 1, and how they affect the customers' ability to achieve intermediate and long-
term outcomes.
A consultation between Objective 8.4 managers and OIG experts to identify
applications of program design and evaluation theory that may enable ORD managers
to strengthen their existing multi-year research plan for pollution prevention and new
technologies.
The first three actions identified extend to ORD's research programs across all EPA GPRA
Goals.
15
-------
16
-------
Exhibit 3:
Office of Research and Development Response
to the Draft Report
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
IS «
OFFICE OF
RESEARCH AND DEVELOPMENT
MEMORANDUM
SUBJECT: ORD Comments on Draft 01 n Evaluation Report,
"Design for Objective 8,4 Co iproved by
Reorienting Focus on Outcfoi:
FROM: Henry L, Longest II
Acting Assistant Adr (8101.R)
TO:
Kwai-Cheung Chan
Assistant Inspector General
Office of Program Evaluation (2410)
Thank you for the opportunity to review the draft program evaluation report, Design for
Objective 8.4 Could Be Improved by Reorienting Focus on Outcomes, This pilot effort
demonstrated the value of program design and evaluation techniques in designing and managing
our strategic goals and programs, and 1 commend your office for the objective and collaborative
approach that the Office of Inspector General {OIG) managers and evaluators used throughout
the process. The collaboration between OIG and the Office of Research and Development
(ORD) ensured evaluation validity in the face of significant and complex questions about the
extent to which environmental research can be managed under the Government Performance and
Results Act (GPRA). ORD learned a great deal about program evaluation, and we have
incorporated these lessons into various facets of our business processes.
EPA has made great progress in implementing GPRA, but many challenges remain. As
OIG stated in its June 2001 report titled EPA's Progress in Using the Government Performance
and Results Act to Manage for Result,';, "EPA has taken many actions to comply with GPRA's
strategic planning requirements, although it faces significant challenges in developing results-
oriented strategies that clearly link annual plans and budgets to accomplishments of long term
outcomes." Developing such linkages for environmental research is an even more challenging
task, but one that ORD takes very seriously. ORD has used its experience with the OIG pilot
program evaluation to further advance its efforts in this area, and we continue to make significant
progress in identifying the impacts and outcomes of our research programs. As an example of
our commitment in this area, during the FY 2001 mid-year reviews of ORD National
Laboratories and Centers, I asked each organization director to discuss outreach efforts and
outcome measurement in the context of the program design/evaluation logic model. In addition,
ORD used the model in its FY 2003 budget presentation to OMB.
Raeycfod/RacycUH* . Printed wits MgM* Oil Based Inks on 100% Rseycfed Paper {20* Posnconsumer)
17
-------
The attachment to this memorandum lists a number of comments on the draft report for
your consideration. Please do not hesitate to contact me or have your staff contact Howard
Cantor (202-564-5236) with any questions regarding our comments.
Attachment
cc: William Farland
Ann Akland
Tim Oppelt
18
-------
Attachment
ORD Comments on Draft OIG Report
Design for Objective 8.4 Could Be Improved by Reorienting Focus on Outcomes
1. On page i of the Executive Summary, the report states that OIG's objective was to
"document and evaluate the program designs for the EPA Strategic Plan's Goal 8 and
Objective 8.4." A similar statement appears on page 1 of the Introduction. ORD
understood the focus of the pilot to be Objective 8.4. While it is clear that some study of
Goal 8 overall was necessary for context, we did not believe that Goal 8 was the focus of
the evaluation. Further, the vast majority of ORD's discussions with the OIG team
pertained to Objective 8.4 only.
2. On page 5 of the report, the first paragraph under Summary of Observations seems to
focus on ORD's multi-year plans. This language (second, third, and fourth sentences)
appears out of place. In addition, language almost identical to these sentences appears on
page 9 under the Planning System Can Be Enhanced section. It is more appropriate there,
and we therefore suggest removing it from page 5.
3. There is a typographical error on page 12. In the last paragraph, "...ORD can benefit be
recognizing..." should be changed to "...ORD can benefit by recognizing..."
4. On page 13, the report states that Figure 3 is based on information gathered from a
number of documents, but does not reference the multi-year plan. Is this an intentional
omission, or an oversight?
19
------- |